PDR.PODM Distributed Memory
Issues
- No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
- Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.
- During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.
- Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which as result, deletes the connectivity information forever.
- The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.
Fixes
Interesting Findings
Delta 0.880
15 MPI ranks depth: 3
15 MPI ranks depth: 4
40 MPI ranks / 40 cores depth: 4
Total Time: 824.29
Total Tasks: 11413
160 MPI ranks / 10 cores depth: 4
Total Time: 378.32
Total Tasks: 12652
After Parallel int to pointer
15 MPI ranks depth: 3
| Sequential | Parallel |
|---|---|
|
|
|
15 MPI ranks depth: 4
Total time: 569.7
Total tasks: 8761
| Sequential | Parallel |
|---|---|
|
|
|
40 MPI ranks depth: 4
Total time: 326.5
Total tasks: 11201
| Sequential | Parallel |
|---|---|
|
|
|
160 MPI ranks 10 cores depth: 4
Total time: 264.3
Total tasks: 12826
| Sequential | Parallel |
|---|---|
|
|
|
After Parallel int to pointer and leaf distribution
15 MPI ranks depth: 4
Total time: 452.3
Total tasks: 8813
| Sequential | Parallel(int2ptr) | Parallel(leafDist,BadEl) |
|---|---|---|
|
|
|
|
40 MPI ranks depth: 4
Total time: 271.9
Total tasks: 11455
| Sequential | Parallel(int2ptr) | Parallel(leafDist,BadEl) |
|---|---|---|
|
|
|
|
160 MPI ranks 10 cores depth: 4
Total time: 218.3
Total tasks: 12517
| Sequential | Parallel(int2ptr) | Parallel(leafDist,BadEl) |
|---|---|---|
|
|
|
|
After Parallel int to pointer, leaf distribution and extra comm thread
15 MPI ranks depth: 3
Total time: 1144.7
Total tasks: 1830
| Sequential | Parallel(best) | Parallel(best+comm_thread) |
|---|---|---|
|
|
File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png | 
|
15 MPI ranks depth: 4
Total time: 354.6
Total tasks: 8892
| Sequential | Parallel(best) | Parallel(best+comm_thread) |
|---|---|---|
|
|
|
|
40 MPI ranks depth: 4
Total time: 208.3
Total tasks: 11468
| Sequential | Parallel(best) | Parallel(best+comm_thread) |
|---|---|---|
|
|
|
|
160 MPI ranks depth: 4
Total time: 202.2
Total tasks: 12715
| Sequential | Parallel(best) | Parallel(best+comm_thread) |
|---|---|---|
|
|
|
|
Delta 0.3780
15 MPI ranks depth: 4
Latest Results
| Cores | Total Time (s) |
|---|---|
| 100 | 1154.338082 |
| 200 | 763.70671 |
| 300 | 537.678638 |
| 400 | 490.365970 |
| 500 | 434.921334 |
| 600 | 466.822803 |
| 700 | 425.273205 |
| 800 | 434.638603 |