Telescopic Approach - Revision history

Nikos: /* Highlights of Findings */

2019-08-30T15:10:38Z

‎Highlights of Findings

Nikos: /* Integration of Loosely Coupled Approach with Tightly Coupled using CDT3D and PREMA */

2019-08-30T15:06:35Z

‎Integration of Loosely Coupled Approach with Tightly Coupled using CDT3D and PREMA

Nikos: /* Data up to 16k cores */

2019-08-30T15:04:59Z

‎Data up to 16k cores

Ctsolakis at 15:04, 30 August 2019

2019-08-30T15:04:41Z

Nikos: /* Highlight of Findings */

2019-08-30T15:04:12Z

‎Highlight of Findings

Ctsolakis at 15:03, 30 August 2019

2019-08-30T15:03:39Z

Nikos: /* Highlight of Findings */

2019-08-30T14:59:07Z

‎Highlight of Findings

Nikos: /* Highlight of Findings */

2019-08-30T14:58:29Z

‎Highlight of Findings

Nikos: /* Year 3 (Interim Report): */

2019-08-30T14:57:32Z

‎Year 3 (Interim Report):

Nikos: /* Year 2: */

2019-08-30T14:56:40Z

‎Year 2:

@@ Line 41: / Line 41: @@
 CRTC  team managed to generate with PDR guaranteed quality 104 billion element mesh using 256 nodes with 64 cores each (i.e., 16,384 cores total) in 1029 seconds. This is the largest mesh size generated so far to the best of our knowledge and suggests:
 #very good end-user productivity and
-#a solid path to effectively utilize about 20,000,000 cores by 2030 by doubling the number of cores used (effectively) every year for the next 12 years.
+#a solid path to effectively utilize about 20,000,000 cores by 2030 by doubling the number of cores used (effectively) every year for the next 10 years.

@@ Line 83: / Line 83: @@
 |}
-=== Integration of Loosely Coupled Approach with Tightly Coupled using CDT3D and PREMA ===
+=== Integration of Loosely Coupled Approach with Tightly Coupled using CDT3D and the Parallel Runtime System (PREMA) ===
 Earlier integration of PDR and Optimistic Parallel approach suggest promising results, despite the fact that the issue of balancing the work-load was not fully addressed i.e., it was addressed only at the inter-node level i.e., only between cores in the same node. In this task, a loosely coupled approach is integrated with a tightly coupled approach and in addition the workload balance is fully addressed transparently by the runtime system at the node level and the Optimistic code at the core level for each individual node.  The Figure bellow depicts the comparison between ....

@@ Line 110: / Line 110: @@
 </ul></div>
-== Data up to 16k cores ==
+== Moving Forward to Extreme Scales: Performance Data from up to 16k cores ==
 {| class="wikitable"
 ! No. of cores

@@ Line 37: / Line 37: @@
-==== Highlights of Findings  ====
+=== Highlights of Findings  ===
 CRTC  team managed to generate with PDR guaranteed quality 104 billion element mesh using 256 nodes with 64 cores each (i.e., 16,384 cores total) in 1029 seconds. This is the largest mesh size generated so far to the best of our knowledge and suggests:
@@ Line 48: / Line 48: @@
 CRTC team identified a new issue (and subsequent requirement): reproducibility. Use of TetGen in distributed memory implementation of the PDR exposed an underlying issue (strong and weak reproducibility) that is not apparent in shared memory implementations. The reproducibility issue (for precise definition see the full report) depends on the underlying sequential mesh generation code to maximize code re-use. In this specific case TetGen was used. When using TetGen over PDR’s distributed memory implementation (i.e., using MPI) a mesh needs to be reconstructed out of a set of points and tetrahedra, in order to populate the data structures of TetGen and to continue  refining the leaf from where it was left before. Although this capability exists as a function in TetGen (tetgenmesh::reconstruct()) the reconstruction procedure is not robust enough even for simple inputs. AFLR is tested at ODU and it passes the weak reproducibility requirement. The same is true for both PODM and CTD3D developed at ODU.
-====  Parallel Data Refinement (PDR) ====
+===  Parallel Data Refinement (PDR) ===
 [[File:With_optimized_3600.png|thumb|left|500px| Figure 4 :  Scalable (weak) speedup using 180 nodes at the PDR level  with 20 cores for each of the nodes managing the speculative fine-grain concurrency. ]]   TetGen's lack of reproducibility when is used in the distributed memory implementation of PDR and difficulties in completing AFLR's modifications for PDR, forced this project to rely on ODU's PODM mesh generation to further test the scalability of PDR method. Figure 4 depicts scalable speed up data for 180 nodes with 20 cores each. At the beginning of the NRA  project PODM was scaling with reasonable parallel efficiency (~ 75%) up to 140 cores, however it was dropping to 60% on 160 cores (using DSM, see [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=High%20Quality%20Real-Time%20Image-to-Mesh%20Conversion%20for%20Finite%20Element%20Simulations&paper_type=2&from=year]).  The introduction of the additional layer (PDR) helped to sustain  parallel efficiency to about 65% while increased the number of cores by more than 20 times.  These data suggest the investment to telescopic approach from both NASA and NSF is working as expected. A detailed analysis will be submitted to this year's IMR'18.
-==== CDT3D ====
+=== CDT3D ===
 While the scalability studies are progressing, ODU invested on the development of parallel (speculative i.e., lower level of the telescopic approach) advancing front with local reconnection method similar to methods implemented in AFLR. CTD3D is an alternative to AFLR and TetGen. However, in contrast to those codes, its strength is its scalability -- functionality will be added as it is need it.  CTD3D code is developed from scratch to: (i) improve boundary recovery for complex geometries (for example AFLR can not recover the faces in [[Isotropic_Mesh_Generation#Refinement_of_DLR-F6_Airbus_type_aircraft | DLRF6-Airbus]]), meet all parallel mesh generation requirements including reproducibility, be highly robust code (for example AFLR can not terminate on the 50K complexity ONERA adaptive case available  [https://github.com/UGAWG/solution-adapt-results/tree/master/onera-m6/euler-m084-a306-fun3d here ] ) and most important easily integrated within the telescopic approach. Preliminary results on the performance and comparison with AFLR presented in the table bellow and full report will by published this summer (2018)  in AIAA Aviation proceedings [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Fine-grained%20Speculative%20Topological%20Transformation%20Scheme%20for%20Local%20Reconnection%20Method&paper_type=1&from=year].  This suggests that fairly quickly we will be able to replace the scalability results presented earlier but this time for complex geometries pertinent to NASA's interests (see [[Isotropic_Mesh_Generation#Refinement_of_Nacelle_Engine.2C_Rocket.2C_and_Launch_Vehicle_with_Two_Boosters_.28lv2b.29 | nacelle engine, rocket and launch vehicle]] and [[Anisotropic_Mesh_Generation#Onera_M6 | Onera M6 ]] ).
@@ Line 83: / Line 83: @@
 |}
-==== Integration of Loosely Coupled Approach with Tightly Coupled using CDT3D and PREMA ====
+=== Integration of Loosely Coupled Approach with Tightly Coupled using CDT3D and PREMA ===
 Earlier integration of PDR and Optimistic Parallel approach suggest promising results, despite the fact that the issue of balancing the work-load was not fully addressed i.e., it was addressed only at the inter-node level i.e., only between cores in the same node. In this task, a loosely coupled approach is integrated with a tightly coupled approach and in addition the workload balance is fully addressed transparently by the runtime system at the node level and the Optimistic code at the core level for each individual node.  The Figure bellow depicts the comparison between ....
@@ Line 94: / Line 94: @@
 |}
-==== Integration of  error-based anisotropic adaptive (CTD3D) grids  with MIT's  CFD solver ====
+=== Integration of  error-based anisotropic adaptive (CTD3D) grids  with MIT's  CFD solver ===
 <div><ul>

← Older revision		Revision as of 15:04, 30 August 2019
Line 37:		Line 37:


−	==== ~~Highlight~~ of Findings ====	+	==== Highlights of Findings ====

	CRTC team managed to generate with PDR guaranteed quality 104 billion element mesh using 256 nodes with 64 cores each (i.e., 16,384 cores total) in 1029 seconds. This is the largest mesh size generated so far to the best of our knowledge and suggests:		CRTC team managed to generate with PDR guaranteed quality 104 billion element mesh using 256 nodes with 64 cores each (i.e., 16,384 cores total) in 1029 seconds. This is the largest mesh size generated so far to the best of our knowledge and suggests:

@@ Line 1: / Line 1: @@
+=== Summary ===
 The main accomplishments have been:
 <ol>

← Older revision		Revision as of 14:59, 30 August 2019
Line 44:		Line 44:


−	[[File: With_optimized_900.png\|frameless\|right\|400px\|Figure 3 : Scalable (weak) speedup using 45 nodes at the PDR level with 20 cores for each of the nodes managing the speculative fine-grain concurrency.]] The results so far are promising. While the methods remains stable (i.e., quality of the element s remains the same) the parallel efficiency remains at the same levels (i.e., about 75%). Figure depicts the scalable (weak) speedup from the integration of PDR with a ODU's mesh generation software using speculative execution at the chip level. A 50% increase (compared to last year) in the number of cores (from 600 to 900) suggest that the parallel efficiency of the PDR & Parallel Optimistic (i.e., speculative method) is sustained. The distributed memory implementation is using MPI+threads and ODUs I2M conversion (PODM) software. The optimized PDR.PODM version includes a number of communication optimizations in the packing and unpacking operation, for the mesh data structure, reduce the frequency of the data exchange while collecting data from neighbors, and “pin” some octree-leaves in certain nodes to minimize data migration. Load balancing is not addressed, since it will be taken care next year ~~(Year 3)~~ from PREMA 2.0.	+	[[File: With_optimized_900.png\|frameless\|right\|400px\|Figure 3 : Scalable (weak) speedup using 45 nodes at the PDR level with 20 cores for each of the nodes managing the speculative fine-grain concurrency.]] The results so far are promising. While the methods remains stable (i.e., quality of the element s remains the same) the parallel efficiency remains at the same levels (i.e., about 75%). Figure depicts the scalable (weak) speedup from the integration of PDR with a ODU's mesh generation software using speculative execution at the chip level. A 50% increase (compared to last year) in the number of cores (from 600 to 900) suggest that the parallel efficiency of the PDR & Parallel Optimistic (i.e., speculative method) is sustained. The distributed memory implementation is using MPI+threads and ODUs I2M conversion (PODM) software. The optimized PDR.PODM version includes a number of communication optimizations in the packing and unpacking operation, for the mesh data structure, reduce the frequency of the data exchange while collecting data from neighbors, and “pin” some octree-leaves in certain nodes to minimize data migration. Load balancing is not addressed, since it will be taken care next year from PREMA 2.0.

	CRTC team identified a new issue (and subsequent requirement): reproducibility. Use of TetGen in distributed memory implementation of the PDR exposed an underlying issue (strong and weak reproducibility) that is not apparent in shared memory implementations. The reproducibility issue (for precise definition see the full report) depends on the underlying sequential mesh generation code to maximize code re-use. In this specific case TetGen was used. When using TetGen over PDR’s distributed memory implementation (i.e., using MPI) a mesh needs to be reconstructed out of a set of points and tetrahedra, in order to populate the data structures of TetGen and to continue refining the leaf from where it was left before. Although this capability exists as a function in TetGen (tetgenmesh::reconstruct()) the reconstruction procedure is not robust enough even for simple inputs. AFLR is tested at ODU and it passes the weak reproducibility requirement. The same is true for both PODM and CTD3D developed at ODU.		CRTC team identified a new issue (and subsequent requirement): reproducibility. Use of TetGen in distributed memory implementation of the PDR exposed an underlying issue (strong and weak reproducibility) that is not apparent in shared memory implementations. The reproducibility issue (for precise definition see the full report) depends on the underlying sequential mesh generation code to maximize code re-use. In this specific case TetGen was used. When using TetGen over PDR’s distributed memory implementation (i.e., using MPI) a mesh needs to be reconstructed out of a set of points and tetrahedra, in order to populate the data structures of TetGen and to continue refining the leaf from where it was left before. Although this capability exists as a function in TetGen (tetgenmesh::reconstruct()) the reconstruction procedure is not robust enough even for simple inputs. AFLR is tested at ODU and it passes the weak reproducibility requirement. The same is true for both PODM and CTD3D developed at ODU.

@@ Line 36: / Line 36: @@
 |}
-'''Full Report : ''' [http://cs.odu.edu/crtc/PDR/cepm_internal/NRA_Annual_Report_Year2_Final.pdf http://cs.odu.edu/crtc/PDR/cepm_internal/NRA_Annual_Report_Year2_Final.pdf]
+==== Highlight of Findings  ====
-The major highlight of Year 2 is that the ODU team managed to generate with PDR guaranteed quality 104 billion element mesh using 256 nodes with 64 cores each (i.e., 16,384 cores total) in 1029 seconds. This is the largest mesh size generated so far to the best of our knowledge and suggests:
+CRTC  team managed to generate with PDR guaranteed quality 104 billion element mesh using 256 nodes with 64 cores each (i.e., 16,384 cores total) in 1029 seconds. This is the largest mesh size generated so far to the best of our knowledge and suggests:
 #very good end-user productivity and
 #a solid path to effectively utilize about 20,000,000 cores by 2030 by doubling the number of cores used (effectively) every year for the next 12 years.
 [[File: With_optimized_900.png|frameless|right|400px|Figure 3 :  Scalable (weak) speedup using 45 nodes at the PDR level  with 20 cores for each of the nodes managing the speculative fine-grain concurrency.]] The results so far are promising. While the methods remains stable (i.e., quality of the element s remains the same) the parallel efficiency remains at the same levels (i.e., about 75%). Figure depicts the scalable (weak) speedup from the integration of PDR with a ODU's mesh generation software using speculative execution at the chip level.  A 50%  increase (compared to last year) in the number of cores (from 600 to 900) suggest that the parallel efficiency of the PDR & Parallel Optimistic (i.e., speculative method) is sustained.  The  distributed memory implementation is using MPI+threads and ODUs I2M conversion (PODM) software.  The optimized PDR.PODM version includes a number of  communication optimizations  in the packing and unpacking operation, for  the mesh data structure, reduce the frequency of the data exchange  while collecting data from neighbors, and “pin” some octree-leaves in certain nodes to minimize data migration. Load balancing is not addressed, since it will be taken care next year (Year 3) from PREMA 2.0.
-Finally, we identified a new issue (and subsequent requirement): reproducibility. Use of TetGen in distributed memory implementation of the PDR exposed an underlying issue (strong and weak reproducibility) that is not apparent in shared memory implementations. The reproducibility issue (for precise definition see the full report) depends on the underlying sequential mesh generation code to maximize code re-use. In this specific case TetGen was used. When using TetGen over PDR’s distributed memory implementation (i.e., using MPI) a mesh needs to be reconstructed out of a set of points and tetrahedra, in order to populate the data structures of TetGen and to continue  refining the leaf from where it was left before. Although this capability exists as a function in TetGen (tetgenmesh::reconstruct()) the reconstruction procedure is not robust enough even for simple inputs. AFLR is tested at ODU and it passes the weak reproducibility requirement. The same is true for both PODM and CTD3D developed at ODU.
+CRTC team identified a new issue (and subsequent requirement): reproducibility. Use of TetGen in distributed memory implementation of the PDR exposed an underlying issue (strong and weak reproducibility) that is not apparent in shared memory implementations. The reproducibility issue (for precise definition see the full report) depends on the underlying sequential mesh generation code to maximize code re-use. In this specific case TetGen was used. When using TetGen over PDR’s distributed memory implementation (i.e., using MPI) a mesh needs to be reconstructed out of a set of points and tetrahedra, in order to populate the data structures of TetGen and to continue  refining the leaf from where it was left before. Although this capability exists as a function in TetGen (tetgenmesh::reconstruct()) the reconstruction procedure is not robust enough even for simple inputs. AFLR is tested at ODU and it passes the weak reproducibility requirement. The same is true for both PODM and CTD3D developed at ODU.
 </li>