crtc.cs.odu.edu - User contributions [en]

PDR.PODM Distributed Memory

2023-05-19T14:49:33Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 90.8 || 49453719
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! !! colspan=2 | MPI !! colspan=2 | PREMA
|-
! Cores !! Total Time (s) !! Number of Elements !! Total Time (s) !! Number of Elements
|-
| 100 || 1151.472406|| 49352359 || 208.746450 || 49347855
|-
| 200 || 763.70671 || 49357898 || 121.326012 || 49353442
|-
| 300 || 537.678638 || 49357092 || 105.812248 || 49351224
|-
| 400 || 490.365970 || 49357881 || 93.101481 || 49351626
|-
| 500 || 434.921334 || 49347357 || 93.119187 || 49351049
|-
| 600 || 466.822803 || 49361647 || 96.243807 || 49345030
|-
| 700 || 425.273205 || 49360615 || 97.721654 || 49346798
|-
| 800 || 434.638603 || 49349629 || 96.723318 || 49341034
|-
|}

[[File:PDR.png]]

Main Page

2023-05-16T14:04:56Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
Congratulations to Dr. Thomadakis for his impressive Ph.D. work on parallel intelligent runtime software systems for HPC adaptive and irregular applications and Domain-Specific Languages.

The AI part of his PhD work ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Using%20Machine%20Learning%20for%20Particle%20Track%20Identification%20in%20the%20CLAS12%20Detector&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Track%20Identification%20for%20CLAS12%20using%20Artificial%20Intelligence&paper_type=1&from=year]) was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged in a recent article ([https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis]) that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables from 35% to approximately 50-80% by adding the AI-based denoising ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] )of the data in the workflow.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work on “steroids” using scalable D-NISQ Quantum Machine Learning. Our preliminary results ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Evaluation%20of%20Scalable%20Quantum%20and%20Classical%20Machine%20Learning%20for%20Particle%20Tracking%20Classification%20in%20Nuclear%20Physics&paper_type=1&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Scalable%20Quantum%20Edge%20Detection%20Method%20for%20D-NISQ%20Imaging%20Simulations:%20Use%20Cases%20from%20Nuclear%20Physics%20and%20Medical%20Image%20Computing&paper_type=1&from=year]) in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

=== Protein-to-Protein Interactions ===
Progress on protein-to-protein interactions using geometric feature analysis: [[COVID-19]].

A slideshows of Molecular Geometric Feature Analysis for SARS-CoV-2: click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T12:51:47Z

Pthomadakis: /* COVID-19 */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Using%20Machine%20Learning%20for%20Particle%20Track%20Identification%20in%20the%20CLAS12%20Detector&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Track%20Identification%20for%20CLAS12%20using%20Artificial%20Intelligence&paper_type=1&from=year]) was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Evaluation%20of%20Scalable%20Quantum%20and%20Classical%20Machine%20Learning%20for%20Particle%20Tracking%20Classification%20in%20Nuclear%20Physics&paper_type=1&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Scalable%20Quantum%20Edge%20Detection%20Method%20for%20D-NISQ%20Imaging%20Simulations:%20Use%20Cases%20from%20Nuclear%20Physics%20and%20Medical%20Image%20Computing&paper_type=1&from=year]) in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

=== COVID-19 ===
Check our work on [[COVID-19]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T12:51:11Z

Pthomadakis: /* Announcements */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Using%20Machine%20Learning%20for%20Particle%20Track%20Identification%20in%20the%20CLAS12%20Detector&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Track%20Identification%20for%20CLAS12%20using%20Artificial%20Intelligence&paper_type=1&from=year]) was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Evaluation%20of%20Scalable%20Quantum%20and%20Classical%20Machine%20Learning%20for%20Particle%20Tracking%20Classification%20in%20Nuclear%20Physics&paper_type=1&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Scalable%20Quantum%20Edge%20Detection%20Method%20for%20D-NISQ%20Imaging%20Simulations:%20Use%20Cases%20from%20Nuclear%20Physics%20and%20Medical%20Image%20Computing&paper_type=1&from=year]) in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

=== COVID-19 ===
Our work on COVID-19 can be found [[COVID-19 here]]

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

COVID-19

2023-05-12T12:48:32Z

Pthomadakis: Created page with "===Molecular Dynamics=== ==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ==== <html> <div style="text-align:center;"> <video width="700..."

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

Main Page

2023-05-12T12:24:03Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Using%20Machine%20Learning%20for%20Particle%20Track%20Identification%20in%20the%20CLAS12%20Detector&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Track%20Identification%20for%20CLAS12%20using%20Artificial%20Intelligence&paper_type=1&from=year]) was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Evaluation%20of%20Scalable%20Quantum%20and%20Classical%20Machine%20Learning%20for%20Particle%20Tracking%20Classification%20in%20Nuclear%20Physics&paper_type=1&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Scalable%20Quantum%20Edge%20Detection%20Method%20for%20D-NISQ%20Imaging%20Simulations:%20Use%20Cases%20from%20Nuclear%20Physics%20and%20Medical%20Image%20Computing&paper_type=1&from=year]) in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T11:47:32Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work ([https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Using%20Machine%20Learning%20for%20Particle%20Track%20Identification%20in%20the%20CLAS12%20Detector&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Track%20Identification%20for%20CLAS12%20using%20Artificial%20Intelligence&paper_type=1&from=year]) was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T11:47:10Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work[https://crtc.cs.odu.edu/pub/html_output/details.php?&title=De-noising%20drift%20chambers%20in%20CLAS12%20using%20convolutional%20auto%20encoders&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Using%20Machine%20Learning%20for%20Particle%20Track%20Identification%20in%20the%20CLAS12%20Detector&paper_type=2&from=year] [https://crtc.cs.odu.edu/pub/html_output/details.php?&title=Track%20Identification%20for%20CLAS12%20using%20Artificial%20Intelligence&paper_type=1&from=year] was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T11:42:10Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T11:41:40Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|200px|left]]

I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T11:41:25Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg|300px|left]]

I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

File:Polykarpos-Grad.jpeg

2023-05-12T11:40:14Z

Pthomadakis: Pthomadakis uploaded a new version of File:Polykarpos-Grad.jpeg

File:Polykarpos-Grad.jpeg

2023-05-12T11:33:51Z

Pthomadakis:

Main Page

2023-05-12T11:33:10Z

Pthomadakis: /* Congratulations to Dr. Thomadakis */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations to Dr. Thomadakis===
[[File:Polykarpos-Grad.jpeg]]

I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work and for defending and officially completing all his Ph.D. requirements, today!

The AI part of his PhD work was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other former members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including current member Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

Next, Dr. Gagik will be looking into applying lessons learned to other Jefferson Lab Halls (sensors) and we will be putting this AI work in “steroids” using Quantum Machine Learning. Our preliminary results in Quantum Computing indicate that this is feasible.

These accomplishments would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

Main Page

2023-05-12T01:56:32Z

Pthomadakis: /* Congratulations */

[[File:Thanksgiving lunch 2019.jpg|600px|thumb|frameless|center| Chrisochoides' Real-Time Computing (or in short CRTC) group before the pandemic.]]

== Announcements ==

=== Congratulations===
I want to offer congratulations to Dr. Thomadakis for his impressive Ph.D. work. Defend and officially complete all his Ph.D. requirements, today!

The AI part of it was completed in collaboration with [https://www.linkedin.com/in/gagik-gavalian-55926725/ Dr. Gagik Gavalian] from [https://www.jlab.org/ Jefferson Lab] and other members of the [https://crtc.cs.odu.edu/Main_Page CRTC], including Kevin Garner. Some of the findings from his research is projected to have a significant impact over the next decade, with a value of approximately $100 million.

These results were confirmed by Nuclear Physicists in a [https://www.odu.edu/article/ai-collaboration-between-odu-jefferson-lab-improves-data-analysis recent article], and Dr. Gavalian presented the findings in [https://indico.jlab.org/event/459/contributions/11745/attachments/9491/13757/CHEP-05-2023-Gavalian.pdf CHEP'23]. [https://www.odu.edu/directory/lawrence-weinstein Dr. Lawrence Weinstein], an ODU Eminent Scholar, professor of physics, and past chair of the Jefferson Lab Users Organization, acknowledged that the project has improved particle trajectory reconstruction, resulting in a 35% increase in the number of reconstructed complicated nuclear collisions. This means that "more physics from the same data" is possible, which would have otherwise cost about $5 million per year. Dr. Gavalian's new results show that there has been an increase in statistics for physics observables of approximately 50-80%.

This accomplishment would not have been possible without the small investment from Jefferson Lab ($100K) and the Richard T. Cheng Endowment ($50K), as well as the help of many others, including the Dragas Foundation, which has assisted Olga Karadimou, an exceptional Ph.D. student at ODU, in staying at ODU with Polykarpos.

===Molecular Dynamics===
==== SARS-CoV-2 D614 (original strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S5-D614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

==== SARS-CoV-2 G614 (new strain) Spike Cleavage Site Interaction with Furin ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/S6-G614-w-furin.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

===Molecular Geometric Feature Analysis===

==== SARS-CoV-2 G614 Spike Cleavage Domain Tunnels (red-blue) Superimposed on D614 Spike Cleavage Domain with D614 Tunnel (purple-orange) using the atomic structure from the original strain ====
<html>
<div style="text-align:center;">
<video width="700" height="500" controls loop>
<source src="https://crtc.cs.odu.edu/assets/videos/D614-G614-TunnelsSuperimposeAnim-web.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
</html>

'''Remark''': Notice the difference between the position of new tunnels where the C & O used to be in the original COVID19 strain

To see other videos please click [[Molecular Geometric Feature Analysis | here]].

==== SARS-CoV-2 D614 vs G614 Mutation in Spike Cleavage Domain ====
<gallery heights=300px widths=300px mode="slideshow">
File:CRTC_ProteinCavities_D614G_1.png
File:CRTC_ProteinCavities_D614G_2.png ‎
File:CRTC_ProteinCavities_D614G_3.png
File:CRTC_ProteinCavities_D614G_4.png
File:CRTC_ProteinCavities_D614G_5.png
File:CRTC_ProteinCavities_D614G_6.png
File:CRTC_ProteinCavities_D614G_7.png
File:CRTC_ProteinCavities_D614G_8.png
</gallery>

To see other slideshows please click [[Molecular Geometric Feature Analysis | here]].

== Introduction ==
Finite Element (FE) mesh generation is a critical component for many (bio-)engineering and science applications. CRTCLab is developing a novel framework for highly scalable and energy-efficient high-quality mesh generation for the Finite Element Analysis in three and four dimensions. CRTC effort and focus is on research activities that combine domain-and application-speciﬁc knowledge with run-time system support to improve energy efficiency and scalability of parallel FE mesh generation codes. Traditionally, parallel FE mesh generation methods and software are developed without considering the architectural features of the supercomputer platforms on which they are eventually used for production. The main reason is the complexity of sequential, and moreover parallel, mesh generation algorithms. As a result, it is too expensive, in terms of labor and time, to customize the performance of parallel mesh generation software for speciﬁc supercomputing architectures. CRTC’s basic research is funded from NSF, NASA, NRA, total $2M and target (bio-)engineering applications are: [[#Extreme-Scale_Parallel_Mesh_Generation_&_Adaptation | Aerospace Industry Applications]] and [[#Medical_Image_Computing | Health Care Industry Applications]].

<center>
'''<big>Principles for Building Practical and Scalable Domain-Specific Environment: Adaptive Data-Driven Mesh Generation for Modeling & Simulations</big>
'''
</center>
[[File:Adaptivity PPoSS.png|thumb|center|600px| This graph indicates the core enabling technologies we develop at CRTC: (i) Adaptive Tessellation Techniques: Theory and Systems and the corresponding and (ii) Parallel Runtime Software Systems required to run on current and emerging Supercomputers. The pictures around indicate the five main applications we target from left to right: (a) Computational Fluid Dynamics, (b) Numerical Flow Simulations used to Predict Flow Diversion Treatment Efficacy of Cerebral Aneurysms, (c) Trauma Brain Injuries, (d) Physic Based Adaptive Deformable Registration used in Image-Guided Neurosurgery, and (e) Imaging for Nuclear Femtography with our collaborators: from NASA/LARC, Stony Brook Medical School, Navy Research Lab, Harvard Medical School, and Physics and Jefferson Lab, respectively. The common denominator for all these applications is adaptive parallel tessellation ( or known in the science and engineering community as mesh generation or image-to-mesh conversion). Parallel mesh generation is "a new research area between the boundaries of two [https://en.wikipedia.org/wiki/Scientific_computing scientific computing] disciplines: [https://en.wikipedia.org/wiki/Computational_geometry computational geometry] and [https://en.wikipedia.org/wiki/Parallel_computing parallel computing]" [https://en.wikipedia.org/wiki/Parallel_mesh_generation]. The applications are listed (left to right) with respect to interactions of input form Computer-Aided Design to Sensor data like MRI, US, etc.]]

'''Broader Impact'''
The proposed activity will have a signiﬁcant and broader impact on research and education, well beyond the bounds of this speciﬁc project and its external collaborators. Other than the direct impact to parallel unstructured mesh generation and runtime systems for supercomputers like Blue Waters, this project will beneﬁt the performance evaluation community. This project will also promote the understanding on CerebroVascular Disease (CVD, or stroke), Image Guided Surgery for Caner, and DBS for Parkisons disease, all of which are leading natural death causes in the US.

'''Intellectual Merit'''
The Intellectual Merit of this project is summarized in terms of three novel contributions: (1) an understanding of the complex computation, communication, and memory utilization patterns in a telescopic superposition of multiple parallel mesh generation codes, (2) new abstractions to efficiently throttle concurrency and to implement race-to-halt execution models at multiple levels of granularity, while orchestrating intra-and inter-layer memory management, data movement, and load balancing to improve energy-efficiency of memory and interconnect components, and (3) a novel approach to leveraging off-the-shelf and state-of-the-art low core meshing codes for highly parallel simulations.

== Extreme-Scale Adaptive/Anisotropic Parallel Mesh Generation ==
<gallery heights=300px widths=300px mode="slideshow">
File:telescopic_and_meshes.jpg
File:CDT3D slide rocket nacelle.jpg ‎
File:Dlr airbus.jpg
File:Scitech 2019 delta wing.jpg
</gallery>

<gallery heights=200px widths=200px mode="packed-hover" caption="CDT3D Meshes using speculative execution model at the chip level">
File:CDT3D.jpg|130px|CDT3D Mesh
File:Rocket_Mesh.jpg|130px|DLR-F6 Airbus
File:Output_grid.jpg|130px|Rocket with Engine Mesh
|- style="text-align: center;"
||||'''CDT3D Meshes using speculative execution model at the chip level'''
</gallery>

=== Overview ===
Finite Element Mesh Generation is a critical component for many (bio-) engineering and science applications. The goal of this project is to deliver a novel Telescopic framework for highly scalable and energy efficient codes. Domain-and application-specific knowledge and run-time system support are combined to improve accuracy of FE computations.

CRTCLab addresses in depth two pillars: (i) high performance computing software runtime systems and (ii) mesh generation & adaptation based on error-metrics for NASA’s Computational Fluid Dynamics (CFD) Vision 2030. Specifically, CRTC’s focus is on the following three objectives: (1) design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation using state-of-the-art functionality supported by algorithms from AFLR and CRTC’s telescopic approach for parallel mesh generation , (2) develop error-based metrics adaptive anisotropic FE mesh generation methods and (3) design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretizations with localizable error estimates. <u> This is a joined project with NASA/LaRC and MIT. </u>

<li style="display: inline-block;">
[[File: SubGroup1.jpg|thumb|right|500px| '''CRTC's sub-group for Extreme Scale Parallel Mesh Generation & Adaptation, from left to right: Nikos Chrisochoides, Kevin Garner, Dana Hammond (former TM from NASA/LaRC), Christos Tsolakis and Polykarpos Thomadakis. ''']]
We have assembled a team of established leaders (see [https://cepm.cs.odu.edu/People#External_Collaborators External Collaborators]) that are currently developing state-of-the-art work on mesh generation and adaptivity issues relevant to NASA’s CFD 2030 Vision and will broadly impact end-user productivity of users throughout DoD and NASA.

'''Objectives'''
:# Design a multi-layered algorithmic and software framework for 3D tetrahedral anisotropic parallel mesh generation methods using state-of-the-art functionality supported by methods implemented in AFLR and CRTC’s telescopic approach for parallel mesh generation.
:# Development of error-based metrics to drive an anisotropic adaptive process
:# Design a power-aware parallel runtime software system for extreme-scale adaptive CFD computations including: (i) mesh generation & adaptation, and (ii) consistent error-based metrics for adaptation of any CFD discretization with localizable error estimates.
</li>

== Real-time Adaptive Mesh Generation in Medical Image Computing ==

This application is using adaptive (isotropic) mesh refinement to improve (in real-time in the context of image-guided neurosurgery) the accuracy of 3D Physics-Based Deformable Registration of intra-operative MRI with pre-operative MRI.

=== Overview ===
<li style="display: inline-block;">
[[File: CT imaging.jpg|thumb|right|500px| '''Deep Brain Stimulation – CT Imaging obtained from Neurosurgical Section of SE PADRECC McGuire VAMC (with VCU)''']]
:* Develop adaptive deformable registration method for Deep Brain Stimulation (DBS). DBS is an effective palliative therapy for patients suffering from Essential Tremor, Parkinson’s disease, and other neurological movement disorders. Modern DBS surgery makes use of stereotactic systems and image guidance to accurately place electrode leads, as well as intra-operative imaging to surveil the location of the leads and guide the surgery. The effectiveness of DBS is directly correlated with the accuracy of DBS electrode lead placement, with more accurate electrode placement leading to better clinical outcomes. <u> This project is a joined with VCU, VA. </u>
</li>

<li style="display: inline-block;">
[[File: Health-care.jpg|thumb|left|500px| '''APBNRR ''']]
:* Develop a new Adaptive Physics-Based Non-Rigid Registration (APBNRR) method developed for image guided brain tumor resection. Existing evaluation indicates that the the registration accuracy and performance of APBNRR are proved sufficient to be of clinical value in the operating room. Malignant gliomas are the most common primary and metastatic brain tumors. Treatment typically includes surgical removal followed by radiotherapy or chemotherapy. Total resection is difficult to achieve during tumor removal because of the infiltrative nature of gliomas and because brain tumors are often embedded in critical functional brain tissue. In image-guided neurosurgery, co-registered pre-operative anatomical, functional, and diffusion tensor imaging can be used to facilitate a maximally safe resection of brain tumors in certain areas of the brain. However, because the brain can deform significantly during surgery, particularly in the presence of tumor resection, non-rigid registration of the preoperative image data to the patient is required. <u> This is a joined project with Harvard Medical School, MA. </u>
</li>

<li style="display: inline-block;">
[[File: ExtremeScale.jpg|thumb|right|500px| '''Images created by the I2M tool using the CBC3D Method''']]
:* Test five new available (in the clinic) neurovascular devices that have not been compared before through angiographic washout analysis and accurate micro CT-based tessellation (i.e., mesh or grid reconstruction) and CFD analysis in idealized sidewall aneurysm models. The CFD simulations and angiographic washout analysis will be performed on flow diverter implantations in actual patient geometries. There is a need for robust and easy-to-use Image-to-Mesh (I2M) conversion methods and software that streamline the discretization of such complex vascular geometries with fine structures with the CFD modeling process. In this project we will focus on the development and validation of fully functional I2M conversion algorithms and software (i.e., tools) for complex vascular geometries with fine structures like stents. <u> This is a joined project with Medical School at Stony Brook University, NY. </u>
</li>

File:PDR.png

2023-03-24T03:18:17Z

Pthomadakis:

PDR.PODM Distributed Memory

2023-03-24T03:17:25Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 107.8 || 216099955
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! !! colspan=2 | MPI !! colspan=2 | PREMA
|-
! Cores !! Total Time (s) !! Number of Elements !! Total Time (s) !! Number of Elements
|-
| 100 || 1151.472406|| 49352359 || 208.746450 || 49347855
|-
| 200 || 763.70671 || 49357898 || 121.326012 || 49353442
|-
| 300 || 537.678638 || 49357092 || 105.812248 || 49351224
|-
| 400 || 490.365970 || 49357881 || 93.101481 || 49351626
|-
| 500 || 434.921334 || 49347357 || 93.119187 || 49351049
|-
| 600 || 466.822803 || 49361647 || 96.243807 || 49345030
|-
| 700 || 425.273205 || 49360615 || 97.721654 || 49346798
|-
| 800 || 434.638603 || 49349629 || 96.723318 || 49341034
|-
|}

[[File:PDR.png]]

File:Pdr.mpi.png

2023-03-24T03:16:29Z

Pthomadakis: Pthomadakis uploaded a new version of File:Pdr.mpi.png

PDR.PODM Distributed Memory

2023-03-24T01:07:58Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 107.8 || 216099955
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! !! colspan=2 | MPI !! colspan=2 | PREMA
|-
! Cores !! Total Time (s) !! Number of Elements !! Total Time (s) !! Number of Elements
|-
| 100 || 1151.472406|| 49352359 || 208.746450 || 49347855
|-
| 200 || 763.70671 || 49357898 || 121.326012 || 49353442
|-
| 300 || 537.678638 || 49357092 || 105.812248 || 49351224
|-
| 400 || 490.365970 || 49357881 || 93.101481 || 49351626
|-
| 500 || 434.921334 || 49347357 || 93.119187 || 49351049
|-
| 600 || 466.822803 || 49361647 || 96.243807 || 49345030
|-
| 700 || 425.273205 || 49360615 || 97.721654 || 49346798
|-
| 800 || 434.638603 || 49349629 || 96.723318 || 49341034
|-
|}

[[File:pdr.mpi.png]]

PDR.PODM Distributed Memory

2023-03-24T01:04:05Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 107.8 || 216099955
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! !! MPI !! !! PREMA !!
|-
! Cores !! Total Time (s) !! Number of Elements !! Total Time (s) !! Number of Elements
|-
| 100 || 1151.472406|| 49352359 || 208.746450 || 49347855
|-
| 200 || 763.70671 || 49357898 || 121.326012 || 49353442
|-
| 300 || 537.678638 || 49357092 || 105.812248 || 49351224
|-
| 400 || 490.365970 || 49357881 || 93.101481 || 49351626
|-
| 500 || 434.921334 || 49347357 || 93.119187 || 49351049
|-
| 600 || 466.822803 || 49361647 || 96.243807 || 49345030
|-
| 700 || 425.273205 || 49360615 || 97.721654 || 49346798
|-
| 800 || 434.638603 || 49349629 || 96.723318 || 49341034
|-
|}

[[File:pdr.mpi.png]]

PDR.PODM Distributed Memory

2023-03-14T20:44:33Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 107.8 || 216099955
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 100 || 1151.472406|| 49352359
|-
| 200 || 763.70671 || 49357898
|-
| 300 || 537.678638 || 49357092
|-
| 400 || 490.365970 || 49357881
|-
| 500 || 434.921334 || 49347357
|-
| 600 || 466.822803 || 49361647
|-
| 700 || 425.273205 || 49360615
|-
| 800 || 434.638603 || 49349629
|-
|}

[[File:pdr.mpi.png]]

PDR.PODM Distributed Memory

2023-03-14T20:44:10Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 107.8 || 216099955
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 100 || 1154.338082 || 49352359
|-
| 200 || 763.70671 || 49357898
|-
| 300 || 537.678638 || 49357092
|-
| 400 || 490.365970 || 49357881
|-
| 500 || 434.921334 || 49347357
|-
| 600 || 466.822803 || 49361647
|-
| 700 || 425.273205 || 49360615
|-
| 800 || 434.638603 || 49349629
|-
|}

[[File:pdr.mpi.png]]

PDR.PODM Distributed Memory

2023-03-14T20:12:09Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Shared MemoryTimes
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 40|| 107.8 || 216099955
|-
|}

{| class="wikitable"
|+ MPI Times
|-
! Cores !! Total Time (s) !! Number of Elements
|-
| 100 || 1154.338082 ||
|-
| 200 || 763.70671 || 49357898
|-
| 300 || 537.678638 || 49357092
|-
| 400 || 490.365970 || 49357881
|-
| 500 || 434.921334 || 49347357
|-
| 600 || 466.822803 || 49361647
|-
| 700 || 425.273205 || 49360615
|-
| 800 || 434.638603 || 49349629
|-
|}

[[File:pdr.mpi.png]]

File:Pdr.mpi.png

2023-03-06T13:02:06Z

Pthomadakis:

PDR.PODM Distributed Memory

2023-03-06T13:01:57Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ MPI Times
|-
! Cores !! Total Time (s)
|-
| 100 || 1154.338082
|-
| 200 || 763.70671
|-
| 300 || 537.678638
|-
| 400 || 490.365970
|-
| 500 || 434.921334
|-
| 600 || 466.822803
|-
| 700 || 425.273205
|-
| 800 || 434.638603
|-
|}

[[File:pdr.mpi.png]]

PDR.PODM Distributed Memory

2023-03-06T12:56:11Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ MPI Times
|-
! Cores !! Total Time (s)
|-
| 100 || 1154.338082
|-
| 200 || 763.70671
|-
| 300 || 537.678638
|-
| 400 || 490.365970
|-
| 500 || 434.921334
|-
| 600 || 466.822803
|-
| 700 || 425.273205
|-
| 800 || 434.638603
|-
|}

PDR.PODM Distributed Memory

2023-03-06T12:55:52Z

Pthomadakis: /* Latest Results */

=== Issues ===
* No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.
[[File:PDR_PODM_Leaves_not_refined.png| 700px]]
* Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but
it does not have any neighbour in the same leaf.

[[File:PDR_PODM_Cells_not_distributed.png| 700px]]

* During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,
it is set to the infinite cell. This causes PODM to crash randomly for some cases.

* Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and
updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which
as result, deletes the connectivity information forever.

* The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,
neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share
a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

=== Fixes ===

[[File:PDR_Fix.png| 700px]]

[[File:Work_Unit_After_Refinement.png| 700px]]

= Interesting Findings =
== Delta 0.880 ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3.png| 700px]]
[[File:PDR PODM Time Break Down 15 d3.png| 700px]]

[[File:PDR PODM Parallelism.png| 700px]]
[[File:PDR_PODM_Histogram_15.png| 700px]]
</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15.png| 700px]]
[[File:PDR PODM Histogram Tasks 15.png| 700px]]
[[File:PDR PODM Time Break Down 15.png| 700px]]

</div>

=== 40 MPI ranks / 40 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 824.29

Total Tasks: 11413

[[File:PDR PODM Histogram Time 40.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40.png| 700px]]
[[File:PDR PODM Time Break Down 40.png| 700px]]
</div>

=== 160 MPI ranks / 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total Time: 378.32

Total Tasks: 12652

[[File:PDR PODM Histogram Time 160 10.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10.png| 700px]]
[[File:PDR PODM Time Break Down 160 10.png| 700px]]
</div>

== After Parallel int to pointer ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time 15 d3 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 d3 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 700px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr.png| 700px]]
|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 569.7

Total tasks: 8761

[[File:PDR PODM Histogram Time 15 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 700px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 700px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 326.5

Total tasks: 11201

[[File:PDR PODM Histogram Time 40 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 700px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 700px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 264.3

Total tasks: 12826

[[File:PDR PODM Histogram Time 160 10 par int2ptr.png| 700px]]
[[File:PDR PODM Histogram Tasks 160 10 par int2ptr.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 700px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 700px]]
|}

</div>

== After Parallel int to pointer and leaf distribution ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 452.3

Total tasks: 8813

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 15 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 271.9

Total tasks: 11455

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histogram Tasks 40 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
|}

</div>

=== 160 MPI ranks 10 cores depth: 4 ===
<div class="mw-collapsible mw-collapsed">
Total time: 218.3

Total tasks: 12517

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(int2ptr)
! Parallel(leafDist,BadEl)
|-
| [[File:PDR PODM Time Break Down 160 10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
|}
</div>

== After Parallel int to pointer, leaf distribution and extra comm thread ==
=== 15 MPI ranks depth: 3 ===
<div class="mw-collapsible mw-collapsed">

Total time: 1144.7

Total tasks: 1830

[[File:PDR PODM Histogram Time 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 d3 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15_d3.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 d3 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 354.6

Total tasks: 8892

[[File:PDR PODM Histogram Time 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 15 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_15.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 15 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 40 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 208.3

Total tasks: 11468

[[File:PDR PODM Histogram Time 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 40 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_40.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 40 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

=== 160 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

Total time: 202.2

Total tasks: 12715

[[File:PDR PODM Histogram Time 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]
[[File:PDR PODM Histrogram Tasks 160 10 par int2ptr leaf dist bad elements comm thread.png| 700px]]

{| class="wikitable" style="text-align: center;
! Sequential
! Parallel(best)
! Parallel(best+comm_thread)
|-
| [[File:PDR_PODM_Time_Break_Down_160_10.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements.png| 500px]]
| [[File:PDR PODM Time Break Down 160 10 par int2ptr leaf dist bad elements comm thread.png| 500px]]

|}

</div>

== Delta 0.3780 ==
=== 15 MPI ranks depth: 4 ===
<div class="mw-collapsible mw-collapsed">

[[File:PDR PODM Histogram Time small_delta 15.png| 700px]]
[[File:PDR PODM Histogram Tasks small_delta 15.png| 700px]]
[[File:PDR PODM Time Break Down small_delta 15.png| 700px]]

</div>

=== Latest Results ===

{| class="wikitable"
|+ Caption text
|-
! Cores !! Total Time (s)
|-
| 100 || 1154.338082
|-
| 200 || 763.70671
|-
| 300 || 537.678638
|-
| 400 || 490.365970
|-
| 500 || 434.921334
|-
| 600 || 466.822803
|-
| 700 || 425.273205
|-
| 800 || 434.638603
|-
|}

PDR.PODM Distributed Memory

2023-03-06T12:49:46Z

Pthomadakis:

How to update publications page

2023-02-10T00:39:44Z

Pthomadakis: /* Adding a new paper */

== Guide to updating CRTC wiki publication page ==
This is a short guide on how to update the CRTC wiki publication page. The publication page lists all publications of the CRTC group and is maintained (like the whole wiki) from the CS department's IT group. If you have any issues or questions, please contact: root@cs.odu.edu.

== Prerequisites ==
In order to be able to access the page to modify the publications list you will need the following prerequisites:
# Have a CS account. All CS students are given such an account when first joining the department.
# Connect to the [https://systems.cs.odu.edu/VPN_Services CS VPN] or use a machine located in the CS department.
## Note that connecting to ODU's VPN will not work for this.
# Be a member of the CRTC group (email root@cs.odu.edu to request this)

== Accessing the publications' administrative page ==
First access the [[Publications]] page. On the top right you should see red link reading "Authenticated Users" as in the picture below:

[[File:Guide-publ-1.png|800px]]

Click on the link and you should be directed to a login page, like below. Enter you CS account credentials and sign in.

(If you get an error, make sure you are connected to the CS department network or VPN.)

[[File:Guide-publ-2.png|800px]]

If everything was successful, you should see the following html page:

[[File:Guide-publ-3.png|800px]]

== Updating the publications page ==
Users can perform the following updates to the page:
# Add a new paper/author/venue/minisymposium
# Modify an existing paper/author/venue/minisymposium
# Remove an existing paper/author/venue/minisymposium

=== Adding a new paper ===
In order to add a new paper, the paper's author(s) and the venue/minisymposium it was submitted to must have been already added to the database through the respective links.

Once this information has been added to the database, it will become available at the respective drop-down menus.

An example of the whole process is shown in the steps below.

*Step 1: Add author (Skip if all co-authors have published at least one paper with the CRTC group):
[[File:Guid-publ-4.png | 400px]]

Simply fill in the fields requeted: [[File:Guid-publ-5.png | 400px]]

Repeat the same process to add more authors.

*Step 2: Add venue/minisymposium (Skip if CRTC group already published there before):
[[File:Guid-publ-6.png | 400px]]

Simply fill in the fields requested as before: [[File:Guid-publ-7.png | 400px]]

*Step 3: Add paper:
[[File:Guid-publ-8.png | 400px]]

Select the venue type (Conference or Journal): [[File:Guid-publ-9.png | 400px]]

Simply fill in the fields requested and choose the respective authors and venue from the drop-down menus:

[[File:Guid-publ-10.png | 800px]]

How to update publications page

2023-02-10T00:37:53Z

Pthomadakis: /* Adding a new paper */

== Guide to updating CRTC wiki publication page ==
This is a short guide on how to update the CRTC wiki publication page. The publication page lists all publications of the CRTC group and is maintained (like the whole wiki) from the CS department's IT group. If you have any issues or questions, please contact: root@cs.odu.edu.

== Prerequisites ==
In order to be able to access the page to modify the publications list you will need the following prerequisites:
# Have a CS account. All CS students are given such an account when first joining the department.
# Connect to the [https://systems.cs.odu.edu/VPN_Services CS VPN] or use a machine located in the CS department.
## Note that connecting to ODU's VPN will not work for this.
# Be a member of the CRTC group (email root@cs.odu.edu to request this)

== Accessing the publications' administrative page ==
First access the [[Publications]] page. On the top right you should see red link reading "Authenticated Users" as in the picture below:

[[File:Guide-publ-1.png|800px]]

Click on the link and you should be directed to a login page, like below. Enter you CS account credentials and sign in.

(If you get an error, make sure you are connected to the CS department network or VPN.)

[[File:Guide-publ-2.png|800px]]

If everything was successful, you should see the following html page:

[[File:Guide-publ-3.png|800px]]

== Updating the publications page ==
Users can perform the following updates to the page:
# Add a new paper/author/venue/minisymposium
# Modify an existing paper/author/venue/minisymposium
# Remove an existing paper/author/venue/minisymposium

=== Adding a new paper ===
In order to add a new paper, the paper's author(s) and the venue/minisymposium it was submitted to must have been already added to the database through the respective links.

Once this information has been added to the database, it will become available at the respective drop-down menus.

An example of the whole process is shown in the steps below.

*Step 1: Add author (Skip if all co-authors have published at least one paper with the CRTC group):
[[File:Guid-publ-4.png | 400px]]

Simply fill in the fields requeted: [[File:Guid-publ-5.png | 400px]]

*Step 2: Add venue/minisymposium (Skip if CRTC group already published there before):
[[File:Guid-publ-6.png | 400px]]

Simply fill in the fields requested as before: [[File:Guid-publ-7.png | 400px]]

*Step 3: Add paper:
[[File:Guid-publ-8.png | 400px]]

Select the venue type (Conference or Journal): [[File:Guid-publ-9.png | 400px]]

Simply fill in the fields requested and choose the respective authors and venue from the drop-down menus:

[[File:Guid-publ-10.png | 800px]]

How to update publications page

2023-02-10T00:37:08Z

Pthomadakis:

== Guide to updating CRTC wiki publication page ==
This is a short guide on how to update the CRTC wiki publication page. The publication page lists all publications of the CRTC group and is maintained (like the whole wiki) from the CS department's IT group. If you have any issues or questions, please contact: root@cs.odu.edu.

== Prerequisites ==
In order to be able to access the page to modify the publications list you will need the following prerequisites:
# Have a CS account. All CS students are given such an account when first joining the department.
# Connect to the [https://systems.cs.odu.edu/VPN_Services CS VPN] or use a machine located in the CS department.
## Note that connecting to ODU's VPN will not work for this.
# Be a member of the CRTC group (email root@cs.odu.edu to request this)

== Accessing the publications' administrative page ==
First access the [[Publications]] page. On the top right you should see red link reading "Authenticated Users" as in the picture below:

[[File:Guide-publ-1.png|800px]]

Click on the link and you should be directed to a login page, like below. Enter you CS account credentials and sign in.

(If you get an error, make sure you are connected to the CS department network or VPN.)

[[File:Guide-publ-2.png|800px]]

If everything was successful, you should see the following html page:

[[File:Guide-publ-3.png|800px]]

== Updating the publications page ==
Users can perform the following updates to the page:
# Add a new paper/author/venue/minisymposium
# Modify an existing paper/author/venue/minisymposium
# Remove an existing paper/author/venue/minisymposium

=== Adding a new paper ===
In order to add a new paper, the paper's author(s) and the venue/minisymposium it was submitted to must have been already added to the database through the respective links.

Once this information has been added to the database, it will become available at the respective drop-down menus.

An example of the whole process is shown in the steps below.

Step 1: Add author (Skip if all co-authors have published at least one paper with the CRTC group):
[[File:Guid-publ-4.png | 400px]]

Simply fill in the fields requeted: [[File:Guid-publ-5.png | 400px]]

Step 2: Add venue/minisymposium (Skip if CRTC group already published there before):
[[File:Guid-publ-6.png | 400px]]

Simply fill in the fields requested as before: [[File:Guid-publ-7.png | 400px]]

Step 2: Add paper:
[[File:Guid-publ-8.png | 400px]]

Select the venue type (Conference or Journal): [[File:Guid-publ-9.png | 400px]]

Simply fill in the fields requested and choose the respective authors and venue from the drop-down menus:

[[File:Guid-publ-10.png | 800px]]

File:Guid-publ-10.png

2023-02-10T00:36:29Z

Pthomadakis:

File:Guid-publ-9.png

2023-02-10T00:36:09Z

Pthomadakis:

File:Guid-publ-8.png

2023-02-10T00:35:53Z

Pthomadakis:

File:Guid-publ-7.png

2023-02-10T00:27:24Z

Pthomadakis:

File:Guid-publ-6.png

2023-02-10T00:25:54Z

Pthomadakis:

File:Guid-publ-5.png

2023-02-10T00:21:20Z

Pthomadakis:

File:Guid-publ-4.png

2023-02-10T00:19:01Z

Pthomadakis:

How to update publications page

2023-02-09T23:58:46Z

Pthomadakis: /* Accessing the publication's administrative page */

File:Guide-publ-3.png

2023-02-09T23:57:02Z

Pthomadakis:

How to update publications page

2023-02-09T23:55:18Z

Pthomadakis:

File:Guide-publ-2.png

2023-02-09T23:52:58Z

Pthomadakis:

File:Guide-publ-1.png

2023-02-09T23:44:06Z

Pthomadakis:

How to update publications page

2023-02-09T23:43:09Z

Pthomadakis:

How to update publications page

2023-02-09T23:39:33Z

Pthomadakis: Created page with "== Guide to updating CRTC wiki publication page == This is a short guide on how to update the CRTC wiki publication page. The publication page lists all publications of the CR..."

Collaborations

2023-02-08T16:57:26Z

Pthomadakis:

The current focus for the CRTC group is on quantum computing, hybrid classical/quantum computing with emphasis on domain-specific languages (a collaboration with VT's Prof. Nikolopoulos), and intelligent runtime software systems for exascale-era heterogeneous architectures. CRTC is open to collaborations in those areas.

The rest of the page lists past (and before the pandemic) collaborations and research activities between CRTC and the broader scientific community from MIT, Harvard Medical School, Stony Brook University, and RWTH Aachen University, Germany.

== Current Collaborations ==

Quantum Computing [https://crtc.cs.odu.edu/Quantum_Computing_Academy_for_Southeast_Virginia]

== Past Collaborations ==

CRTC with SBU (Prof. Jim Jiao): Scalable Geometry Abstractions for Extreme-Scale Parallel Mesh Generation

CRTC with DoE/ANL (Dr. Pete Beckman): Multi-Layered Runtime System on Exascale OS (ARGO)

CRTC with NIA (Hiro Nishikawa) and SBU (Prof. Jim Jiao): mesh generation for high-order convergence schemes

CRTC with Center for Computational Engineering Science at RWTH Aachen University, Germany (Prof. Marek Bern): Parallel Space-Time Finite Element Simulations

CRTC with Medical School at SBU (Prof. Barry Lieber and Dr. Chader Sadasivan): Prediction of Long Term Efficacy of Cerebral Aneurysm Bypass by Flow Diversion

CRTC with Medical School at Harvard (Prof. Ron Kikinis, Prof. Alex Golby, and Dr. Sarah Frisken): Deformable Registration for Brain Tumor Resection

CRTC with School of Medicine at VCU (Prof. Kathy Holloway): Deformable Registration for Deep Brain Stimulation

CRTC with NYU (Prof. Daniele Panozzo): Parallel swappable mesh operations project

CRTC with NCSA/UIUC (Dr. Andriy Kot, NCSA, UIUC): Multi-Layered Runtime System

CRTC with Queen's Univ. Belfast, UK (Prof. Nikolopoulos): Power-Aware High-Performance Computing Systems

CRTC with MIT (Prof. David Darmofal): Parallel Error-based Grid Adaptation

CRTC with MSU (Prof. David Marcum): Modifications of AFLR for parallel mesh generation

CRTC with NASA/LaRC (Dr. Mike Park): Parallel Adaptive Anisotropic Mesh Generation

Quantum Computing Academy for Southeast Virginia

2023-02-08T16:52:40Z

Pthomadakis: Created blank page