PDR.PODM Distributed Memory

Issues

• No reuse of leaves refined by worker nodes. The picture below shows the issue. Two neighbour leaves (0,1) each refined as the main leaf (0 top, 1 bottom) but not refined as a neighor.

• Current algorithm uses neighbour traversal to distribute cells to octree leaving some cells out in some cases. Such a case can happen when a cell is part of an octree leaf based on its circumcenter but

it does not have any neighbour in the same leaf.

• During unpacking the incident cell for each vertex is not set correctly. Specifically, in the case that the initial incident cell is not part of the working unit (Leaf + LVL.1 Neighbours) and thus is not local,

it is set to the infinite cell. This causes PODM to crash randomly for some cases.

• Another issue comes from the way global IDs are updated for each cell's neighbors' IDs. The code that updates the cell's connectivity using global IDs takes the neighbor's pointer, retrieves its global ID and

updates the neighborID field. However, when the neighbour is part of another work unit's leaf and is not local this pointer is NULL. In this case the neighborID field is wrongly reset to the infinite cell ID, which as result, deletes the connectivity information forever.

• The function that unpacks the required leaves before refinement does not discard duplicate vertices. Duplicate vertices will always be present since each leaf is packed and sent individually, and as a result,

neighbouring leaves will include the shared vertices. Because duplicate vertices are not handled, multiple vertex objects are created that are in fact the same point geometrically. Thus, two cells that share a common vertex could have pointers to two different vertex objects and, as a result, each cell views a different state about the same vertex.

Fixes

Interesting Findings

15 MPI ranks depth: 3

polling_times[1] = 62.977124

polling_times[2] = 81.895647

polling_times[3] = 79.853833

polling_times[4] = 121.689584

polling_times[5] = 67.187857

polling_times[6] = 77.674953

polling_times[7] = 110.503308

polling_times[8] = 63.927371

polling_times[9] = 63.174693

polling_times[10] = 77.656739

polling_times[11] = 84.785231

polling_times[12] = 86.950730

polling_times[13] = 67.230592

polling_times[14] = 70.842911

int_to_pointer_times[1] = 413.575395

int_to_pointer_times[2] = 393.641700

int_to_pointer_times[3] = 436.249681

int_to_pointer_times[4] = 388.363599

int_to_pointer_times[5] = 454.809466

int_to_pointer_times[6] = 464.654607

int_to_pointer_times[7] = 451.311534

int_to_pointer_times[8] = 417.156188

int_to_pointer_times[9] = 458.370492

int_to_pointer_times[10] = 443.813062

int_to_pointer_times[11] = 466.672172

int_to_pointer_times[12] = 405.445182

int_to_pointer_times[13] = 466.928191

int_to_pointer_times[14] = 427.620845

clear_element_lists_times[1] = 0.000762

clear_element_lists_times[2] = 0.000687

clear_element_lists_times[3] = 0.000647

clear_element_lists_times[4] = 0.000621

clear_element_lists_times[5] = 0.000627

clear_element_lists_times[6] = 0.000670

clear_element_lists_times[7] = 0.000724

clear_element_lists_times[8] = 0.000560

clear_element_lists_times[9] = 0.000868

clear_element_lists_times[10] = 0.002675

clear_element_lists_times[11] = 0.000628

clear_element_lists_times[12] = 0.000851

clear_element_lists_times[13] = 0.001053

clear_element_lists_times[14] = 0.001273

polling_all_times[1] = 140.294989

polling_all_times[2] = 171.339134

polling_all_times[3] = 158.837717

polling_all_times[4] = 230.522151

polling_all_times[5] = 151.359351

polling_all_times[6] = 168.926307

polling_all_times[7] = 236.011997

polling_all_times[8] = 145.662843

polling_all_times[9] = 159.411167

polling_all_times[10] = 171.122498

polling_all_times[11] = 194.755214

polling_all_times[12] = 186.417060

polling_all_times[13] = 152.213076

polling_all_times[14] = 162.141306

wait_in_queue_times1[1] = 2027.081317

wait_in_queue_times1[2] = 2004.238402

wait_in_queue_times1[3] = 2078.419998

wait_in_queue_times1[4] = 2007.550366

wait_in_queue_times1[5] = 1986.886886

wait_in_queue_times1[6] = 1951.253771

wait_in_queue_times1[7] = 1942.383973

wait_in_queue_times1[8] = 2005.419375

wait_in_queue_times1[9] = 1997.842771

wait_in_queue_times1[10] = 1997.673105

wait_in_queue_times1[11] = 2000.956408

wait_in_queue_times1[12] = 1974.561631

wait_in_queue_times1[13] = 1979.973832

wait_in_queue_times1[14] = 2072.972642

wait_in_queue_times2[1] = 3162.564690

wait_in_queue_times2[2] = 3152.861625

wait_in_queue_times2[3] = 3103.681840

wait_in_queue_times2[4] = 3093.594562

wait_in_queue_times2[5] = 3094.828513

wait_in_queue_times2[6] = 3068.933678

wait_in_queue_times2[7] = 3007.889167

wait_in_queue_times2[8] = 3148.827740

wait_in_queue_times2[9] = 3080.482366

wait_in_queue_times2[10] = 3088.332840

wait_in_queue_times2[11] = 3042.559130

wait_in_queue_times2[12] = 3121.678383

wait_in_queue_times2[13] = 3079.916276

wait_in_queue_times2[14] = 3124.285036

mesher_elements[1] = 2288798

mesher_elements[2] = 2536028

mesher_elements[3] = 2709194

mesher_elements[4] = 4263686

mesher_elements[5] = 2157859

mesher_elements[6] = 3134463

mesher_elements[7] = 4804167

mesher_elements[8] = 1975299

mesher_elements[9] = 2373811

mesher_elements[10] = 2721700

mesher_elements[11] = 1998762

mesher_elements[12] = 2323134

mesher_elements[13] = 4775266

mesher_elements[14] = 2789597

meshing_times[1] = 36.443444

meshing_times[2] = 37.558417

meshing_times[3] = 45.448564

meshing_times[4] = 40.453400

meshing_times[5] = 43.791719

meshing_times[6] = 40.182756

meshing_times[7] = 44.701021

meshing_times[8] = 39.561876

meshing_times[9] = 47.492666

meshing_times[10] = 46.783555

meshing_times[11] = 40.445111

meshing_times[12] = 39.889953

meshing_times[13] = 47.444813

meshing_times[14] = 35.083527

leaf_distribution_time[1] = 72.064785

leaf_distribution_time[2] = 67.755446

leaf_distribution_time[3] = 78.394198

leaf_distribution_time[4] = 72.630350

leaf_distribution_time[5] = 77.081779

leaf_distribution_time[6] = 78.999842

leaf_distribution_time[7] = 77.059384

leaf_distribution_time[8] = 72.617468

leaf_distribution_time[9] = 69.875876

leaf_distribution_time[10] = 71.842582

leaf_distribution_time[11] = 73.979892

leaf_distribution_time[12] = 71.419246

leaf_distribution_time[13] = 72.170305

leaf_distribution_time[14] = 72.779034

total_num_mesher_elements = 43167663

initial_number_of_elements = 2315899

initial_io_time = 9.116445

initial_allocation_time = 0.004795

initial_meshing_time = 3.937135

initial_mesh_distribution_time = 2.665803

get_leaf_elements_time = 0.331813

pointer_to_int_time = 0.138809

pack_data_time = 0.064877

send_data_time = 0.191445

sub_total_time = 3831.977334

max_meshing_time = 47.492666

max_io_time = 9.596057

max_allocation_time = 9.961161

max_recv_time = 0.000000

max_polling_time = 121.689584

max_int_to_pointer_time = 466.928191

max_clear_element_lists_time = 0.002675

max_polling_all_time = 236.011997

max_wait_in_queue_time1 = 2078.419998

max_wait_in_queue_time2 = 3162.564690

elements_per_second = 7311014.065501

elements_per_second1 = 588219.279339

elements_per_second2 = 20765032.278210

master_idle_time = 3831.174962

surface_box_time = 9.046863

edt_time = 0.566338

octree_time = 0.122400

• How many times migrated a leaf (main and neighbor) ?
• How many times a leaf is scheduled ? (as main leaf)
• Packed size for each transmission (MB)/time?
• Meshing time each time it was main
• Initial meshing time
• Per process :
  • Meshing
  • Preprocessing (pack/unpack)
  • waiting time
• Waiting time per process ?
• Elements before and after each time it was used (main/neighbor) ?

leaf_id	init elements	num_migrations	num_scheduled	num_refined	max_size_migration (KB)	min_size_migration (KB)	avg_size_migration (KB)	max_time_migration (sec)	min_time_migration (sec)	avg_time_migration (sec)
0	5140	1	0	1
1	4921	1	0	1
2	4880	1	0	1
3	4714	1	0	1
4	5044	1	0	1
5	4668	1	0	1
6	4741	1	0	1
7	4470	1	1	0
8	4854	1	0	1
9	4733	1	0	1
10	4628	1	0	1
11	4719	1	0	1
12	4747	1	0	1
13	4692	1	0	1
14	4484	1	0	1
15	4439	1	0	1
16	4847	2	0	2	415	415	415.0	0.000405	0.000405	0.000405
17	4597	6	0	6	394	394	394.0	0.015056	0.000377	0.003564199999999999
18	4875	4	0	4	417	417	417.0	0.030162	0.000719	0.016067
19	4704	9	0	9	405	404	404.0	0.019694	0.000544	0.007640875
20	4611	2	0	2	395	395	395.0	0.000401	0.000401	0.000401
21	4515	10	0	10	387	387	387.0	0.023126	0.000373	0.0037203333333333324
22	4715	7	0	7	404	404	404.0	0.016818	0.000444	0.006300166666666667
23	4557	21	1	20	403	394	400.0	0.04192	0.000398	0.01055885
24	4570	10	0	10	392	392	392.0	0.030502	0.000373	0.00599111111111111
25	4666	13	0	13	400	400	400.0	0.047174	0.00068	0.01136525
26	4746	15	0	15	531	407	510.0	0.049097	0.00055	0.00914507142857143
27	4868	17	0	17	621	417	593.0	0.052032	0.000761	0.014502000000000001
28	4481	19	0	19	384	384	384.0	0.019985	0.000293	0.0026930555555555557
29	4509	26	0	26	387	387	387.0	0.034138	0.000437	0.006108079999999999
30	4590	34	4	30	10613	395	8870.0	1.144701	0.000322	0.13719339393939395
31	4752	39	4	35	20063	415	15899.0	2.210566	0.000639	0.17472076315789475
32	4904	1	0	1
33	4767	1	0	1
34	4667	1	0	1
35	4510	1	0	1
36	4929	1	0	1
37	4771	1	0	1
38	4754	1	0	1
39	4459	1	0	1
40	4686	1	0	1
41	4609	1	0	1
42	4489	1	0	1
43	4465	1	0	1
44	4641	1	0	1
45	4707	1	0	1
46	4539	1	0	1
47	4540	1	0	1
48	4813	3	0	3	412	412	412.0	0.000928	0.000405	0.0006665
49	4425	17	0	17	380	380	380.0	0.012858	0.000314	0.0024941250000000002
50	4620	11	0	11	396	396	396.0	0.022816	0.000463	0.006890599999999999
51	4525	32	0	32	388	388	388.0	0.061139	0.000387	0.00679132258064516
52	4762	4	0	4	408	408	408.0	0.046096	0.018953	0.031662
53	4583	19	0	19	393	393	393.0	0.018943	0.000514	0.006347722222222222
54	4628	14	0	14	397	397	397.0	0.041793	0.000596	0.012489384615384614
55	4521	38	1	37	394	389	393.0	0.040649	0.000447	0.009107486486486486
56	4533	33	0	33	388	388	388.0	0.029611	0.000359	0.004132593750000001
57	4476	46	0	46	384	384	384.0	0.038396	0.000364	0.006301111111111111
58	4603	54	4	50	9376	395	8107.0	0.805219	0.000388	0.08217690566037736
59	4798	63	5	58	22391	411	21441.0	2.142464	0.000345	0.19465403225806455
60	4490	36	0	36	385	385	385.0	0.023103	0.000337	0.005721457142857143
61	4512	45	0	45	387	387	387.0	0.038251	0.000424	0.00701115909090909
62	4582	60	6	54	9883	393	8875.0	1.112903	0.000483	0.10731481355932204
63	4699	67	7	60	21908	403	21305.0	1.537622	0.000399	0.1713687424242424
64	4818	1	0	1
65	4816	2	0	2	412	412	412.0	0.00045	0.00045	0.00045
66	4626	1	0	1
67	4727	2	0	2	405	405	405.0	0.000464	0.000464	0.000464
68	4623	1	0	1
69	4837	2	0	2	414	414	414.0	0.000473	0.000473	0.000473
70	4454	1	1	0
71	4429	2	0	2	381	381	381.0	0.000438	0.000438	0.000438
72	4896	1	0	1
73	5117	1	0	1
74	4669	1	0	1
75	4841	1	0	1
76	4804	1	0	1
77	4988	1	0	1
78	4587	1	1	0
79	4616	1	0	1
80	4746	15	0	15	407	407	407.0	0.018839	0.000579	0.010480928571428572
81	4600	17	0	17	394	394	394.0	0.033308	0.000398	0.0047910625
82	4909	22	0	22	627	429	571.0	0.051229	0.000809	0.014659999999999996
83	4768	25	0	25	607	408	570.0	0.054998	0.000675	0.013141125000000002
84	4596	31	0	31	394	394	394.0	0.018374	0.000358	0.005900033333333332
85	4484	33	0	33	385	385	385.0	0.040269	0.000366	0.00621278125
86	4879	45	4	41	21582	818	14618.0	0.856606	0.000981	0.086496
87	4722	54	6	48	19733	413	15960.0	1.694223	0.000403	0.22801369811320754
88	4637	12	0	12	397	397	397.0	0.0184	0.000352	0.006041727272727273
89	4901	6	0	6	419	419	419.0	0.024881	0.000641	0.0060704
90	4698	19	0	19	527	515	522.0	0.032079	0.000546	0.012114944444444442
91	4873	9	0	9	418	418	418.0	0.011423	0.000847	0.004115875
92	4534	24	0	24	388	388	388.0	0.021887	0.00037	0.005878608695652173
93	4708	11	0	11	403	403	403.0	0.015462	0.000409	0.0031311999999999998
94	4609	42	5	37	10745	520	10079.0	0.66151	0.001178	0.13872612195121953
95	4558	22	0	22	408	391	405.0	0.031273	0.000437	0.007883619047619047
96	4594	1	0	1
97	4676	3	0	3	401	401	401.0	0.000473	0.000463	0.000468
98	4465	1	0	1
99	4467	3	0	3	384	384	384.0	0.00045	0.000442	0.000446
100	4726	1	0	1
101	4658	2	0	2	399	399	399.0	0.000494	0.000494	0.000494
102	4508	1	0	1
103	4525	2	0	2	388	388	388.0	0.00049	0.00049	0.00049
104	4718	2	0	2	405	405	405.0	0.000476	0.000476	0.000476
105	4967	2	0	2	425	425	425.0	0.00051	0.00051	0.00051
106	4481	2	0	2	385	385	385.0	0.000452	0.000452	0.000452
107	4571	2	0	2	392	392	392.0	0.000471	0.000471	0.000471
108	4786	1	0	1
109	4823	1	0	1
110	4574	1	1	0
111	4680	1	0	1
112	4574	51	0	51	392	392	392.0	0.047324	0.000392	0.005151880000000001
113	4489	52	0	52	385	385	385.0	0.060409	0.000312	0.005352901960784314
114	4736	72	4	68	23594	414	17669.0	1.006988	0.000343	0.08559649295774648
115	4735	83	7	76	22078	406	18987.0	2.747555	0.000391	0.22780202439024386
116	4481	47	0	47	385	385	385.0	0.03056	0.000375	0.005288021739130433
117	4531	46	0	46	389	389	389.0	0.043212	0.000331	0.006133044444444443
118	4789	69	7	62	20996	418	17145.0	1.757034	0.000356	0.1959893823529412
119	4608	81	6	75	21187	396	18847.0	2.456932	0.000378	0.22400851249999998
120	4518	36	0	36	387	387	387.0	0.028003	0.000351	0.004179828571428572
121	4695	17	0	17	402	402	402.0	0.036991	0.000394	0.005922875
122	4632	63	5	58	8865	463	8604.0	0.732422	0.000591	0.10695459677419356
123	4671	33	0	33	401	400	401.0	0.026674	0.000354	0.007890125000000001
124	4484	33	0	33	385	385	385.0	0.036508	0.000326	0.006485437499999999
125	4649	16	0	16	399	399	399.0	0.025154	0.00034	0.006072800000000001
126	4461	60	5	55	6350	5843	6265.0	0.527465	0.008833	0.08390650847457627
127	4697	34	0	34	403	403	403.0	0.033889	0.00043	0.006612545454545454
128	5001	5	0	5	429	429	429.0	0.017186	0.000428	0.00848025
129	4783	14	0	14	499	491	494.0	0.071658	0.000541	0.015102230769230767
130	4770	5	0	5	408	408	408.0	0.036432	0.000566	0.012671
131	4780	12	0	12	467	412	438.0	0.03341	0.000402	0.012121272727272725
132	4577	10	0	10	392	392	392.0	0.020947	0.000367	0.006680222222222223
133	4618	31	3	28	5218	4818	4942.0	0.351238	0.006392	0.0750992
134	4663	10	0	10	399	399	399.0	0.037043	0.000432	0.009523222222222221
135	4555	28	1	27	3236	470	1796.0	0.107946	0.000541	0.023689148148148147
136	4842	24	0	24	833	421	538.0	0.08271	0.000418	0.013151782608695652
137	4958	27	0	27	840	433	655.0	0.063228	0.000467	0.014496038461538463
138	4799	22	0	22	813	412	730.0	0.055436	0.000558	0.007774952380952382
139	4911	26	0	26	842	421	739.0	0.123103	0.000465	0.025620559999999997
140	4801	55	1	54	35078	764	10374.0	2.521878	0.000641	0.10560070370370372
141	4818	62	2	60	35230	814	21732.0	1.25825	0.00093	0.14713504918032783
142	4742	53	4	49	33400	415	27391.0	1.310808	0.000573	0.2354048461538462
143	4829	61	4	57	35186	421	26418.0	3.787715	0.000825	0.30994716666666655
144	4887	2	0	2	418	418	418.0	0.014348	0.014348	0.014348
145	4587	8	0	8	393	393	393.0	0.040848	0.000721	0.010354142857142857
146	5103	1	0	1
147	4833	3	0	3	414	414	414.0	0.016732	0.000898	0.008815
148	4692	4	0	4	402	402	402.0	0.033889	0.000556	0.016683
149	4557	16	1	15	391	391	391.0	0.018683	0.000472	0.004319266666666668
150	4964	1	0	1
151	4674	6	0	6	400	400	400.0	0.028551	0.000983	0.013714399999999998
152	4799	16	0	16	493	411	482.0	0.037937	0.000534	0.0101036
153	4674	21	0	21	678	661	674.0	0.09922	0.000717	0.018367699999999994
154	4845	7	0	7	415	415	415.0	0.00591	0.00042	0.002009
155	4861	11	0	11	417	417	417.0	0.032705	0.000533	0.0148554
156	4616	37	2	35	10955	396	9495.0	0.84543	0.000558	0.14547416666666665
157	4896	47	4	43	29802	814	28902.0	1.4604	0.005043	0.23156036956521733
158	4867	16	0	16	418	417	418.0	0.036092	0.000502	0.011479066666666666
159	4743	24	0	24	514	415	507.0	0.052268	0.000587	0.017611521739130432
160	4728	20	0	20	406	406	406.0	0.020009	0.000408	0.008198631578947367
161	4539	53	3	50	1305	410	954.0	0.092867	0.000431	0.013970576923076926
162	4650	22	0	22	398	398	398.0	0.027525	0.000387	0.00920314285714286
163	4462	51	2	49	1281	384	1026.0	0.173192	0.000357	0.012442220000000002
164	4741	25	0	25	407	407	407.0	0.027902	0.000388	0.006532625
165	4557	61	3	58	3903	486	1465.0	0.084045	0.000469	0.012070450000000003
166	4749	28	0	28	408	408	408.0	0.036397	0.000423	0.005662185185185185
167	4547	62	6	56	5675	393	4951.0	0.556404	0.000379	0.06199437704918033
168	4811	90	5	85	33750	417	18049.0	2.345832	0.000446	0.1369888876404494
169	4866	97	2	95	35510	426	18884.0	2.747791	0.000411	0.22452585416666662
170	4733	89	3	86	34065	406	28104.0	1.726143	0.00034	0.19082899999999994
171	5014	95	7	88	35596	430	23707.0	3.052361	0.000409	0.22742587234042558
172	4820	96	1	95	34465	421	7042.0	0.218778	0.000472	0.03614748421052631
173	4922	101	2	99	35520	430	19368.0	3.092187	0.000365	0.15891331
174	4901	96	5	91	35540	420	26870.0	4.254684	0.000395	0.25204304210526324
175	5132	101	4	97	35517	439	27090.0	2.22364	0.000619	0.18806341
176	4672	13	0	13	400	400	400.0	0.029952	0.000399	0.009411
177	4585	35	0	35	394	393	394.0	0.032544	0.00043	0.0068634117647058835
178	4895	4	0	4	419	419	419.0	0.020005	0.000999	0.009127999999999999
179	4585	14	0	14	393	393	393.0	0.036209	0.000378	0.00787546153846154
180	4775	17	0	17	410	410	410.0	0.028318	0.000435	0.00724625
181	4460	42	3	39	555	387	538.0	0.052798	0.000598	0.012619195121951218
182	4883	6	0	6	418	418	418.0	0.032337	0.00095	0.010706400000000001
183	4655	19	0	19	398	398	398.0	0.029974	0.000575	0.009053555555555558
184	4798	67	3	64	20596	411	18456.0	1.975423	0.000372	0.24117915151515149
185	4741	77	6	71	35275	803	34594.0	2.824109	0.001208	0.24719185526315796
186	4787	31	0	31	517	410	493.0	0.039667	0.000379	0.0081323
187	4709	43	0	43	817	412	803.0	0.075649	0.000401	0.019051404761904762
188	4792	71	5	66	25521	804	23645.0	1.902453	0.001311	0.2771063428571429
189	4848	78	7	71	35180	33582	34843.0	1.660215	0.052097	0.22248788311688308
190	4672	35	0	35	539	410	519.0	0.048332	0.000468	0.014152470588235292
191	4668	44	0	44	813	787	805.0	0.079299	0.00091	0.01713302325581395
192	4925	32	0	32	842	815	827.0	0.066662	0.001037	0.017131677419354837
193	4824	35	0	35	838	422	565.0	0.070966	0.000469	0.012869823529411765
194	4929	33	0	33	838	431	545.0	0.0761	0.000461	0.010275656250000003
195	5073	37	0	37	857	435	680.0	0.080225	0.000457	0.013417944444444446
196	5103	68	2	66	35249	33674	34415.0	2.730947	0.049441	0.22863904477611938
197	4888	77	4	73	35319	816	14284.0	1.240214	0.001182	0.10229803947368421
198	4840	71	2	69	35261	808	11567.0	1.443657	0.000858	0.09159465714285712
199	4888	80	4	76	35230	428	18831.0	2.501586	0.00039	0.20689413924050631
200	4728	27	0	27	809	414	671.0	0.034728	0.000428	0.008012884615384614
201	5041	13	0	13	441	432	437.0	0.034372	0.000528	0.007971250000000001
202	4847	28	0	28	808	420	748.0	0.065262	0.000702	0.014953111111111108
203	4908	14	0	14	427	421	425.0	0.021254	0.000447	0.004600615384615384
204	4647	60	4	56	34849	404	19815.0	1.498058	0.000727	0.1357884576271186
205	4766	33	0	33	802	408	598.0	0.057507	0.000471	0.008826031249999998
206	4697	62	4	58	32343	640	22419.0	1.908858	0.000594	0.2163177540983606
207	4657	34	0	34	660	399	549.0	0.045163	0.00034	0.008172454545454544
208	4784	25	0	25	665	414	559.0	0.098567	0.000493	0.010815208333333333
209	4789	27	0	27	652	415	587.0	0.04927	0.000755	0.008286576923076924
210	4903	15	0	15	420	420	420.0	0.033982	0.000423	0.008029357142857144
211	4868	17	0	17	418	418	418.0	0.02337	0.000375	0.0108668125
212	4946	56	5	51	29192	432	13847.0	1.780761	0.00083	0.21104609090909096
213	4782	57	6	51	25840	613	15069.0	1.086749	0.000654	0.1849354285714286
214	4662	33	0	33	496	400	435.0	0.068018	0.000394	0.013121562499999998
215	4705	34	0	34	424	404	415.0	0.034822	0.000379	0.009416333333333332
216	4791	20	0	20	470	463	468.0	0.032153	0.000722	0.008768894736842102
217	4913	10	0	10	421	421	421.0	0.032853	0.000919	0.008131888888888888
218	4898	12	0	12	420	420	420.0	0.017379	0.000882	0.00776381818181818
219	5131	6	0	6	439	439	439.0	0.039579	0.000904	0.016645600000000003
220	4633	43	6	37	6194	5740	6012.0	0.663521	0.00775	0.09799988095238095
221	4604	22	0	22	395	395	395.0	0.031822	0.000335	0.007726523809523809
222	4693	25	0	25	402	402	402.0	0.042673	0.000347	0.011762958333333335
223	4852	11	0	11	416	416	416.0	0.032192	0.000376	0.0140429
224	5129	105	2	103	35629	834	10904.0	1.313984	0.000826	0.07052285576923077
225	4807	118	3	115	35596	421	13050.0	1.741935	0.000428	0.08700684615384614
226	5067	109	3	106	35571	442	10643.0	1.891894	0.000451	0.07780433333333334
227	5074	122	3	119	35594	434	13368.0	1.853441	0.00046	0.12165690909090908
228	5124	102	1	101	35642	830	4125.0	1.295953	0.000978	0.035116
229	4867	118	4	114	35667	425	10893.0	2.992325	0.000421	0.15099828205128202
230	5084	106	2	104	35719	443	5771.0	1.426547	0.000462	0.036069704761904756
231	5047	123	4	119	35571	433	17022.0	1.856241	0.000405	0.17661686885245903
232	4688	91	8	83	30024	407	19189.0	1.538931	0.00042	0.16070555555555555
233	4685	50	0	50	466	402	438.0	0.033421	0.000364	0.004877795918367347
234	4849	94	7	87	27433	420	21737.0	2.013652	0.000462	0.21656583870967744
235	4749	51	0	51	431	407	424.0	0.038251	0.000408	0.007048679999999999
236	4734	88	6	82	26492	636	19175.0	2.213252	0.000884	0.20057063218390805
237	4618	51	0	51	399	396	397.0	0.039464	0.000356	0.006376559999999999
238	4849	90	6	84	26933	420	25067.0	2.684458	0.000411	0.2438207528089887
239	4611	51	0	51	398	396	398.0	0.034461	0.000375	0.009286019999999999
240	4792	87	4	83	35199	418	27784.0	2.947441	0.000439	0.24737167441860455
241	4888	88	8	80	34145	422	29209.0	3.038667	0.000449	0.28930131034482776
242	4888	54	0	54	834	419	696.0	0.060277	0.000397	0.013741905660377356
243	4770	53	0	53	732	409	662.0	0.057123	0.000396	0.008603365384615385
244	4858	83	7	76	35283	808	31108.0	2.79696	0.001199	0.29113229268292684
245	4740	86	7	79	34881	414	31750.0	2.356987	0.00043	0.28524752941176473
246	4802	50	0	50	833	420	753.0	0.078215	0.000421	0.016389816326530613
247	4793	49	0	49	794	410	743.0	0.073413	0.000413	0.012781625000000003
248	4579	65	5	60	8885	470	8654.0	0.668934	0.000592	0.1190871875
249	4770	33	0	33	409	409	409.0	0.03466	0.000364	0.00937528125
250	4775	37	0	37	410	410	410.0	0.040477	0.000378	0.007337555555555554
251	4974	16	0	16	426	426	426.0	0.042725	0.000416	0.010310999999999999
252	4625	62	5	57	11588	505	11335.0	0.897079	0.000747	0.14752404918032785
253	4691	32	0	32	402	402	402.0	0.037126	0.000361	0.010469322580645162
254	4617	34	0	34	400	396	399.0	0.026872	0.000357	0.008994151515151514
255	4872	15	0	15	417	417	417.0	0.048537	0.000463	0.014429857142857145
256	5234	1	0	1
257	4651	1	0	1
258	4671	1	0	1
259	4538	1	1	0
260	4869	2	0	2	418	418	418.0	0.000393	0.000393	0.000393
261	4671	2	0	2	400	400	400.0	0.000377	0.000377	0.000377
262	4724	2	0	2	405	405	405.0	0.000385	0.000385	0.000385
263	4521	2	0	2	388	388	388.0	0.000367	0.000367	0.000367
264	4664	1	0	1
265	4673	1	0	1
266	4515	1	0	1
267	4513	1	0	1
268	4666	2	0	2	400	400	400.0	0.000386	0.000386	0.000386
269	4558	2	0	2	391	391	391.0	0.000835	0.000835	0.000835
270	4510	2	0	2	386	386	386.0	0.000364	0.000364	0.000364
271	4569	2	0	2	392	392	392.0	0.000979	0.000979	0.000979
272	4784	9	0	9	410	410	410.0	0.031732	0.000407	0.0154075
273	4564	26	0	26	391	391	391.0	0.029549	0.000406	0.0072178
274	4513	18	0	18	387	387	387.0	0.022865	0.0005	0.007096
275	4623	43	2	41	1022	475	887.0	0.095425	0.000505	0.0129385
276	4625	13	0	13	396	396	396.0	0.01638	0.000374	0.0026061666666666664
277	4448	30	0	30	381	381	381.0	0.022676	0.000334	0.007312275862068967
278	4669	22	0	22	400	400	400.0	0.031952	0.000445	0.007112190476190477
279	4555	51	5	46	2084	392	1823.0	0.18679	0.000434	0.02254728
280	4579	44	0	44	393	393	393.0	0.030298	0.000337	0.007696604651162791
281	4548	49	0	49	390	390	390.0	0.038349	0.000415	0.007028229166666667
282	4609	68	5	63	15042	401	10994.0	1.12152	0.000535	0.11490973134328362
283	4522	71	5	66	22051	395	14898.0	1.732319	0.00041	0.18782437142857147
284	4517	47	0	47	387	387	387.0	0.026073	0.000347	0.006048499999999999
285	4468	49	0	49	383	383	383.0	0.051758	0.000349	0.00749775
286	4629	78	6	72	17070	396	13240.0	1.05898	0.000353	0.1782791558441558
287	4560	83	6	77	24214	391	19350.0	1.801677	0.000539	0.15831074390243902
288	4873	1	0	1
289	4669	1	0	1
290	4675	1	0	1
291	4513	1	1	0
292	4929	1	0	1
293	5066	1	0	1
294	4898	1	0	1
295	4664	1	0	1
296	4691	1	0	1
297	4692	1	0	1
298	4497	1	0	1
299	4513	1	0	1
300	5283	1	0	1
301	4870	1	0	1
302	4666	1	0	1
303	4626	1	0	1
304	4698	10	0	10	403	403	403.0	0.022218	0.000496	0.007235666666666666
305	4364	22	0	22	374	374	374.0	0.025065	0.000348	0.008061333333333334
306	4743	15	0	15	407	407	407.0	0.042199	0.000556	0.010222642857142857
307	4534	36	4	32	3796	3500	3751.0	0.339467	0.004786	0.04757462857142857
308	4876	5	0	5	418	418	418.0	0.001252	0.000553	0.00085425
309	4606	11	0	11	394	394	394.0	0.027692	0.000346	0.004638000000000001
310	4846	8	0	8	415	415	415.0	0.032326	0.00048	0.006603
311	4659	19	0	19	446	441	445.0	0.031849	0.000426	0.006205500000000001
312	4480	34	0	34	385	385	385.0	0.039385	0.000337	0.008765848484848484
313	4500	37	0	37	386	386	386.0	0.031724	0.000372	0.007139055555555557
314	4620	57	6	51	17655	516	16870.0	1.384116	0.000595	0.2914791428571429
315	4704	62	6	56	24510	23497	24332.0	3.002376	0.033815	0.30594893442622956
316	4647	17	0	17	398	398	398.0	0.02186	0.000347	0.0048635625
317	4652	20	0	20	399	399	399.0	0.024712	0.000374	0.005520368421052632
318	4784	30	0	30	621	412	603.0	0.049087	0.000383	0.009207137931034483
319	4775	36	0	36	702	689	699.0	0.073271	0.000809	0.02001354285714286
320	4667	1	0	1
321	4609	2	0	2	395	395	395.0	0.000479	0.000479	0.000479
322	4479	1	1	0
323	4526	2	0	2	388	388	388.0	0.00048	0.00048	0.00048
324	4683	2	0	2	402	402	402.0	0.000982	0.000982	0.000982
325	4626	3	0	3	397	397	397.0	0.000728	0.000455	0.0005915
326	4427	2	0	2	380	380	380.0	0.005516	0.005516	0.005516
327	4560	3	0	3	391	391	391.0	0.000854	0.000505	0.0006795
328	4695	1	0	1
329	4812	1	0	1
330	4414	1	0	1
331	4847	1	0	1
332	4680	1	0	1
333	5107	1	0	1
334	4459	1	0	1
335	4684	1	0	1
336	4476	47	0	47	384	384	384.0	0.050207	0.000405	0.006834717391304347
337	4530	45	0	45	389	389	389.0	0.039705	0.000359	0.006810113636363635
338	4679	73	1	72	20668	790	13127.0	0.885686	0.000988	0.1188362361111111
339	4551	80	4	76	22115	398	15998.0	1.533126	0.000373	0.18685337974683539
340	4438	46	0	46	381	381	381.0	0.034865	0.000346	0.005006955555555557
341	4535	45	0	45	389	389	389.0	0.045882	0.000314	0.005172159090909092
342	4583	83	5	78	25585	401	16405.0	2.383814	0.000477	0.17421218292682927
343	4616	88	5	83	23634	396	20156.0	1.244632	0.000397	0.13510183908045975
344	4405	31	0	31	378	378	378.0	0.019572	0.00033	0.005825899999999999
345	4587	16	0	16	394	394	394.0	0.037612	0.000338	0.006420733333333334
346	4625	59	5	54	11188	492	10465.0	0.971001	0.00074	0.15938786206896552
347	4681	32	0	32	404	402	403.0	0.044012	0.000363	0.008428870967741937
348	4516	30	0	30	387	387	387.0	0.018138	0.000428	0.005765896551724139
349	4644	16	0	16	398	398	398.0	0.028985	0.000383	0.0034938666666666667
350	4576	61	5	56	14406	564	14023.0	0.8327	0.000917	0.1307434
351	4634	32	0	32	445	400	442.0	0.030715	0.000381	0.010195903225806451
352	4695	1	0	1
353	4604	2	0	2	394	394	394.0	0.000634	0.000634	0.000634
354	4535	1	1	0
355	4542	2	0	2	390	390	390.0	0.000964	0.000964	0.000964
356	4827	1	0	1
357	4861	2	0	2	417	417	417.0	0.00103	0.00103	0.00103
358	4647	1	0	1
359	4644	2	0	2	398	398	398.0	0.001944	0.001944	0.001944
360	4732	1	0	1
361	4910	1	0	1
362	4470	1	1	0
363	4737	1	0	1
364	4920	1	0	1
365	5125	1	0	1
366	4627	1	0	1
367	4884	1	0	1
368	4597	35	0	35	394	394	394.0	0.022988	0.000372	0.007171117647058824
369	4465	34	0	34	383	383	383.0	0.018807	0.000364	0.005009666666666667
370	4697	64	7	57	26323	698	24090.0	1.030364	0.001276	0.1931083333333333
371	4641	65	5	60	23805	642	22240.0	1.984989	0.000734	0.21763387499999998
372	4664	19	0	19	400	400	400.0	0.055555	0.000399	0.005936333333333333
373	4694	19	0	19	402	402	402.0	0.026396	0.00044	0.005943944444444444
374	4787	37	0	37	726	416	680.0	0.056771	0.000655	0.012783333333333336
375	4809	37	0	37	697	418	678.0	0.061221	0.000786	0.010880361111111111
376	4483	21	0	21	385	385	385.0	0.021498	0.000383	0.007484349999999999
377	4677	11	0	11	401	401	401.0	0.014198	0.000361	0.0026276
378	4587	45	5	40	15751	15109	15656.0	1.321379	0.022048	0.22873999999999992
379	4732	22	0	22	504	496	502.0	0.05338	0.000536	0.01617090476190476
380	4727	11	0	11	405	405	405.0	0.008131	0.000495	0.0025351
381	4908	6	0	6	420	420	420.0	0.017446	0.000564	0.0041475999999999996
382	4730	23	0	23	594	585	592.0	0.070342	0.00065	0.013495272727272724
383	4843	11	0	11	417	417	417.0	0.029881	0.000694	0.009650099999999998
384	4768	30	0	30	408	408	408.0	0.033587	0.000436	0.008093793103448276
385	4622	68	5	63	12362	11678	11955.0	1.077353	0.015741	0.15566234328358206
386	4874	31	0	31	419	419	419.0	0.040599	0.000441	0.009222733333333335
387	4584	67	3	64	9041	540	5292.0	0.971151	0.000695	0.08751284848484848
388	4603	31	0	31	394	394	394.0	0.018894	0.000413	0.005867733333333333
389	4566	72	2	70	13225	526	6177.0	1.109463	0.000478	0.08201674647887325
390	4598	27	0	27	394	394	394.0	0.022414	0.000396	0.005126923076923076
391	4591	66	3	63	4665	396	2706.0	0.450731	0.000324	0.051266800000000015
392	4474	104	3	101	35626	770	11495.0	3.062043	0.000846	0.1439266796116505
393	4738	105	1	104	35668	803	6554.0	2.010033	0.000893	0.07351879807692309
394	4839	101	4	97	35569	421	15567.0	2.387245	0.000436	0.17715432999999997
395	4866	104	2	102	35649	424	8060.0	1.379257	0.000412	0.06237811650485437
396	4639	112	4	108	35643	407	12872.0	1.777006	0.000386	0.11943261261261262
397	4878	120	4	116	35628	427	11879.0	1.943148	0.00039	0.1040158487394958
398	4754	104	4	100	34782	408	20463.0	2.427207	0.000376	0.20670459223300966
399	4939	117	5	112	35556	423	21489.0	2.864864	0.000462	0.22705166379310343
400	4586	21	0	21	394	394	394.0	0.047973	0.00046	0.010071199999999999
401	4490	49	3	46	676	576	642.0	0.101147	0.000792	0.015656104166666667
402	4958	9	0	9	425	425	425.0	0.040535	0.000399	0.01302675
403	4625	24	0	24	396	396	396.0	0.029705	0.000375	0.0073292608695652165
404	4603	16	0	16	395	395	395.0	0.040206	0.000358	0.012326733333333333
405	4562	43	3	40	397	391	395.0	0.079189	0.000322	0.009798333333333334
406	4810	7	0	7	412	412	412.0	0.032593	0.000369	0.010081833333333333
407	4682	22	0	22	402	402	402.0	0.078205	0.000331	0.01077247619047619
408	4663	76	5	71	24699	407	20216.0	2.342201	0.00051	0.27327171999999983
409	4714	81	3	78	35366	799	16662.0	2.013789	0.00095	0.1843358625
410	4717	40	0	40	480	405	453.0	0.046294	0.000392	0.012555666666666666
411	4781	47	0	47	827	419	544.0	0.062231	0.00048	0.011631804347826086
412	4680	71	5	66	16395	401	14611.0	1.435987	0.000375	0.25640355714285706
413	4808	81	6	75	33768	753	26507.0	2.607196	0.000963	0.2028044125
414	4701	37	0	37	410	403	408.0	0.044317	0.000376	0.007218472222222221
415	4776	44	0	44	691	411	582.0	0.093714	0.000391	0.01211246511627907
416	4537	21	0	21	389	389	389.0	0.059459	0.000367	0.006652749999999999
417	4589	50	4	46	18833	583	15336.0	1.570065	0.000702	0.19838595918367344
418	4707	16	0	16	404	404	404.0	0.02651	0.00088	0.007709066666666667
419	4560	43	3	40	6224	391	5333.0	0.46456	0.000331	0.0632607619047619
420	4907	11	0	11	420	420	420.0	0.045555	0.000518	0.0084925
421	4756	26	0	26	655	412	585.0	0.059082	0.000385	0.010911120000000002
422	4997	8	0	8	428	428	428.0	0.018264	0.000888	0.004188142857142856
423	4662	22	0	22	507	399	490.0	0.033772	0.000353	0.008025714285714285
424	4573	82	5	77	35277	402	22154.0	2.837039	0.00038	0.1992594567901235
425	4818	89	5	84	35182	806	27487.0	2.668561	0.000923	0.2510617159090909
426	4656	75	4	71	33359	399	26972.0	3.605494	0.000393	0.23070352702702704
427	4733	85	6	79	35186	803	26906.0	2.148385	0.001151	0.2534797261904762
428	4791	43	0	43	824	410	598.0	0.061868	0.000381	0.012763071428571426
429	4801	50	0	50	828	418	692.0	0.105314	0.000458	0.01660542857142857
430	4864	39	0	39	813	417	708.0	0.077749	0.000402	0.011219026315789474
431	4819	47	0	47	830	419	704.0	0.12412	0.000404	0.01775360869565218
432	4802	10	0	10	412	412	412.0	0.032166	0.001549	0.017082666666666666
433	4539	28	1	27	389	389	389.0	0.03609	0.000333	0.007853185185185185
434	4879	4	0	4	419	419	419.0	0.006336	0.001397	0.003171666666666667
435	4611	14	0	14	395	395	395.0	0.032176	0.000359	0.004410692307692308
436	4963	4	0	4	426	426	426.0	0.027804	0.003123	0.018338666666666666
437	4680	13	0	13	401	401	401.0	0.053547	0.000353	0.014278583333333332
438	5121	1	0	1
439	4895	6	0	6	419	419	419.0	0.002139	0.000369	0.0009146
440	4603	51	5	46	8138	394	7770.0	0.621619	0.000386	0.11887989999999997
441	4716	59	6	53	27410	26301	27193.0	2.486348	0.040779	0.2281169827586206
442	4632	26	0	26	397	397	397.0	0.074324	0.000361	0.007343720000000001
443	4744	32	0	32	416	415	416.0	0.039913	0.000351	0.006988709677419356
444	4702	25	0	25	457	403	451.0	0.043997	0.0004	0.013928625
445	4706	30	0	30	664	650	661.0	0.112822	0.000748	0.017773827586206895
446	4859	12	0	12	417	417	417.0	0.031074	0.000372	0.004786181818181818
447	4733	16	0	16	405	405	405.0	0.033453	0.000516	0.006051666666666667
448	4897	107	6	101	35565	33683	34325.0	2.645486	0.048617	0.2759087735849056
449	4650	117	2	115	35626	787	11202.0	1.791986	0.000833	0.06600013793103447
450	5058	109	4	105	35732	831	12667.0	2.009291	0.000859	0.11539344444444445
451	4974	123	4	119	35507	434	14205.0	1.231043	0.000446	0.09115389344262297
452	5005	122	1	121	35599	832	7269.0	1.296529	0.000915	0.0766922561983471
453	4770	127	6	121	35574	417	12855.0	3.740406	0.000396	0.12524660317460318
454	5064	122	4	118	35542	442	12207.0	1.872247	0.000406	0.09492408264462804
455	4957	130	5	125	35538	424	14981.0	1.966265	0.000435	0.14356921705426354
456	4554	86	5	81	34474	397	19853.0	2.973064	0.000401	0.15387572941176472
457	4677	46	0	46	723	401	519.0	0.085646	0.000392	0.011321000000000003
458	4732	89	4	85	30690	411	18431.0	1.739284	0.000686	0.12749639772727275
459	4695	48	0	48	498	403	445.0	0.070023	0.000396	0.00976304255319149
460	4533	88	6	82	35102	397	21665.0	3.031303	0.00034	0.17297327586206898
461	4772	45	0	45	830	409	634.0	0.13479	0.000348	0.017723477272727273
462	4644	90	4	86	27091	400	18316.0	2.759531	0.000395	0.1462850786516854
463	4641	46	0	46	453	398	429.0	0.104275	0.000357	0.007952155555555554
464	4788	82	4	78	35243	419	22832.0	1.589552	0.000755	0.18275070370370367
465	4735	86	4	82	34216	412	21257.0	3.345671	0.000393	0.2542693764705883
466	4892	48	0	48	832	419	617.0	0.048519	0.000446	0.013317765957446806
467	4819	49	0	49	726	413	554.0	0.038178	0.000361	0.0082098125
468	4822	83	5	78	34963	420	22461.0	2.44696	0.000458	0.20808632926829274
469	4702	84	5	79	29335	407	24187.0	2.070507	0.000404	0.1608520240963856
470	4645	44	0	44	758	399	593.0	0.05755	0.000389	0.01065479069767442
471	4582	45	0	45	515	394	476.0	0.034257	0.000358	0.006913636363636363
472	4662	60	5	55	11136	491	8511.0	1.083163	0.000576	0.1518878813559322
473	4734	31	0	31	406	406	406.0	0.048088	0.000472	0.010535466666666667
474	4866	33	0	33	417	417	417.0	0.038007	0.000418	0.009182343750000002
475	4893	17	0	17	419	419	419.0	0.06294	0.000539	0.0093403125
476	4484	58	7	51	5364	419	5145.0	0.261769	0.000609	0.047153982456140345
477	4727	29	0	29	405	405	405.0	0.034762	0.000366	0.0063151785714285705
478	4697	31	0	31	403	403	403.0	0.028912	0.000386	0.0056612
479	4925	16	0	16	422	422	422.0	0.042014	0.000402	0.009589733333333333
480	4768	93	7	86	35230	417	18915.0	1.356041	0.000426	0.11487182608695652
481	4712	92	7	85	35223	411	24164.0	1.253467	0.000408	0.13666809890109888
482	4747	92	4	88	35260	415	23816.0	4.385892	0.000431	0.2551370549450549
483	4810	92	5	87	35180	420	27049.0	2.063221	0.000409	0.20561160439560433
484	4779	52	0	52	831	410	545.0	0.109925	0.000376	0.013762313725490201
485	5030	51	0	51	854	431	657.0	0.113004	0.000493	0.011624299999999999
486	4864	51	0	51	838	417	626.0	0.108181	0.000395	0.01531288
487	4769	49	0	49	829	410	690.0	0.168691	0.000383	0.017501604166666667
488	4529	64	6	58	34834	782	28737.0	0.840911	0.000907	0.15293260317460325
489	4700	31	0	31	806	411	712.0	0.064498	0.000452	0.013004799999999999
490	4534	66	5	61	26892	504	23483.0	1.320047	0.000522	0.14104806153846153
491	4722	32	0	32	509	405	491.0	0.043317	0.0004	0.009192580645161293
492	4762	33	0	33	817	414	715.0	0.13401	0.000606	0.018534718749999995
493	4914	16	0	16	430	420	427.0	0.032486	0.000537	0.011139133333333336
494	4825	33	0	33	737	415	675.0	0.095071	0.000385	0.0099645
495	4936	15	0	15	424	423	424.0	0.030645	0.00039	0.01171842857142857
496	4563	62	5	57	30962	750	28752.0	2.497906	0.001256	0.2965245409836066
497	4604	59	6	53	21928	502	20663.0	0.991498	0.000552	0.12354541379310344
498	4704	33	0	33	443	404	438.0	0.039556	0.000417	0.010182187499999999
499	4581	32	0	32	395	393	395.0	0.030057	0.000359	0.005512225806451612
500	4679	32	0	32	709	407	666.0	0.08843	0.000411	0.01497041935483871
501	4697	29	0	29	583	404	567.0	0.041782	0.000377	0.008812607142857143
502	4859	17	0	17	416	416	416.0	0.02546	0.000419	0.004625125
503	4940	15	0	15	423	423	423.0	0.029683	0.000334	0.006086142857142858
504	4587	41	4	37	2074	1818	2041.0	0.08827	0.002139	0.0208681
505	4644	20	0	20	398	398	398.0	0.016099	0.000404	0.003742052631578947
506	4663	22	0	22	400	400	400.0	0.038095	0.000426	0.008067095238095238
507	4810	11	0	11	412	412	412.0	0.022261	0.000391	0.0048446999999999995
508	4595	20	0	20	405	401	404.0	0.031597	0.000323	0.008233894736842103
509	4723	9	0	9	404	404	404.0	0.036299	0.000321	0.011594375
510	4803	10	0	10	411	411	411.0	0.029019	0.000336	0.005391999999999999
511	5033	4	0	4	430	430	430.0	0.002389	0.000346	0.0010416666666666667

40 MPI ranks depth: 4

Total Time: 824.29

Total Tasks: 11413