CNF:Machine Learning: Difference between revisions

From crtc.cs.odu.edu
Jump to navigation Jump to search
← Older edit
Ctsolakis (talk | contribs)
BrainMapping, CNF, HPC, I2MConversion, NDAM, Segmentation, UltraSound3D, Bureaucrats, external, internal, Administrators
2,152 edits
No edit summary
Nikos (talk | contribs)
BrainMapping, CNF, HPC, I2MConversion, NDAM, Segmentation, UltraSound3D, Bureaucrats, external, internal, Administrators
842 edits
 
(131 intermediate revisions by 3 users not shown)
Line 5: Line 5:
states where all combinations of segments in drift chambers have to be considered for firing best track candidates. At high luminosity the number of random segments (unrelated to the tracks)
states where all combinations of segments in drift chambers have to be considered for firing best track candidates. At high luminosity the number of random segments (unrelated to the tracks)
are increasing and as a result the number of possible combinations also increases, making the whole process longer.
are increasing and as a result the number of possible combinations also increases, making the whole process longer.
==The Goal==
==The Goal==
Using machine learning one can recognize the patterns that are valid in order to find the correct track faster.  
Using machine learning one can recognize the patterns that are valid in order to find the correct track faster.  
Line 16: Line 17:
An example of the input to be used by the model can be found in [https://userweb.jlab.org/~gavalian/ML/data/dc/dc_training_svm.txt].
An example of the input to be used by the model can be found in [https://userweb.jlab.org/~gavalian/ML/data/dc/dc_training_svm.txt].


=Timeframe=
=== Data Samples ===
The testing data are given in samples. A sample consists of multiple rows, the first row which is a valid track (labeled as 1) and all the invalid ones(labeled as 0) that come after it, the next valid track represents the beginning of a new sample. For example, consider the following dataset:
 
  _labels___ features ____
|  1      x11 x12 x13 ...|
|  0      x21 x22 x23 ...|
|  0      x31 x32 x33 ...|
|  0      x41 x42 x43 ...|
|  1      x51 x52 x53 ...|
|  0      x61 x62 x63 ...|
----------------------------
This dataset contains two samples:
        Sample 1                      Sample 2
  _labels___ features ____    _labels___ features ____
|  1      x11 x12 x13 ...| |  1      x51 x52 x53 ...|
|  0      x21 x22 x23 ...| |  0      x61 x62 x63 ...|
|  0      x31 x32 x33 ...| ---------------------------
|  0      x41 x42 x43 ...|
----------------------------
 
=== Predictions and Accuracy ===
A prediction is performed with respect to a sample as a whole and is considered correct or not based on whether the model was able to identify which of the rows in a sample is a valid track. The accuracy is then measured by counting the number of samples where the model correctly identified the valid track and dividing this number by the total number of samples. In the previous example, if we have the following predicitions:
 
      Sample 1                                Sample 2
  _labels___ features ____    prediction  _labels___ features ____  prediction
|  '''1'''      x11 x12 x13 ...|      '''1'''      |  '''1'''      x51 x52 x53 ...|    '''0'''
|  0      x21 x22 x23 ...|      0      |  0      x61 x62 x63 ...|    1
|  0      x31 x32 x33 ...|      1      ---------------------------
|  0      x41 x42 x43 ...|      0
----------------------------
 
The model has correctly predicted the valid track of the 1st sample, but not for the second. So the accuracy will be 50%. Notice that even though we have also mispredicted some invalid tracks as valid ones this does not hurt accuracy since we only care about not losing a valid track.
 
=== Possible Issue with Dataset ===
[[Image:incorrect_label_track.png|thumb|250px|right|upright=2|An invalid track incorrectly labeled as valid in the 2019-10-1 dataset.]]
In the dataset of 2019-10-1 we have at least one experiment that looks to be incorrectly labeled. The track can be seen on the picture on the right.
 
= Phase 2 =
In Phase 2 we want to use Recurrent Neural Networks (RNNs) to generate data based on previous observations. Specifically, the RNNs will be trained using the observations on the first 24 layers of the drift chambers to predict the remaining 12. This prediction will be used to narrow down the search space of the last two segments, the ones which draw most of the noise.
=== The Data ===
For this phase, only the valid tracks are taken into account. The input to the RNNs will be the first 24 values of each valid row (with label 1) in the 36cols/row format as data and the rest columns will be used as the prediction targets.
 
For example, a dataset like this:
 
1 1:29.00000 2:30.00000 3:29.00000 4:29.00000 5:29.00000 6:0.00000 7:30.00000 8:30.00000 9:29.50000 10:30.00000 11:29.00000 12:30.00000 ...
0 1:23.00000 2:23.00000 3:23.00000 4:23.00000 5:22.00000 6:23.00000 7:26.00000 8:27.00000 9:26.00000 10:26.00000 11:26.00000 12:26.00000 ...
1 1:12.00000 2:12.50000 3:12.00000 4:12.00000 5:11.00000 6:11.50000 7:13.00000 8:13.00000 9:13.00000 10:13.00000 11:12.00000 12:12.00000 ...
0 1:5.00000 2:5.00000 3:4.00000 4:0.00000 5:4.00000 6:4.00000 7:13.00000 8:13.00000 9:13.00000 10:13.00000 11:12.00000 12:12.00000 ...
1 1:64.00000 2:64.00000 3:64.00000 4:64.00000 5:64.00000 6:64.00000 7:71.00000 8:71.00000 9:71.00000 10:71.00000 11:71.00000 12:71.00000 ...
0 1:54.00000 2:54.00000 3:54.00000 4:54.00000 5:54.00000 6:54.00000 7:60.00000 8:60.00000 9:60.00000 10:60.00000 11:60.00000 12:60.00000 ...
1 1:23.00000 2:23.00000 3:23.00000 4:23.00000 5:22.00000 6:22.00000 7:26.00000 8:26.00000 9:26.00000 10:26.00000 11:25.00000 12:26.00000 ...
0 1:14.00000 2:15.00000 3:14.00000 4:14.00000 5:13.00000 6:14.00000 7:17.00000 8:17.00000 9:17.00000 10:17.00000 11:16.00000 12:16.00000 ...
1 1:55.00000 2:55.00000 3:55.00000 4:56.00000 5:55.00000 6:55.50000 7:59.00000 8:60.00000 9:59.00000 10:60.00000 11:59.00000 12:60.00000 ...
0 1:38.00000 2:39.00000 3:38.00000 4:39.00000 5:39.00000 6:40.00000 7:43.00000 8:44.00000 9:43.00000 10:44.00000 11:44.00000 12:45.00000 ...
1 1:12.00000 2:12.00000 3:12.00000 4:12.00000 5:11.00000 6:11.00000 7:14.00000 8:14.00000 9:13.00000 10:13.00000 11:12.50000 12:13.00000 ...
0 1:12.00000 2:12.00000 3:12.00000 4:12.00000 5:11.00000 6:11.00000 7:0.00000 8:18.00000 9:0.00000 10:17.00000 11:17.00000 12:17.00000 ...
1 1:43.00000 2:44.00000 3:43.00000 4:43.00000 5:43.00000 6:43.00000 7:43.00000 8:43.00000 9:43.00000 10:43.00000 11:43.00000 12:43.00000 ...
0 1:32.00000 2:33.00000 3:32.00000 4:33.00000 5:32.00000 6:33.00000 7:34.00000 8:34.00000 9:34.00000 10:34.00000 11:34.00000 12:34.00000 ...
1 1:25.00000 2:26.00000 3:25.00000 4:25.00000 5:25.00000 6:25.00000 7:25.00000 8:26.00000 9:25.00000 10:25.00000 11:24.00000 12:25.00000 ...
 
Will be converted to:
1:29.00000 2:30.00000 3:29.00000 4:29.00000 5:29.00000 6:0.00000 7:30.00000 8:30.00000 9:29.50000 10:30.00000 11:29.00000 12:30.00000 ...
1:12.00000 2:12.50000 3:12.00000 4:12.00000 5:11.00000 6:11.50000 7:13.00000 8:13.00000 9:13.00000 10:13.00000 11:12.00000 12:12.00000 ...
1:64.00000 2:64.00000 3:64.00000 4:64.00000 5:64.00000 6:64.00000 7:71.00000 8:71.00000 9:71.00000 10:71.00000 11:71.00000 12:71.00000 ...
1:23.00000 2:23.00000 3:23.00000 4:23.00000 5:22.00000 6:22.00000 7:26.00000 8:26.00000 9:26.00000 10:26.00000 11:25.00000 12:26.00000 ...
1:55.00000 2:55.00000 3:55.00000 4:56.00000 5:55.00000 6:55.50000 7:59.00000 8:60.00000 9:59.00000 10:60.00000 11:59.00000 12:60.00000 ...
1:12.00000 2:12.00000 3:12.00000 4:12.00000 5:11.00000 6:11.00000 7:14.00000 8:14.00000 9:13.00000 10:13.00000 11:12.50000 12:13.00000 ...
1:43.00000 2:44.00000 3:43.00000 4:43.00000 5:43.00000 6:43.00000 7:43.00000 8:43.00000 9:43.00000 10:43.00000 11:43.00000 12:43.00000 ...
1:25.00000 2:26.00000 3:25.00000 4:25.00000 5:25.00000 6:25.00000 7:25.00000 8:26.00000 9:25.00000 10:25.00000 11:24.00000 12:25.00000 ...
 
The features, in this case, are the first 24 columns, while the prediction targets are the rest 12, e.g.:
 
Training features:
 
1:29.00000 2:30.00000 3:29.00000 4:29.00000 5:29.00000 6:0.00000 7:30.00000 8:30.00000 9:29.50000 10:30.00000 11:29.00000 12:30.00000 13:25.00000 14:26.00000 15:25.00000 16:25.00000 17:24.00000 18:0.00000 19:24.00000 20:24.00000 21:24.00000 22:24.00000 23:23.00000 24:23.00000
 
Prediction targets:
25:14.50000 26:14.50000 27:13.50000 28:14.00000 29:13.00000 30:13.00000 31:13.00000 32:13.00000 33:12.00000 34:12.00000 35:11.00000 36:11.00000
 
=== Goal ===
The goal of this phase is to predict the track that the particle followed in the last 2 segments of the drift chamber. Having this kind of information, the search space will be narrow
enough, so that the models built in the previous phase will be able to make correct inference about which track is the valid one.
 
[[File:RNN ex 3210 labeled.png]]
 
==Scope of Work for 2019-20 Academic Year==
To be stated explicitly.
 
==Timeframe==
From: https://github.com/gavalian/clas12ai/blob/master/docs/project_scope.txt
 
* by end of year: Have the software runnable, and implement in the current workflow for our detector reconstruction.
**October 1: Finalize training and testing procedures. Implement validation procedure to validate test sample, where accuracy of the network is determined from sample of N candidates.
**December 1: provide software (probably in docker container) that will accept training set as an input and produce a trained neural network. Provide software that can evaluate track candidates and return sorted list ordered by goodness probability. First draft version can be in python.
 
* Starting early January: start investigating RNNs, divide training sample into three regions, each 12 entries in the training sample represent one input. At this point the objective is to be able to:  '''"predict what next 12 states will be, and based on 24 entries predict what next 12 should be. Investigate networks that can provide best accuracy".'''
** Depending on progress (sometime in March) start implementing software that can automate the process of training and prediction.
* By end of Spring complete research and start the writing of a paper to be completed by end of Summer.
 
===Week 1===
===Week 1===
Create the pipeline
Create the pipeline
Line 487: Line 586:
|}
|}


===Week 2===
===Week 2 & 3===
*Try the models on latest data
*Try the models on latest data
*Change the way we evaluate network accuracy
*Change the way we evaluate network accuracy
Line 494: Line 593:
*Try and get accuracy with 1/3 of data, 1/2 and all of the data
*Try and get accuracy with 1/3 of data, 1/2 and all of the data
*Try to do some more tweaking
*Try to do some more tweaking
{| class="wikitable sortable mw-collapsible mw-collapsed"
|+ <strong style="white-space:nowrap"> Results for new accuracy model </strong>
! class="unsortable"| Method/Train set
! Split
! Test Set/Format
! Test Accuracy
! Training Accuracy
! Time to train (sec)
! Time to predict (sec)
! Figure
|-
| CNN/10-1-19
| 80/20
| 10-1-19/4032
| 0.975
| 0.932
| 209
| 0.001
| [[File:cnn_acc_10-1.png|240px]] [[File:cnn_loss_10-1.png|240px]]
|-
| ExtraTrees/10-1-19
| N/A
| 10-1-19/36
| 1.0
| 1.0
| 10.7
| 1.9
|
|-
| rowspan="2"| CNN/09-16-19
| rowspan="2"| 80/20
| 09-16-19/4032
| 0.936
| rowspan="2"|0.937
| rowspan="2"|446
| rowspan="2"|0.002
| rowspan="2"|[[File:cnn_acc.png|240px]] [[File:cnn_loss.png|240px]]
|-
| 09-27-19/4032
| 0.134
|-
| rowspan="2"|CNN/09-27-19
| rowspan="2"|80/20
| 09-27-19/4032
| 0.827
| rowspan="2"|0.87
| rowspan="2"|190
| rowspan="2"|0.001
| rowspan="2"|[[File:cnn_acc_09-27.png|240px]] [[File:cnn_loss_09-27.png|240px]]
|-
| 09-16-19/4032
| 0.487
|-
| rowspan="3"| CNN/09-16+27-19
| rowspan="3"| 80/20
| 09-16+27-19/4032
| 0.897
| rowspan="3"|0.909
| rowspan="3"|630
| rowspan="3"|0.001
| rowspan="3"|[[File:cnn_acc_09-16+27.png|240px]] [[File:cnn_loss_09-16+27.png|240px]]
|-
| 09-16-19/4032
| 0.910
|-
| 09-27-19/4032
| 0.845
|-
| LinearSVM / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.0
|
| 38866.6
| 0.0
|
|-
| RandomForest / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.512
|
| 4.7
| 0.0
|
|-
| QDA / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.752
|
| 0.4
| 0.0
|
|-
| AdaBoost / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.0
|
| 15.7
| 0.0
|
|-
| DecisionTree / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.601
|
| 3.9
| 0.0
|
|-
| NearestNeighbors / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.563
|
| 78.4
| 0.0
|
|-
| NaiveBayes / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.535
|
| 0.1
| 0.0
|
|-
| RBFSVM / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 36
| 0.0
|
| 7297.2
| 0.0
|
|-
| RandomForest / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 4032
| 0.147
|
| 72.8
| 0.0
|
|-
| AdaBoost / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 4032
| 0.0
|
| 990.7
| 0.0
|
|-
| DecisionTree / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 4032
| 0.362
|
| 588.5
| 0.0
|
|-
| NearestNeighbors / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 4032
| 0.545
|
| 31519.1
| 5.8
|
|-
| NaiveBayes / 2019-09-16+27
| 80/20
| 2019-09-16+27 / 4032
| 0.776
|
| 14.7
| 0.0
|
|-
| RandomForest / 2019-09-27
| 80/20
| 2019-09-27 / 6
| 0.63
|
| 0.7
| 0.0
|
|-
| QDA / 2019-09-27
| 80/20
| 2019-09-27 / 6
| 0.818
|
| 0.0
| 0.0
|
|-
| AdaBoost / 2019-09-27
| 80/20
| 2019-09-27 / 6
| 0.102
|
| 3.7
| 0.0
|
|-
| DecisionTree / 2019-09-27
| 80/20
| 2019-09-27 / 6
| 0.652
|
| 0.3
| 0.0
|
|-
| NearestNeighbors / 2019-09-27
| 80/20
| 2019-09-27 / 6
| 0.763
|
| 0.4
| 0.0
|
|-
| NaiveBayes / 2019-09-27
| 80/20
| 2019-09-27 / 6
| 0.058
|
| 0.1
| 0.0
|
|-
| RandomForest / 2019-09-27
| 80/20
| 2019-09-27 / 36
| 0.447
|
| 1.0
| 0.0
|
|-
| QDA / 2019-09-27
| 80/20
| 2019-09-27 / 36
| 0.218
|
| 0.2
| 0.0
|
|-
| AdaBoost / 2019-09-27
| 80/20
| 2019-09-27 / 36
| 0.012
|
| 8.5
| 0.0
|
|-
| DecisionTree / 2019-09-27
| 80/20
| 2019-09-27 / 36
| 0.528
|
| 0.8
| 0.0
|
|-
| NearestNeighbors / 2019-09-27
| 80/20
| 2019-09-27 / 36
| 0.494
|
| 5.5
| 0.0
|
|-
| NaiveBayes / 2019-09-27
| 80/20
| 2019-09-27 / 36
| 0.311
|
| 0.1
| 0.0
|
|-
| RandomForest / 2019-09-27
| 80/20
| 2019-09-27 / 4032
| 0.123
|
| 17.6
| 0.0
|
|-
| AdaBoost / 2019-09-27
| 80/20
| 2019-09-27 / 4032
| 0.011
|
| 229.7
| 0.0
|
|-
| DecisionTree / 2019-09-27
| 80/20
| 2019-09-27 / 4032
| 0.374
|
| 144.4
| 0.0
|
|-
| NaiveBayes / 2019-09-27
| 80/20
| 2019-09-27 / 4032
| 0.355
|
| 3.0
| 0.0
|
|-
| RandomForest / 2019-09-16
| 80/20
| 2019-09-16 / 36
| 0.522
|
| 2.7
| 0.0
|
|-
| QDA / 2019-09-16
| 80/20
| 2019-09-16 / 36
| 0.801
|
| 0.4
| 0.0
|
|-
| AdaBoost / 2019-09-16
| 80/20
| 2019-09-16 / 36
| 0.0
|
| 14.6
| 0.0
|
|-
| DecisionTree / 2019-09-16
| 80/20
| 2019-09-16 / 36
| 0.609
|
| 2.4
| 0.0
|
|-
| NaiveBayes / 2019-09-16
| 80/20
| 2019-09-16 / 36
| 0.447
|
| 0.1
| 0.0
|
|-
| RandomForest / 2019-09-16
| 80/20
| 2019-09-16 / 4032
| 0.167
|
| 47.8
| 0.0
|
|-
| AdaBoost / 2019-09-16
| 80/20
| 2019-09-16 / 4032
| 0.0
|
| 477.1
| 0.0
|
|-
| DecisionTree / 2019-09-16
| 80/20
| 2019-09-16 / 4032
| 0.383
|
| 402.7
| 0.0
|
|-
| NaiveBayes / 2019-09-16
| 80/20
| 2019-09-16 / 4032
| 0.569
|
| 7.9
| 0.0
|
|}
'''Note:''' Training accuracy is evaluated per data row and not per sample. Testing accuracy is evaluated per sample. <br/>
'''Note 2:''' The false negatives, in this case, are all the samples that were mispredicted. In another words false negatives % = 1- acc %.
=== Week 4 ===
[[Image:acc_for_diff_sizes.png|thumb|350px|right|upright=2|Accuracy metrics score for different size of training set using CNNs.]]
* Keep more accuracy metrics for our testing procedure, specifically:
# The ratio of samples where the valid track was correctly detected. (A1)
# Same as 1, but also had invalid tracks miss-predicted as valid ones (False Positives). (Ac)
# Same as 1, but also the valid track has the highest probability for label 1 of all rows in the sample. (Ah)
# The ration of samples where the valid track was not detected (False Negatives). (Af)
* Train on fractions of the train set (e.g 20%,40%,60%,80%) and create figures of all the metrics with respect to the fraction used.
{| class="wikitable sortable mw-collapsible mw-collapsed"
|+ <strong style="white-space:nowrap"> Results with multiple accuracy metrics </strong>
! class="unsortable"| Method/Train set
! Split
! Test Set/Format
! A1
! Ac
! Ah
! Af
! Training Acc
! Validation Acc
! Time to train (sec)
! Time to predict (sec)
|-
| rowspan="10"| CNN/10-7-19
| 10/90
| 10-7-19/4032
| 0.915
| 0.511
| 0.832
| 0.084
| 0.9389
| 0.900
| 199
| 0.0012
|-
| 20/80
| 10-7-19/4032
| 0.935
| 0.450
| 0.863
| 0.064
| 0.9310
| 0.9136
| 226.1
| 0.0012
|-
| 30/70
| 10-7-19/4032
| 0.945
| 0.389
| 0.878
| 0.054
| 0.934
| 0.924
| 257.1
| 0.0012
|-
| 40/60
| 10-7-19/4032
| 0.955
| 0.382
| 0.885
| 0.044
| 0.934
| 0.926
| 288.1
| 0.001
|-
| 50/50
| 10-7-19/4032
| 0.944
| 0.331
| 0.880
| 0.055
| 0.934
| 0.931
| 319
| 0.001
|-
| 60/40
| 10-7-19/4032
| 0.954
| 0.353
| 0.888
| 0.045
| 0.934
| 0.93
| 347.4
| 0.0013
|-
| 70/30
| 10-7-19/4032
| 0.954
| 0.350
| 0.885
| 0.045
| 0.9343
| 0.9325
| 380.8
| 0.0014
|-
| 80/20
| 10-7-19/4032
| 0.959
| 0.327
| 0.892
| 0.040
| 0.9344
| 0.9315
| 412.4
| 0.0014
|-
| 90/10
| 10-7-19/4032
| 0.956
| 0.332
| 0.880
| 0.043
| 0.9347
| 0.9332
| 444.3
| 0.0014
|-
| '''100/0'''
| '''10-7-19/4032'''
| '''0.964'''
| '''0.301'''
| '''0.894'''
| '''0.035'''
| '''0.934'''
| '''N/A'''
| '''457'''
| '''0.0012'''
|-
| rowspan="11"| ExtraTrees/10-7-19
| 10/90
| 10-7-19/6
| 0.923
| 0.241
| 0.914
| 0.077
| 1.0
| 0.92
| 0.2
| 0.000005
|-
| 20/80
| 10-7-19/6
| 0.921
| 0.217
| 0.914
| 0.078
| 1.0
| 0.924
| 0.3
| 0.000005
|-
| 30/70
| 10-7-19/6
| 0.923
| 0.217
| 0.912
| 0.076
| 1.0
| 0.925
| 0.4
| 0.000005
|-
| 40/60
| 10-7-19/6
| 0.927
| 0.205
| 0.914
| 0.072
| 1.0
| 0.927
| 0.6
| 0.000005
|-
| 50/50
| 10-7-19/6
| 0.930
| 0.204
| 0.916
| 0.069
| 1.0
| 0.928
| 0.7
| 0.000005
|-
| 60/40
| 10-7-19/6
| 0.929
| 0.206
| 0.918
| 0.070
| 1.0
| 0.915
| 0.8
| 0.000005
|-
| 70/30
| 10-7-19/6
| 0.932
| 0.203
| 0.92
| 0.067
| 1.0
| 0.93
| 1.0
| 0.000005
|-
| 80/20
| 10-7-19/6
| 0.933
| 0.199
| 0.918
| 0.066
| 1.0
| 0.917
| 1.2
| 0.000005
|-
| 90/10
| 10-7-19/6
| 0.933
| 0.197
| 0.919
| 0.066
| 0.9999
| 0.931
| 1.3
| 0.000005
|-
| '''100/0'''
| '''10-7-19/6'''
| '''0.933'''
| '''0.199'''
| '''0.919'''
| '''0.066'''
| '''0.9999'''
| '''N/A'''
| '''1.7'''
| '''0.000005'''
|-
| 100/0
| 10-7-19/36
| 0.910
| 0.339
| 0.883
| 0.089
| 0.9999
| N/A
| 8.4
| 0.000005
|-
| rowspan="2"| MLP/10-7-19
| '''100/0'''
| '''10-7-19/6'''
| '''0.965'''
| '''0.202'''
| '''0.921'''
| '''0.034'''
| '''0.947'''
| '''N/A'''
| '''252 (CPU)'''
| '''0.000004'''
|-
| 100/0
| 10-7-19/36
| 0.920
| 0.312
| 0.849
| 0.079
| 0.921
| N/A
| 546 (CPU)
| 0.00001
|-
|}
==== Observations ====
As seen in the 100/0 split rows:
* CNN is 3.1% more accurate at detecting valid samples than the ET
* ET has 10.1% less detected samples containing false positive predictions (ET has less false positives than the CNN)
* ET is 2.5% more accurate at giving the highest probability of being 1 to the valid row in a sample than the CNN
* ET is 269x faster at training than the CNN (for CNN/4032 and ET/6)
* ET is 240x faster at prediction (inference) than the CNN (for CNN/4032 and ET/6)
* MLP is 0.1% more accurate at detecting valid samples than the CNN
* MLP has 9.9% less detected samples containing false positive predictions (MLP has less false positives than the CNN but more than the ET)
* MLP is 2.6% more accurate at giving the highest probability of being 1 to the valid row in a sample than the CNN
* MLP is 1.8x faster at training than the CNN (for GPU CNN/4032 and CPU MLP/6)
* MLP is 300x faster at prediction (inference) than the CNN (for CNN/4032 and MLP/6)
* MLP and ET perform best on the feature-extracted 6 format
=== Week 6 ===
*Deliver the python script that does the training and testing for CNN, MLP, and ExtraTrees.
*Also, provide a function that can read data from C and pass it back to python.
=== Week 7 ===
* Experiment with RNNs and try to generate new data sequences
{| class="wikitable mw-collapsible mw-collapsed"
|+ <strong style="white-space:nowrap"> Preliminary results for RNNs </strong>
! Method/Train set
! Test Set/Format
! Loss Function
! Training loss
! Validation loss
! Time to train (sec)
! Time to predict (sec)
|-
| LSTM/10-7-19
| 10-7-19 / 36
| MAE
|
| 1.36
|
|
|}
{| class="wikitable mw-collapsible mw-collapsed"
|+ <strong style="white-space:nowrap"> Some random cases demonstrating the performance of the model </strong>
|[[File:RNN ex 7.png|420px]]
|[[File:RNN ex 17.png|420px]]
|[[File:RNN ex 47.png|420px]]
|-
|[[File:RNN ex 33.png|420px]]
|[[File:RNN ex 167.png|420px]]
|[[File:RNN ex 3210.png|420px]]
|}
==== Predict 1 ====
[[Image:Predict-1.png|thumb|350px|right|upright=2|Training/Validation loss versus epoch in predict 1 case.]]
Training Performance:
Loss (MAE): 0.0556 - MSE: 0.0093
Validation Performance:
Loss (MAE): 0.0490 - MSE: 0.0076
Test Performance:
Loss (MAE): 0.050 - MSE: 0.0085
First 9 rows true values vs predicted values:
[1.575] [1.5128074]
[1.898] [1.9050792]
[4.931] [4.928837]
[2.326] [2.3151793]
[0.566] [0.51875186]
[1.349] [1.3323742]
[0.997] [0.9642346]
[0.759] [0.7421971]
[0.766] [0.7594936]
==== Predict 2 ====
[[Image:Predict2.png|thumb|350px|right|upright=2|Training/Validation loss versus epoch in predict 2 case.]]
Training Performance:
Loss (MAE):  0.0329 - MSE: 0.0042
Validation Performance:
Loss (MAE): 0.0373 - MSE: 0.0042
Test Performance:
Loss (MAE): 0.037 - MSE: 0.0046
First 9 rows true values vs predicted values:
[1.575 0.457] [1.5281367  0.46965083]
[1.898 0.296] [1.9137108  0.29100114]
[4.931 0.212] [5.0380673  0.21810552]
[2.326 0.261] [2.3030581  0.25453362]
[0.566 0.753] [0.4852956 0.6849611]
[1.349 0.396] [1.3475827  0.39197764]
[0.997 0.556] [0.9059359  0.55739856]
[0.759 0.506] [0.71836287 0.5392053 ]
[0.766 0.6  ] [0.7425585  0.60809374]
==== Predict 3 ====
[[Image:Predict3.png|thumb|350px|right|upright=2|Training/Validation loss versus epoch in predict 3 case.]]
Training Performance:
Loss (MAE): 0.0297 - MSE: 0.0035
Validation Performance:
Loss (MAE): 0.0274 - MSE: 0.0030
Test Performance:
Loss (MAE): 0.0272 - MSE: 0.0033
First 9 rows real values vs predicted values
[1.575 0.457 0.138] [1.5253325  0.4581103  0.14659211]
[1.898 0.296 0.667] [1.9686979  0.2962818  0.66101694]
[4.931 0.212 0.365] [4.981064  0.21543622 0.3699692 ]
[2.326 0.261 0.095] [2.220339  0.2587005  0.09193501]
[0.566 0.753 1.443] [0.55893284 0.73924404 1.4928124 ]
[1.349 0.396 0.531] [1.3435515  0.39372128 0.5413537 ]
[0.997 0.556 0.621] [1.0042843  0.558294  0.63084584]
[0.759 0.506 0.784] [0.7833784  0.5464319  0.81894845]
[0.766 0.6  0.785] [0.77423286 0.6099417  0.7522259 ]
= Relevant Literature =
* https://ieeexplore.ieee.org/abstract/document/8853611
* http://gososkov.ru/Student%20school-2015-2018/37-45-paper-6.pdf
* https://arxiv.org/pdf/1810.06111.pdf
* https://dl4physicalsciences.github.io/files/nips_dlps_2017_28.pdf
* https://indico.cern.ch/event/567550/papers/2629737/files/6146-hepTrkX_acat.pdf
* https://arxiv.org/pdf/1812.03859.pdf
* https://ieeexplore.ieee.org/document/8588707
= References =
= References =
* https://userweb.jlab.org/~gavalian/ML/data/dc/dc_training_svm.txt
* https://userweb.jlab.org/~gavalian/ML/data/dc/dc_training_svm.txt
* https://userweb.jlab.org/~gavalian/presentations/ML-Tracking.pdf
* https://userweb.jlab.org/~gavalian/presentations/ML-Tracking.pdf
* https://deeplearning4j.org/docs/latest/deeplearning4j-cheat-sheet
* https://deeplearning4j.org/docs/latest/deeplearning4j-cheat-sheet
* https://userweb.jlab.org/~gavalian/ML/data/
* https://deephyper.readthedocs.io/en/latest/
* https://automl.github.io/auto-sklearn/master/
* https://www.alcf.anl.gov/presentations
* https://userweb.jlab.org/~gavalian/presentations/AI-Tracking-progress.pdf
* Gagik's presentation to CLAS12 Tracking AI 2020 at Jlab: https://userweb.jlab.org/~gavalian/presentations/AI2020-SUMMARY-GAVALIAN.pdf

Latest revision as of 08:21, 28 March 2020

This is the page for the CNF-Machine Learning project. It will be updated frequently with reports about the progress of the project.

The Problem

Traditional tracking algorithms are computationally intensive, especially for high luminosity experiments with multi-track final states where all combinations of segments in drift chambers have to be considered for firing best track candidates. At high luminosity the number of random segments (unrelated to the tracks) are increasing and as a result the number of possible combinations also increases, making the whole process longer.

The Goal

Using machine learning one can recognize the patterns that are valid in order to find the correct track faster. The model will be trained on real pre-labeled data and as an outcome, it will be able to label track combinations as valid or not.

The Data

A representation of the drift chambers. The red squares represent a sensor that has detected a hit

The drift chambers consist of 6 layers, of 6 wires, of 112 sensors each for a total of 4032 sensors (see picture on the right). The data provided let us know whether a sensor has detected a hit or not. Those detections might be part of the trajectory that we want to track or can be irrelevant (noise). The labeled data consist of all the possible combinations that form a track as rows (events) and the state of each sensor (detected something or not) as columns (features). The label provides information on whether a combination produces the valid track or not. An example of the input to be used by the model can be found in [1].

Data Samples

The testing data are given in samples. A sample consists of multiple rows, the first row which is a valid track (labeled as 1) and all the invalid ones(labeled as 0) that come after it, the next valid track represents the beginning of a new sample. For example, consider the following dataset: 

 _labels___ features ____ 
|   1       x11 x12 x13 ...|
|   0       x21 x22 x23 ...|
|   0       x31 x32 x33 ...|
|   0       x41 x42 x43 ...|
|   1       x51 x52 x53 ...|
|   0       x61 x62 x63 ...|
----------------------------

This dataset contains two samples: 

       Sample 1                      Sample 2
 _labels___ features ____     _labels___ features ____ 
|   1       x11 x12 x13 ...| |  1       x51 x52 x53 ...|
|   0       x21 x22 x23 ...| |  0       x61 x62 x63 ...|
|   0       x31 x32 x33 ...| --------------------------- 
|   0       x41 x42 x43 ...| 
----------------------------

Predictions and Accuracy

A prediction is performed with respect to a sample as a whole and is considered correct or not based on whether the model was able to identify which of the rows in a sample is a valid track. The accuracy is then measured by counting the number of samples where the model correctly identified the valid track and dividing this number by the total number of samples. In the previous example, if we have the following predicitions: 

      Sample 1                                Sample 2
 _labels___ features ____    prediction   _labels___ features ____   prediction
|   1       x11 x12 x13 ...|      1      |  1       x51 x52 x53 ...|     0
|   0       x21 x22 x23 ...|      0      |  0       x61 x62 x63 ...|     1
|   0       x31 x32 x33 ...|      1      --------------------------- 
|   0       x41 x42 x43 ...|      0
----------------------------

The model has correctly predicted the valid track of the 1st sample, but not for the second. So the accuracy will be 50%. Notice that even though we have also mispredicted some invalid tracks as valid ones this does not hurt accuracy since we only care about not losing a valid track.

Possible Issue with Dataset

An invalid track incorrectly labeled as valid in the 2019-10-1 dataset.

In the dataset of 2019-10-1 we have at least one experiment that looks to be incorrectly labeled. The track can be seen on the picture on the right.

Phase 2

In Phase 2 we want to use Recurrent Neural Networks (RNNs) to generate data based on previous observations. Specifically, the RNNs will be trained using the observations on the first 24 layers of the drift chambers to predict the remaining 12. This prediction will be used to narrow down the search space of the last two segments, the ones which draw most of the noise.

The Data

For this phase, only the valid tracks are taken into account. The input to the RNNs will be the first 24 values of each valid row (with label 1) in the 36cols/row format as data and the rest columns will be used as the prediction targets.

For example, a dataset like this: 

1 1:29.00000 2:30.00000 3:29.00000 4:29.00000 5:29.00000 6:0.00000 7:30.00000 8:30.00000 9:29.50000 10:30.00000 11:29.00000 12:30.00000 ...
0 1:23.00000 2:23.00000 3:23.00000 4:23.00000 5:22.00000 6:23.00000 7:26.00000 8:27.00000 9:26.00000 10:26.00000 11:26.00000 12:26.00000 ...
1 1:12.00000 2:12.50000 3:12.00000 4:12.00000 5:11.00000 6:11.50000 7:13.00000 8:13.00000 9:13.00000 10:13.00000 11:12.00000 12:12.00000 ...
0 1:5.00000 2:5.00000 3:4.00000 4:0.00000 5:4.00000 6:4.00000 7:13.00000 8:13.00000 9:13.00000 10:13.00000 11:12.00000 12:12.00000 ...
1 1:64.00000 2:64.00000 3:64.00000 4:64.00000 5:64.00000 6:64.00000 7:71.00000 8:71.00000 9:71.00000 10:71.00000 11:71.00000 12:71.00000 ...
0 1:54.00000 2:54.00000 3:54.00000 4:54.00000 5:54.00000 6:54.00000 7:60.00000 8:60.00000 9:60.00000 10:60.00000 11:60.00000 12:60.00000 ...
1 1:23.00000 2:23.00000 3:23.00000 4:23.00000 5:22.00000 6:22.00000 7:26.00000 8:26.00000 9:26.00000 10:26.00000 11:25.00000 12:26.00000 ...
0 1:14.00000 2:15.00000 3:14.00000 4:14.00000 5:13.00000 6:14.00000 7:17.00000 8:17.00000 9:17.00000 10:17.00000 11:16.00000 12:16.00000 ...
1 1:55.00000 2:55.00000 3:55.00000 4:56.00000 5:55.00000 6:55.50000 7:59.00000 8:60.00000 9:59.00000 10:60.00000 11:59.00000 12:60.00000 ...
0 1:38.00000 2:39.00000 3:38.00000 4:39.00000 5:39.00000 6:40.00000 7:43.00000 8:44.00000 9:43.00000 10:44.00000 11:44.00000 12:45.00000 ...
1 1:12.00000 2:12.00000 3:12.00000 4:12.00000 5:11.00000 6:11.00000 7:14.00000 8:14.00000 9:13.00000 10:13.00000 11:12.50000 12:13.00000 ...
0 1:12.00000 2:12.00000 3:12.00000 4:12.00000 5:11.00000 6:11.00000 7:0.00000 8:18.00000 9:0.00000 10:17.00000 11:17.00000 12:17.00000 ...
1 1:43.00000 2:44.00000 3:43.00000 4:43.00000 5:43.00000 6:43.00000 7:43.00000 8:43.00000 9:43.00000 10:43.00000 11:43.00000 12:43.00000 ...
0 1:32.00000 2:33.00000 3:32.00000 4:33.00000 5:32.00000 6:33.00000 7:34.00000 8:34.00000 9:34.00000 10:34.00000 11:34.00000 12:34.00000 ...
1 1:25.00000 2:26.00000 3:25.00000 4:25.00000 5:25.00000 6:25.00000 7:25.00000 8:26.00000 9:25.00000 10:25.00000 11:24.00000 12:25.00000 ...

Will be converted to: 

1:29.00000 2:30.00000 3:29.00000 4:29.00000 5:29.00000 6:0.00000 7:30.00000 8:30.00000 9:29.50000 10:30.00000 11:29.00000 12:30.00000 ...
1:12.00000 2:12.50000 3:12.00000 4:12.00000 5:11.00000 6:11.50000 7:13.00000 8:13.00000 9:13.00000 10:13.00000 11:12.00000 12:12.00000 ...
1:64.00000 2:64.00000 3:64.00000 4:64.00000 5:64.00000 6:64.00000 7:71.00000 8:71.00000 9:71.00000 10:71.00000 11:71.00000 12:71.00000 ...
1:23.00000 2:23.00000 3:23.00000 4:23.00000 5:22.00000 6:22.00000 7:26.00000 8:26.00000 9:26.00000 10:26.00000 11:25.00000 12:26.00000 ...
1:55.00000 2:55.00000 3:55.00000 4:56.00000 5:55.00000 6:55.50000 7:59.00000 8:60.00000 9:59.00000 10:60.00000 11:59.00000 12:60.00000 ...
1:12.00000 2:12.00000 3:12.00000 4:12.00000 5:11.00000 6:11.00000 7:14.00000 8:14.00000 9:13.00000 10:13.00000 11:12.50000 12:13.00000 ...
1:43.00000 2:44.00000 3:43.00000 4:43.00000 5:43.00000 6:43.00000 7:43.00000 8:43.00000 9:43.00000 10:43.00000 11:43.00000 12:43.00000 ...
1:25.00000 2:26.00000 3:25.00000 4:25.00000 5:25.00000 6:25.00000 7:25.00000 8:26.00000 9:25.00000 10:25.00000 11:24.00000 12:25.00000 ...

The features, in this case, are the first 24 columns, while the prediction targets are the rest 12, e.g.:

Training features: 

1:29.00000 2:30.00000 3:29.00000 4:29.00000 5:29.00000 6:0.00000 7:30.00000 8:30.00000 9:29.50000 10:30.00000 11:29.00000 12:30.00000 13:25.00000 14:26.00000 15:25.00000 16:25.00000 17:24.00000 18:0.00000 19:24.00000 20:24.00000 21:24.00000 22:24.00000 23:23.00000 24:23.00000

Prediction targets: 

25:14.50000 26:14.50000 27:13.50000 28:14.00000 29:13.00000 30:13.00000 31:13.00000 32:13.00000 33:12.00000 34:12.00000 35:11.00000 36:11.00000

Goal

The goal of this phase is to predict the track that the particle followed in the last 2 segments of the drift chamber. Having this kind of information, the search space will be narrow enough, so that the models built in the previous phase will be able to make correct inference about which track is the valid one.

Scope of Work for 2019-20 Academic Year

To be stated explicitly.

Timeframe

From: https://github.com/gavalian/clas12ai/blob/master/docs/project_scope.txt

  • by end of year: Have the software runnable, and implement in the current workflow for our detector reconstruction.
    • October 1: Finalize training and testing procedures. Implement validation procedure to validate test sample, where accuracy of the network is determined from sample of N candidates.
    • December 1: provide software (probably in docker container) that will accept training set as an input and produce a trained neural network. Provide software that can evaluate track candidates and return sorted list ordered by goodness probability. First draft version can be in python.
  • Starting early January: start investigating RNNs, divide training sample into three regions, each 12 entries in the training sample represent one input. At this point the objective is to be able to: "predict what next 12 states will be, and based on 24 entries predict what next 12 should be. Investigate networks that can provide best accuracy".
    • Depending on progress (sometime in March) start implementing software that can automate the process of training and prediction.
  • By end of Spring complete research and start the writing of a paper to be completed by end of Summer.

Week 1

Create the pipeline

  • Load data in Spark Dataframes
  • Split data in training/validation sets
  • Experiment with different classification methods
    • Tree-based
    • SVM
  • Train Model
  • Validate
  • Report accuracy and confusion matrix
Results for input with 4032 features
Method Split Accuracy False Positives False Negatives Training Accuracy Time to train (sec) Time to predict (sec)
Majority Voting Ensemble 80/20 0.962 1.54 2.26 1.0
Multilayer Perceptron 80/20 0.953 2.17 2.5 1.0
Extremely Randomized Trees 80/20 0.952 3.96 0.77
CNN 80/20 0.971 160
k-Nearest Neighbors 80/20 0.933 0 6.4 0.97 2.5 0.02
Linear SVM 80/20 0.764 21 2.5 0.81 658 0.01
RBF SVM 80/20 0.724 27.6 0.0 1.0 3571 0.02
Gaussian Process 80/20 0.724 27.6 0.0 0.724 3484 0.01
Decision Tree 80/20 0.726 26.6 0 0.73 3.18 2.0e-06
Random Forest 80/20 0.724 27.6 0 0.724 0.2 3.2e-06
Neural Net 80/20 0.90 7.6 2.3 0.982 142 1.5-06
AdaBoost 80/20 0.725 24.6 2.9 0.734 48.5 0.0001
Gaussian Naive Bayes 80/20 0.353 0 64.3 0.355 1.1 1.8e-05
QDA 80/20 0.947 5.3 0 1.0 12 5.1e-05
Multilayer Perceptron 60/40 0.922 4 3.8
Extremely Randomized Trees 60/40 0.93 5.82 1.16
k-Nearest Neighbors 60/40 0.916 2.27 6.17 0.966
Linear SVM 60/40 0.756 22.2 2.1 0.797
RBF SVM 60/40 0.721 27.9 0 1
Gaussian Process 60/40 0.721 27.9 0 0.72
Decision Tree 60/40 0.723 26 1.64 0.737
Random Forest 60/40 0.721 27.9 0 0.72
Neural Net 60/40 0.9 7 2.95 0.99
AdaBoost 60/40 0.722 24.6 3.15 0.736
Gaussian Naive Bayes 60/40 0.355 0.59 63.9 0.354
QDA 60/40 0.758 22.6 1.58 0.816
Results for input with 36 features
Method Split Accuracy False Positives % False Negatives % Training Accuracy Time to train (sec) Time to predict (sec)
Multilayer Perceptron N/A 0.905 3.4 6.1 0.963
k-Nearest Neighbors 80/20 0.853 6.8 7.9 0.919
Linear SVM 80/20 0.846 0 15.4 0.848
RBF SVM 80/20 0.846 0 15.4 1.0
Decision Tree 80/20 0.846 0 15.4 0.848
Random Forest 80/20 0.846 0 15.4 0.848
Neural Net 80/20 0.873 0.9 11.7 0.874
AdaBoost 80/20 0.846 0 15.4 0.848
Naive Bayes 80/20 0.659 24.8 9.4 0.659
QDR 80/20 0.619 34.4 3.7 0.617

Week 2 & 3

  • Try the models on latest data
  • Change the way we evaluate network accuracy
    • A row labeled with 1 represents the start of a new sample
  • The scope (Gagik)
  • Try and get accuracy with 1/3 of data, 1/2 and all of the data
  • Try to do some more tweaking
Results for new accuracy model
Method/Train set Split Test Set/Format Test Accuracy Training Accuracy Time to train (sec) Time to predict (sec) Figure
CNN/10-1-19 80/20 10-1-19/4032 0.975 0.932 209 0.001
ExtraTrees/10-1-19 N/A 10-1-19/36 1.0 1.0 10.7 1.9
CNN/09-16-19 80/20 09-16-19/4032 0.936 0.937 446 0.002
09-27-19/4032 0.134
CNN/09-27-19 80/20 09-27-19/4032 0.827 0.87 190 0.001
09-16-19/4032 0.487
CNN/09-16+27-19 80/20 09-16+27-19/4032 0.897 0.909 630 0.001
09-16-19/4032 0.910
09-27-19/4032 0.845
LinearSVM / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.0 38866.6 0.0
RandomForest / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.512 4.7 0.0
QDA / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.752 0.4 0.0
AdaBoost / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.0 15.7 0.0
DecisionTree / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.601 3.9 0.0
NearestNeighbors / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.563 78.4 0.0
NaiveBayes / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.535 0.1 0.0
RBFSVM / 2019-09-16+27 80/20 2019-09-16+27 / 36 0.0 7297.2 0.0
RandomForest / 2019-09-16+27 80/20 2019-09-16+27 / 4032 0.147 72.8 0.0
AdaBoost / 2019-09-16+27 80/20 2019-09-16+27 / 4032 0.0 990.7 0.0
DecisionTree / 2019-09-16+27 80/20 2019-09-16+27 / 4032 0.362 588.5 0.0
NearestNeighbors / 2019-09-16+27 80/20 2019-09-16+27 / 4032 0.545 31519.1 5.8
NaiveBayes / 2019-09-16+27 80/20 2019-09-16+27 / 4032 0.776 14.7 0.0
RandomForest / 2019-09-27 80/20 2019-09-27 / 6 0.63 0.7 0.0
QDA / 2019-09-27 80/20 2019-09-27 / 6 0.818 0.0 0.0
AdaBoost / 2019-09-27 80/20 2019-09-27 / 6 0.102 3.7 0.0
DecisionTree / 2019-09-27 80/20 2019-09-27 / 6 0.652 0.3 0.0
NearestNeighbors / 2019-09-27 80/20 2019-09-27 / 6 0.763 0.4 0.0
NaiveBayes / 2019-09-27 80/20 2019-09-27 / 6 0.058 0.1 0.0
RandomForest / 2019-09-27 80/20 2019-09-27 / 36 0.447 1.0 0.0
QDA / 2019-09-27 80/20 2019-09-27 / 36 0.218 0.2 0.0
AdaBoost / 2019-09-27 80/20 2019-09-27 / 36 0.012 8.5 0.0
DecisionTree / 2019-09-27 80/20 2019-09-27 / 36 0.528 0.8 0.0
NearestNeighbors / 2019-09-27 80/20 2019-09-27 / 36 0.494 5.5 0.0
NaiveBayes / 2019-09-27 80/20 2019-09-27 / 36 0.311 0.1 0.0
RandomForest / 2019-09-27 80/20 2019-09-27 / 4032 0.123 17.6 0.0
AdaBoost / 2019-09-27 80/20 2019-09-27 / 4032 0.011 229.7 0.0
DecisionTree / 2019-09-27 80/20 2019-09-27 / 4032 0.374 144.4 0.0
NaiveBayes / 2019-09-27 80/20 2019-09-27 / 4032 0.355 3.0 0.0
RandomForest / 2019-09-16 80/20 2019-09-16 / 36 0.522 2.7 0.0
QDA / 2019-09-16 80/20 2019-09-16 / 36 0.801 0.4 0.0
AdaBoost / 2019-09-16 80/20 2019-09-16 / 36 0.0 14.6 0.0
DecisionTree / 2019-09-16 80/20 2019-09-16 / 36 0.609 2.4 0.0
NaiveBayes / 2019-09-16 80/20 2019-09-16 / 36 0.447 0.1 0.0
RandomForest / 2019-09-16 80/20 2019-09-16 / 4032 0.167 47.8 0.0
AdaBoost / 2019-09-16 80/20 2019-09-16 / 4032 0.0 477.1 0.0
DecisionTree / 2019-09-16 80/20 2019-09-16 / 4032 0.383 402.7 0.0
NaiveBayes / 2019-09-16 80/20 2019-09-16 / 4032 0.569 7.9 0.0

Note: Training accuracy is evaluated per data row and not per sample. Testing accuracy is evaluated per sample.
Note 2: The false negatives, in this case, are all the samples that were mispredicted. In another words false negatives % = 1- acc %.

Week 4

Accuracy metrics score for different size of training set using CNNs.
  • Keep more accuracy metrics for our testing procedure, specifically:
  1. The ratio of samples where the valid track was correctly detected. (A1)
  2. Same as 1, but also had invalid tracks miss-predicted as valid ones (False Positives). (Ac)
  3. Same as 1, but also the valid track has the highest probability for label 1 of all rows in the sample. (Ah)
  4. The ration of samples where the valid track was not detected (False Negatives). (Af)
  • Train on fractions of the train set (e.g 20%,40%,60%,80%) and create figures of all the metrics with respect to the fraction used.
Results with multiple accuracy metrics
Method/Train set Split Test Set/Format A1 Ac Ah Af Training Acc Validation Acc Time to train (sec) Time to predict (sec)
CNN/10-7-19 10/90 10-7-19/4032 0.915 0.511 0.832 0.084 0.9389 0.900 199 0.0012
20/80 10-7-19/4032 0.935 0.450 0.863 0.064 0.9310 0.9136 226.1 0.0012
30/70 10-7-19/4032 0.945 0.389 0.878 0.054 0.934 0.924 257.1 0.0012
40/60 10-7-19/4032 0.955 0.382 0.885 0.044 0.934 0.926 288.1 0.001
50/50 10-7-19/4032 0.944 0.331 0.880 0.055 0.934 0.931 319 0.001
60/40 10-7-19/4032 0.954 0.353 0.888 0.045 0.934 0.93 347.4 0.0013
70/30 10-7-19/4032 0.954 0.350 0.885 0.045 0.9343 0.9325 380.8 0.0014
80/20 10-7-19/4032 0.959 0.327 0.892 0.040 0.9344 0.9315 412.4 0.0014
90/10 10-7-19/4032 0.956 0.332 0.880 0.043 0.9347 0.9332 444.3 0.0014
100/0 10-7-19/4032 0.964 0.301 0.894 0.035 0.934 N/A 457 0.0012
ExtraTrees/10-7-19 10/90 10-7-19/6 0.923 0.241 0.914 0.077 1.0 0.92 0.2 0.000005
20/80 10-7-19/6 0.921 0.217 0.914 0.078 1.0 0.924 0.3 0.000005
30/70 10-7-19/6 0.923 0.217 0.912 0.076 1.0 0.925 0.4 0.000005
40/60 10-7-19/6 0.927 0.205 0.914 0.072 1.0 0.927 0.6 0.000005
50/50 10-7-19/6 0.930 0.204 0.916 0.069 1.0 0.928 0.7 0.000005
60/40 10-7-19/6 0.929 0.206 0.918 0.070 1.0 0.915 0.8 0.000005
70/30 10-7-19/6 0.932 0.203 0.92 0.067 1.0 0.93 1.0 0.000005
80/20 10-7-19/6 0.933 0.199 0.918 0.066 1.0 0.917 1.2 0.000005
90/10 10-7-19/6 0.933 0.197 0.919 0.066 0.9999 0.931 1.3 0.000005
100/0 10-7-19/6 0.933 0.199 0.919 0.066 0.9999 N/A 1.7 0.000005
100/0 10-7-19/36 0.910 0.339 0.883 0.089 0.9999 N/A 8.4 0.000005
MLP/10-7-19 100/0 10-7-19/6 0.965 0.202 0.921 0.034 0.947 N/A 252 (CPU) 0.000004
100/0 10-7-19/36 0.920 0.312 0.849 0.079 0.921 N/A 546 (CPU) 0.00001

Observations

As seen in the 100/0 split rows:

  • CNN is 3.1% more accurate at detecting valid samples than the ET
  • ET has 10.1% less detected samples containing false positive predictions (ET has less false positives than the CNN)
  • ET is 2.5% more accurate at giving the highest probability of being 1 to the valid row in a sample than the CNN
  • ET is 269x faster at training than the CNN (for CNN/4032 and ET/6)
  • ET is 240x faster at prediction (inference) than the CNN (for CNN/4032 and ET/6)
  • MLP is 0.1% more accurate at detecting valid samples than the CNN
  • MLP has 9.9% less detected samples containing false positive predictions (MLP has less false positives than the CNN but more than the ET)
  • MLP is 2.6% more accurate at giving the highest probability of being 1 to the valid row in a sample than the CNN
  • MLP is 1.8x faster at training than the CNN (for GPU CNN/4032 and CPU MLP/6)
  • MLP is 300x faster at prediction (inference) than the CNN (for CNN/4032 and MLP/6)
  • MLP and ET perform best on the feature-extracted 6 format

Week 6

  • Deliver the python script that does the training and testing for CNN, MLP, and ExtraTrees.
  • Also, provide a function that can read data from C and pass it back to python.

Week 7

  • Experiment with RNNs and try to generate new data sequences
Preliminary results for RNNs
Method/Train set Test Set/Format Loss Function Training loss Validation loss Time to train (sec) Time to predict (sec)
LSTM/10-7-19 10-7-19 / 36 MAE 1.36
Some random cases demonstrating the performance of the model

Predict 1

Training/Validation loss versus epoch in predict 1 case.

Training Performance:

Loss (MAE): 0.0556 - MSE: 0.0093

Validation Performance:

Loss (MAE): 0.0490 - MSE: 0.0076

Test Performance:

Loss (MAE): 0.050 - MSE: 0.0085

First 9 rows true values vs predicted values: 

[1.575] [1.5128074]
[1.898] [1.9050792]
[4.931] [4.928837]
[2.326] [2.3151793]
[0.566] [0.51875186]
[1.349] [1.3323742]
[0.997] [0.9642346]
[0.759] [0.7421971]
[0.766] [0.7594936]

Predict 2

Training/Validation loss versus epoch in predict 2 case.

Training Performance:

Loss (MAE): 0.0329 - MSE: 0.0042

Validation Performance:

Loss (MAE): 0.0373 - MSE: 0.0042

Test Performance:

Loss (MAE): 0.037 - MSE: 0.0046

First 9 rows true values vs predicted values: 

[1.575 0.457] [1.5281367  0.46965083]
[1.898 0.296] [1.9137108  0.29100114]
[4.931 0.212] [5.0380673  0.21810552]
[2.326 0.261] [2.3030581  0.25453362]
[0.566 0.753] [0.4852956 0.6849611]
[1.349 0.396] [1.3475827  0.39197764]
[0.997 0.556] [0.9059359  0.55739856]
[0.759 0.506] [0.71836287 0.5392053 ]
[0.766 0.6  ] [0.7425585  0.60809374]

Predict 3

Training/Validation loss versus epoch in predict 3 case.

Training Performance:

Loss (MAE): 0.0297 - MSE: 0.0035

Validation Performance:

Loss (MAE): 0.0274 - MSE: 0.0030

Test Performance:

Loss (MAE): 0.0272 - MSE: 0.0033

First 9 rows real values vs predicted values 

[1.575 0.457 0.138] [1.5253325  0.4581103  0.14659211]
[1.898 0.296 0.667] [1.9686979  0.2962818  0.66101694]
[4.931 0.212 0.365] [4.981064   0.21543622 0.3699692 ]
[2.326 0.261 0.095] [2.220339   0.2587005  0.09193501]
[0.566 0.753 1.443] [0.55893284 0.73924404 1.4928124 ]
[1.349 0.396 0.531] [1.3435515  0.39372128 0.5413537 ]
[0.997 0.556 0.621] [1.0042843  0.558294   0.63084584]
[0.759 0.506 0.784] [0.7833784  0.5464319  0.81894845]
[0.766 0.6   0.785] [0.77423286 0.6099417  0.7522259 ]

Relevant Literature

References

Retrieved from "https://crtc.cs.odu.edu/index.php?title=CNF:Machine_Learning&oldid=5541"