CNF:Machine Learning

This is the page for the CNF-Machine Learning project. It will be updated frequently with reports about the progress of the project.

The Problem

Traditional tracking algorithms are computationally intensive, especially for high luminosity experiments with multi-track final states where all combinations of segments in drift chambers have to be considered for firing best track candidates. At high luminosity the number of random segments (unrelated to the tracks) are increasing and as a result the number of possible combinations also increases, making the whole process longer.

The Goal

Using machine learning one can recognize the patterns that are valid in order to find the correct track faster. The model will be trained on real pre-labeled data and as an outcome, it will be able to label track combinations as valid or not.

The Data

A representation of the drift chambers. The red squares represent a sensor that has detected a hit

The drift chambers consist of 6 layers, of 6 wires, of 112 sensors each for a total of 4032 sensors (see picture on the right). The data provided let us know whether a sensor has detected a hit or not. Those detections might be part of the trajectory that we want to track or can be irrelevant (noise). The labeled data consist of all the possible combinations that form a track as rows (events) and the state of each sensor (detected something or not) as columns (features). The label provides information on whether a combination produces the valid track or not. An example of the input to be used by the model can be found in [1].

Data Samples

The testing data are given in samples. A sample consists of multiple rows, the first row which is a valid track (labeled as 1) and all the invalid ones(labeled as 0) that come after it, the next valid track represents the beginning of a new sample. For example, consider the following dataset:

 _labels___ features ____ 
|   1       x11 x12 x13 ...|
|   0       x21 x22 x23 ...|
|   0       x31 x32 x33 ...|
|   0       x41 x42 x43 ...|
|   1       x51 x52 x53 ...|
|   0       x61 x62 x63 ...|
----------------------------

This dataset contains two samples:

       Sample 1                      Sample 2
 _labels___ features ____     _labels___ features ____ 
|   1       x11 x12 x13 ...| |  1       x51 x52 x53 ...|
|   0       x21 x22 x23 ...| |  0       x61 x62 x63 ...|
|   0       x31 x32 x33 ...| --------------------------- 
|   0       x41 x42 x43 ...| 
----------------------------

Predictions and Accuracy

A prediction is performed with respect to a sample as a whole and is considered correct or not based on whether the model was able to identify which of the rows in a sample is a valid track. The accuracy is then measured by counting the number of samples where the model correctly identified the valid track and dividing this number by the total number of samples. In the previous example, if we have the following predicitions:

      Sample 1                                Sample 2
 _labels___ features ____    prediction   _labels___ features ____   prediction
|   1       x11 x12 x13 ...|      1      |  1       x51 x52 x53 ...|     0
|   0       x21 x22 x23 ...|      0      |  0       x61 x62 x63 ...|     1
|   0       x31 x32 x33 ...|      1      --------------------------- 
|   0       x41 x42 x43 ...|      0
----------------------------

The model has correctly predicted the valid track of the 1st sample, but not for the second. So the accuracy will be 50%. Notice that even though we have also mispredicted some invalid tracks as valid ones this does not hurt accuracy since we only care about not losing a valid track.

Scope of Work for 2019-20 Academic Year

To be stated explicitly.

Timeframe

From: https://github.com/gavalian/clas12ai/blob/master/docs/project_scope.txt

• by end of year: Have the software runnable, and implement in the current workflow for our detector reconstruction.
  • October 1: Finalize training and testing procedures. Implement validation procedure to validate test sample, where accuracy of the network is determined from sample of N candidates.
  • December 1: provide software (probably in docker container) that will accept training set as an input and produce a trained neural network. Provide software that can evaluate track candidates and return sorted list ordered by goodness probability. First draft version can be in python.

• Starting early January: start investigating RNNs, divide training sample into three regions, each 12 entries in the training sample represent one input. At this point the objective is to be able to: "predict what next 12 states will be, and based on 24 entries predict what next 12 should be. Investigate networks that can provide best accuracy".
  • Depending on progress (sometime in March) start implementing software that can automate the process of training and prediction.
• By end of Spring complete research and start the writing of a paper to be completed by end of Summer.

Week 1

Create the pipeline

• Load data in Spark Dataframes
• Split data in training/validation sets
• Experiment with different classification methods
  • Tree-based
  • SVM
• Train Model
• Validate
• Report accuracy and confusion matrix

Results for input with 4032 features

Method	Split	Accuracy	False Positives	False Negatives	Training Accuracy	Time to train (sec)	Time to predict (sec)
Majority Voting Ensemble	80/20	0.962	1.54	2.26	1.0
Multilayer Perceptron	80/20	0.953	2.17	2.5	1.0
Extremely Randomized Trees	80/20	0.952	3.96	0.77
CNN	80/20	0.971				160
k-Nearest Neighbors	80/20	0.933	0	6.4	0.97	2.5	0.02
Linear SVM	80/20	0.764	21	2.5	0.81	658	0.01
RBF SVM	80/20	0.724	27.6	0.0	1.0	3571	0.02
Gaussian Process	80/20	0.724	27.6	0.0	0.724	3484	0.01
Decision Tree	80/20	0.726	26.6	0	0.73	3.18	2.0e-06
Random Forest	80/20	0.724	27.6	0	0.724	0.2	3.2e-06
Neural Net	80/20	0.90	7.6	2.3	0.982	142	1.5-06
AdaBoost	80/20	0.725	24.6	2.9	0.734	48.5	0.0001
Gaussian Naive Bayes	80/20	0.353	0	64.3	0.355	1.1	1.8e-05
QDA	80/20	0.947	5.3	0	1.0	12	5.1e-05
Multilayer Perceptron	60/40	0.922	4	3.8
Extremely Randomized Trees	60/40	0.93	5.82	1.16
k-Nearest Neighbors	60/40	0.916	2.27	6.17	0.966
Linear SVM	60/40	0.756	22.2	2.1	0.797
RBF SVM	60/40	0.721	27.9	0	1
Gaussian Process	60/40	0.721	27.9	0	0.72
Decision Tree	60/40	0.723	26	1.64	0.737
Random Forest	60/40	0.721	27.9	0	0.72
Neural Net	60/40	0.9	7	2.95	0.99
AdaBoost	60/40	0.722	24.6	3.15	0.736
Gaussian Naive Bayes	60/40	0.355	0.59	63.9	0.354
QDA	60/40	0.758	22.6	1.58	0.816

Results for input with 36 features

Method	Split	Accuracy	False Positives %	False Negatives %	Training Accuracy	Time to train (sec)	Time to predict (sec)
Multilayer Perceptron	N/A	0.905	3.4	6.1	0.963
k-Nearest Neighbors	80/20	0.853	6.8	7.9	0.919
Linear SVM	80/20	0.846	0	15.4	0.848
RBF SVM	80/20	0.846	0	15.4	1.0
Decision Tree	80/20	0.846	0	15.4	0.848
Random Forest	80/20	0.846	0	15.4	0.848
Neural Net	80/20	0.873	0.9	11.7	0.874
AdaBoost	80/20	0.846	0	15.4	0.848
Naive Bayes	80/20	0.659	24.8	9.4	0.659
QDR	80/20	0.619	34.4	3.7	0.617

Week 2 & 3

• Try the models on latest data
• Change the way we evaluate network accuracy
  • A row labeled with 1 represents the start of a new sample
• The scope (Gagik)
• Try and get accuracy with 1/3 of data, 1/2 and all of the data
• Try to do some more tweaking

Results for new accuracy model

Method/Train set	Split	Dataset/Format	Accuracy	Training Accuracy	Time to train (sec)	Time to predict (sec)	Figure
CNN/09-16-19	80/20	09-16-19/4032	0.936	0.937	446	0.002
		09-27-19/4032	0.134
CNN/09-27-19	80/20	09-27-19/4032	0.827	0.87	190	0.001
		09-16-19/4032	0.487
CNN/09-16+27-19	80/20	09-16+27-19/4032	0.897	0.909	630	0.001
		09-16-19/4032	0.910
		09-27-19/4032	0.845
NeuralNet / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.323		412.5	0.0
LinearSVM / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.0		38866.6	0.0
RandomForest / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.512		4.7	0.0
QDA / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.752		0.4	0.0
AdaBoost / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.0		15.7	0.0
DecisionTree / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.601		3.9	0.0
NearestNeighbors / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.563		78.4	0.0
NaiveBayes / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.535		0.1	0.0
RBFSVM / 2019-09-16+27	80/20	2019-09-16+27 / 36	0.0		7297.2	0.0
NeuralNet / 2019-09-16+27	80/20	2019-09-16+27 / 4032	0.0		1045.3	0.0
RandomForest / 2019-09-16+27	80/20	2019-09-16+27 / 4032	0.147		72.8	0.0
AdaBoost / 2019-09-16+27	80/20	2019-09-16+27 / 4032	0.0		990.7	0.0
DecisionTree / 2019-09-16+27	80/20	2019-09-16+27 / 4032	0.362		588.5	0.0
NearestNeighbors / 2019-09-16+27	80/20	2019-09-16+27 / 4032	0.545		31519.1	5.8
NaiveBayes / 2019-09-16+27	80/20	2019-09-16+27 / 4032	0.776		14.7	0.0
NeuralNet / 2019-09-27	80/20	2019-09-27 / 6	0.0		7.6	0.0
RandomForest / 2019-09-27	80/20	2019-09-27 / 6	0.63		0.7	0.0
QDA / 2019-09-27	80/20	2019-09-27 / 6	0.818		0.0	0.0
AdaBoost / 2019-09-27	80/20	2019-09-27 / 6	0.102		3.7	0.0
DecisionTree / 2019-09-27	80/20	2019-09-27 / 6	0.652		0.3	0.0
NearestNeighbors / 2019-09-27	80/20	2019-09-27 / 6	0.763		0.4	0.0
NaiveBayes / 2019-09-27	80/20	2019-09-27 / 6	0.058		0.1	0.0
NeuralNet / 2019-09-27	80/20	2019-09-27 / 36	0.64		103.3	0.0
RandomForest / 2019-09-27	80/20	2019-09-27 / 36	0.447		1.0	0.0
QDA / 2019-09-27	80/20	2019-09-27 / 36	0.218		0.2	0.0
AdaBoost / 2019-09-27	80/20	2019-09-27 / 36	0.012		8.5	0.0
DecisionTree / 2019-09-27	80/20	2019-09-27 / 36	0.528		0.8	0.0
NearestNeighbors / 2019-09-27	80/20	2019-09-27 / 36	0.494		5.5	0.0
NaiveBayes / 2019-09-27	80/20	2019-09-27 / 36	0.311		0.1	0.0
NeuralNet / 2019-09-27	80/20	2019-09-27 / 4032	0.098		420.1	0.0
RandomForest / 2019-09-27	80/20	2019-09-27 / 4032	0.123		17.6	0.0
AdaBoost / 2019-09-27	80/20	2019-09-27 / 4032	0.011		229.7	0.0
DecisionTree / 2019-09-27	80/20	2019-09-27 / 4032	0.374		144.4	0.0
NaiveBayes / 2019-09-27	80/20	2019-09-27 / 4032	0.355		3.0	0.0
NeuralNet / 2019-09-16	80/20	2019-09-16 / 36	0.411		211.2	0.0
RandomForest / 2019-09-16	80/20	2019-09-16 / 36	0.522		2.7	0.0
QDA / 2019-09-16	80/20	2019-09-16 / 36	0.801		0.4	0.0
AdaBoost / 2019-09-16	80/20	2019-09-16 / 36	0.0		14.6	0.0
DecisionTree / 2019-09-16	80/20	2019-09-16 / 36	0.609		2.4	0.0
NaiveBayes / 2019-09-16	80/20	2019-09-16 / 36	0.447		0.1	0.0
NeuralNet / 2019-09-16	80/20	2019-09-16 / 4032	0.0		1071.3	0.0
RandomForest / 2019-09-16	80/20	2019-09-16 / 4032	0.167		47.8	0.0
AdaBoost / 2019-09-16	80/20	2019-09-16 / 4032	0.0		477.1	0.0
DecisionTree / 2019-09-16	80/20	2019-09-16 / 4032	0.383		402.7	0.0
NaiveBayes / 2019-09-16	80/20	2019-09-16 / 4032	0.569		7.9	0.0

Note: Training accuracy is evaluated per data row and not per sample. Testing accuracy is evaluated per sample.

References

• https://userweb.jlab.org/~gavalian/ML/data/dc/dc_training_svm.txt
• https://userweb.jlab.org/~gavalian/presentations/ML-Tracking.pdf
• https://deeplearning4j.org/docs/latest/deeplearning4j-cheat-sheet
• https://userweb.jlab.org/~gavalian/ML/data/