Internal:APBNRRMachineLearning

Machine Learning Paper Checklist

• [IN PROGRESS] Collect training and evaluation data
• [DONE] Categorize and parse training data for machine learning
  • [DONE] Build program (CSV Generator) that parses APBNRR output data, categorizes it via human input, and constructs a CSV file for training, evaluation, or prediction
• [DONE] Research neural network architectures
• [IN PROGRESS] Build neural network model with Keras and TensorFlow
• Train model with training data from up to 27 cases
• Evaluate the model with evaluation data from up to 3 cases
• Finalize paper

Neural Network Model Iteration 5 (Not Started)

1. Consider utilizing an ensemble of neural networks. This essentially involves training multiple models and then combining their outputs usually by averaging them, producing better results.

Neural Network Ensemble

Neural Network Model Iteration 4 (In Progress)

1. Consider solely utilizing the SGD optimizer, as a finely-tuned SGD typically yields better results that adaptive gradient optimizers such as Adam and Nadam.

Results

  Stochastic Gradient Descent (SGD) Optimizer
  ============================================
  Training MSE: 2.23 (HD)
  Evaluation MSE: 1.29 (HD)

  Nesterov Momentum Adam (Nadam) Optimizer
  ============================================
  Training MSE: 1.74 (HD)
  Evaluation MSE: 1.52 (HD)

Case 11 Actual HD Values Distribution

SGD vs Nadam Predictied HD Values Distribution

• 

  SGD HD Value Predictions Distribution

• 

  Nadam HD Value Predictions Distribution

Notes

1. Using SGD yields superior model generalization than when using Nadam and variants. This is because SGD is more stable and can converge to better minima.

2. Nadam seems to over-fit, which can be inferred from the higher evaluation MSE and lower training MSE compared to SGD but also from the shape of the distribution of predicted HD values. The distribution of predicted HD values when utilizing Nadam is skewed towards the mean of the HD values in case 11, which points to that the model is over-fitting towards that mean. In contrast, the distribution of predicted values using the SGD optimizer is more uniform, which means that the model is more generalized, and not biased towards the mean.

3. The better generalization with SGD can also be seen from the predicted HD value ranges. SGD yielded values in a greater range than when using Nadam, which is similar to the range of the actual HD values of case 11.

Neural Network Model Iteration 3

In this iteration the following tasks have been accomplished:

1. Implemented model serialization to allow reusing the model for later training, evaluation, and prediction.
2. Implemented standalone prediction logic.
3. Added deformation size as a new feature.

Results

  SPECIFIED TUMOR LOCATION & DEFORMATION SIZE
  ============================================
  Training MSE: 1.81 (HD)
  Evaluation MSE: 1.39 (HD)

Case 11 HD Values Prediction Distribution

Notes

1. Introduced deformation size as a new feature to help the model learn better. The deformation size is determined based on the rigid registration error using the below distribution:

  Deformation Size (Based on Rigid Registration HD Error)
  ========================================================
  1 -> Very Small (0 - 4.99)
  2 -> Small (5 - 14.99)
  3 -> Medium (15 - 21.99)
  4 -> Large (22 - 28.99)
  5 -> Very Large (29 - 36.99)
  6 -> Extremely Large (37+)

2. The deformation size definitely helps the model learn better, however it may allow it to overfit (memorize the data). For this reason some anti-overfitting measures like dropout layers may be required.

3. The updated model can now predict HD values in greater ranges, including in the high 1s ranges and as far as up to the 10s range

Neural Network Model Iteration 2

Network Architecture: (IN) 13 x 64 x 64 x 64 x (OUT) 1

Optimizer: Nadam (Nesterov Momentum Adam)

Training Time: ~8 minutes

Epochs (Loops over training data): 8

Training Batch Size: 32 parameter sets

Training Data: Case 8, Case 9, Case 10, Case 12, Case 13, Case 14, Case 17, Case 27 (about 1.98 million total parameter sets)

Evaluation Data: Case 11 (127,440 parameter sets)

Results

  UNSPECIFIED TUMOR LOCATION
  ===========================================
  Training MSE: 4.00 (HD)
  Evaluation MSE: 1.45 (HD)

  SPECIFIED TUMOR LOCATION (4 lobes)
  ============================================
  Training MSE: 4.01 (HD)
  Evaluation MSE: 1.47 (HD)

  SPECIFIED TUMOR LOCATION 
  (4 lobes, left-right, and some combinations)
  ============================================
  Training MSE: 2.92 (HD)
  Evaluation MSE: 1.30 (HD)

  SPECIFIED PATIENT'S GENDER & TUMOR LOCATION 
  (4 lobes, left-right, and some combinations)
  ============================================
  Training MSE: 2.73 (HD)
  Evaluation MSE: 1.42 (HD)

Notes

1. The second iteration of the neural network features a slightly wider architecture, with 11 more neurons per hidden layer. This was done to allow the network to train more effectively.

2. The training data included data from the execution of case 27, which is an extreme case of a glioma in which a quarter of the brain was removed.

3. The training MSE doubled, which is due to the inclusion of case 27.

4. The number of epochs (loops over training data) were increased, as I noticed that the evaluation MSE decreased slightly with a few more epochs. The slightly increased epochs allowed the network to find a better minimum.

5. The training batch size was significantly decreased as a smaller batch size leads to better generalization at the expense of lower training speed. In our case, training does not take long, so a smaller batch size is feasible.

6. The model was executed on the raw parameter sets, as well as on parameter sets for which the location of the tumor and the patient's gender were specified.

7. Not specifying the location of the tumor (or gender) led to a very large training error, as expected. However, the evaluation error was quite low.

8. Specifying the location of the tumor based on the lobe it is located at, did not decrease the MSE. On the contrary, it was slightly increased. I concluded that this feature is not specific or accurate enough for our case. For example, some tumors are not located on only one lobe, but have parts in perhaps two or three.

9. Specifying the location on the tumor based on the lobe it is located at, whether it is on the left or right portion of the lobe as well as if the tumor has portions in multiple lobes led to a substantial decreases in both the training and evaluation MSE.

10. Specifying the location of the tumor based on the lobe it is located at, whether it is on the left or right portion of the lobe, whether the tumor has portions in multiple lobes, as well as the gender of the patient led to a further decrease in the training MSE and a slight increase in the evaluation MSE compared with just specifying the location of the tumor. The gender was introduced as a feature because female and male brains have some structural differences as well as slight differences in volume which could affect results. However, the actual positive impact of specifying the gender is questionable, since it is probably enabling the network to better memorize the training data instead of better understanding it as the evaluation error increased. As such, this feature will probably be removed.

Neural Network Model Iteration 1

Network Architecture: (IN) 13 x 53 x 53 x 53 x (OUT) 1

Optimizer: Nadam (Nesterov Momentum Adam)

Training Time: ~5 minutes

Epochs (Loops over training data): 5

Training Batch Size: 128 parameter sets

Training Data: Case 8, Case 9, Case 10, Case 12, Case 13, Case 14, Case 17 (about 1.93 million total parameter sets)

Evaluation Data: Case 11 (127,440 parameter sets)

1. Using more neurons in the hidden layers improves accuracy

2. Categorizing the data by the location of the tumor had a major impact in the results, summarized below:

  UNSPECIFIED TUMOR LOCATION
  ==========================
  Training MSE: 2.89 (HD)
  Evaluation MSE: 1.24 (HD)

  SPECIFIED TUMOR LOCATION 
  =========================
  Training MSE: 1.81 (HD)
  Evaluation MSE: 1.76 (HD)

As visible above, when specifying the location of the tumor in the data, the MSE in the training data decreased by 1.08. However, when predicting the evaluation data the MSE increased by 0.52. This is probably because there is not enough data for the network to properly establish tumor location relationships. The evaluation data was from Case 11, which is R Frontal with a PTR operation, and the only case with a similar tumor location in the training data was Case 14 which is R Frontal with a CTR operation. The different operation here might also play a role in this difference.

3. It seems that the addition of more descriptive features has a positive overall impact in the accuracy of the neural network

4. Perhaps we need to come up with more features to describe the data

5. Increasing the number of neurons is also probably necessary (widening the network).

It is obvious that the MSE is too high. This is firstly because this early model was trained with data from only 7 cases, and evaluated with only 1 case. However, there are strong indicators that more descriptive features are needed to help the neural network better establish relationships for the data.

There is also a lot of noise in the data, primarily because of the non-deterministic nature of APBNRR when using multiple threads. This randomness in the data hinders the neural network's performance.

Data

Media Paper Table 4

Rigid Registration Error Distribution

Frameworks

TensorFlow

TensorFlow™ is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains.

Keras

Keras is a high-level neural network API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

• Allows for easy and fast prototyping (through user friendliness, modularity, and extensibility).
• Supports both convolutional networks and recurrent networks, as well as combinations of the two.
• Runs seamlessly on CPU and GPU.

Scikit-Learn

A machine learning library for Python. It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.

Pandas

An open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. It is probably the most popular framework used for data structures (such as for holding training data) in machine learning.

Feed-Forward Neural Network Architecture Construction

There is generally no standard formula for constructing neural networks. Their architecture heavily varies from problem to problem. There are however ways to optimize a neural network.

Number of Neurons in the Input Layer

The number of neurons in the input layer should be equal to the number of features (input parameters) of the problem.

Number of Neurons in the Output Layer

The number of neurons in the output layer should be equal to the number of labels (targets or results to predict) of the problem.

Number of Hidden Layers

For most problems, one hidden layer is sufficient. Two hidden layers can model any function. More than two hidden layers are rare, but are used when a problem is very complex. Theoretically though, there is no reason to use more than two hidden layers.

Number of Neurons in the Hidden Layers

1. Trial and Error

   The simplest method, where a different number of neurons in each hidden layer is tried successively to reach a lower error rate. There are two main ways of going about doing this:

   1. Forward Approach: Start with few neurons in each layer and progressively add more until a desirable error rate has been achieved.
   2. Backward Approach: Start with many neurons in each layer, and progressively remove neurons until a desirable error rate has been achieved.
2. Rules of Thumb

   There are three main rules of thumb for choosing the number of neurons in a hidden layer:

   1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.
   2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
   3. The number of hidden neurons should be less than twice the size of the input layer.
3. Pruning and Constructive Methods

   Pruning and constructive methods include several automatic algorithms for optimizing the size and speed of a neural network. These methods typically work by incrementally removing the least effective neurons. There are several pruning algorithms, and some of the more well-known are:

   1. Optimal Brain Damage: Works by incrementally removing the neuron that causes the least decrease of training error.
   2. Optimal Brain Surgeon: An improved version of Optimal Brain Damage. Works by incrementally removing the neuron that causes the least decrease of training error.
4. Grid Search

   Brute force different numbers of neurons. Not practical for many real world problems.

5. Neuroevolution and Genetic Algorithms

   Utilize evolutionary and genetic algorithms that can generate or optimize a neural network's topology and parameters. These methods are relatively newer than the previous ones.

Resources

1. https://towardsdatascience.com/7-practical-deep-learning-tips-97a9f514100e
2. http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf
3. https://www.ijcsmc.com/docs/papers/November2014/V3I11201499a19.pdf
4. https://arxiv.org/pdf/1711.07655.pdf
5. http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
6. https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464
7. https://developers.google.com/machine-learning/crash-course/