Internal:APBNRRMachineLearning

Machine Learning Paper Checklist

• [IN PROGRESS] Collect training and evaluation data
• Categorize and parse training data for machine learning
  • [DONE] Build program (CSV Generator) that parses APBNRR output data, categorizes it via human input, and constructs a CSV file for training, evaluation, or prediction
• [IN PROGRESS] Research neural network architectures
• [IN PROGRESS] Build neural network model with Keras and TensorFlow
• Train model with training data from up to 27 cases
• Evaluate the model with evaluation data from up to 3 cases
• Finalize paper

Early Neural Network Model Results

Network Architecture: (IN) 13 x 53 x 53 x 53 x (OUT) 1

Training Data: Case 8, Case 9, Case 10, Case 12, Case 13, Case 14, Case 17 (about 2 million parameter sets)

Evaluation Data: Case 11 (127,440 parameter sets)

1. Using more neurons in the hidden layers improves accuracy

2. Categorizing the data by the location of the tumor had a major impact in the results, summarized below:

  NO TUMOR LOCATION FEATURE
  ==========================
  Training MSE: 2.89 (HD)
  Evaluation MSE: 1.24 (HD)

  WITH TUMOR LOCATION FEATURE
  ===========================
  Training MSE: 1.81 (HD)
  Evaluation MSE: 1.76 (HD)

As visible above, when specifying the location of the tumor in the data, the MSE in the training data decreased by 1.08. However, when predicting the evaluation data the MSE increased by 0.52. This is probably because there is not enough data for the network to properly establish tumor location relationships. The evaluation data was from Case 11, which is R Frontal with a PTR operation, and the only case with a similar tumor location in the training data was Case 14 which is R Frontal with a CTR operation. The different operation here might also play a role in this difference.

3. It seems that the addition of more descriptive features has a positive overall impact in the accuracy of the neural network

4. Perhaps we need to come up with more features to describe the data

It is obvious that the MSE is too high. This is firstly because this early model was trained with data from only 7 cases, and evaluated with only 1 case. However, there are strong indicators that more descriptive features are needed to help the neural network better establish relationships for the data.

There is also a lot of noise in the data, primarily because of the non-deterministic nature of APBNRR when using multiple threads. This randomness in the data hinders the neural network's performance.

Frameworks

TensorFlow

TensorFlow™ is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains.

Keras

Keras is a high-level neural network API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

• Allows for easy and fast prototyping (through user friendliness, modularity, and extensibility).
• Supports both convolutional networks and recurrent networks, as well as combinations of the two.
• Runs seamlessly on CPU and GPU.

Scikit-Learn

A machine learning library for Python. It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.

Pandas

An open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. It is probably the most popular framework used for data structures (such as for holding training data) in machine learning.

Feed-Forward Neural Network Architecture Construction

There is generally no standard formula for constructing neural networks. Their architecture heavily varies from problem to problem. There are however ways to optimize a neural network.

Number of Neurons in the Input Layer

The number of neurons in the input layer should be equal to the number of features (input parameters) of the problem.

Number of Neurons in the Output Layer

The number of neurons in the output layer should be equal to the number of labels (targets or results to predict) of the problem.

Number of Hidden Layers

For most problems, one hidden layer is sufficient. Two hidden layers can model any function. More than two hidden layers are rare, but are used when a problem is very complex. Theoretically though, there is no reason to use more than two hidden layers.

Number of Neurons in the Hidden Layers

1. Trial and Error

   The simplest method, where a different number of neurons in each hidden layer is tried successively to reach a lower error rate. There are two main ways of going about doing this:

   1. Forward Approach: Start with few neurons in each layer and progressively add more until a desirable error rate has been achieved.
   2. Backward Approach: Start with many neurons in each layer, and progressively remove neurons until a desirable error rate has been achieved.
2. Rules of Thumb

   There are three main rules of thumb for choosing the number of neurons in a hidden layer:

   1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.
   2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
   3. The number of hidden neurons should be less than twice the size of the input layer.
3. Pruning and Constructive Methods

   Pruning and constructive methods include several automatic algorithms for optimizing the size and speed of a neural network. These methods typically work by incrementally removing the least effective neurons. There are several pruning algorithms, and some of the more well-known are:

   1. Optimal Brain Damage: Works by incrementally removing the neuron that causes the least decrease of training error.
   2. Optimal Brain Surgeon: An improved version of Optimal Brain Damage. Works by incrementally removing the neuron that causes the least decrease of training error.
4. Grid Search

   Brute force different numbers of neurons. Not practical for many real world problems.

5. Neuroevolution and Genetic Algorithms

   Utilize evolutionary and genetic algorithms that can generate or optimize a neural network's topology and parameters. These methods are relatively newer than the previous ones.

Resources

1. http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf
2. https://www.ijcsmc.com/docs/papers/November2014/V3I11201499a19.pdf
3. https://arxiv.org/pdf/1711.07655.pdf
4. http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
5. https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464
6. https://developers.google.com/machine-learning/crash-course/