Internal:APBNRRMachineLearning

Machine Learning Paper Checklist

• Collect training data
• Categorize and parse training data for machine learning
  • Build program that parses APBNRR output data, categorizes it via human input, and constructs a CSV file for training, evaluation, or prediction
• Research neural network architectures
• Build neural network model with Keras and Tensorflow
• Train model with training data from up to 27 cases
• Evaluate the model with evaluation data from up to 3 cases
• Finalize paper

Frameworks

TensorFlow

TensorFlow™ is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains.

Keras

Keras is a high-level neural network API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

• Allows for easy and fast prototyping (through user friendliness, modularity, and extensibility).
• Supports both convolutional networks and recurrent networks, as well as combinations of the two.
• Runs seamlessly on CPU and GPU.

Scikit-Learn

A machine learning library for Python. It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.

Pandas

An open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. It is probably the most popular tool used for data structures (such as for holding training data) in machine learning.

Feed-Forward Neural Network Architecture Construction Methods

There is generally no standard formula for constructing neural networks. Their architecture heavily varies from problem to problem. There are however ways to optimize a neural network.

Number of Neurons in the Input Layer

The number of neurons in the input layer should be equal to the number of features of the problem.

Number of Neurons in the Output Layer

The number of neurons in the output layer should be equal to the number of labels (targets or result[s] to predict) of the problem.

Number of Hidden Layers

For most problems, one hidden layer is sufficient. Two hidden layers can model any function. More than two hidden layers are rare, but are used when a problem is very complex. Theoretically though, there is no reason to use more than two hidden layers.

Number of Neurons in the Hidden Layers

1. Trial and Error

   The simplest method, where a different number of neurons in each hidden layer is tried successively to reach a lower error rate. There are two main ways of going about doing this:

   1. Forward Approach: Start with few neurons in each layer and progressively add more until a desirable error rate has been achieved.
   2. Backward Approach: Start with many neurons in each layer, and progressively remove neurons until a desirable error rate has been achieved.
2. Rules of Thumb

   There are three main rules of thumb for choosing the number of neurons in a hidden layer:

   1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.
   2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
   3. The number of hidden neurons should be less than twice the size of the input layer.
3. Pruning and Constructive Methods

   Pruning and constructive methods include several automatic algorithms for optimizing the size and speed of a neural network. These methods typically work by incrementally removing the least effective neurons. There are several pruning algorithms, and some of the more well-known are:

   1. Optimal Brain Damage: Works by incrementally removing the neuron that causes the least decrease of training error.
   2. Optimal Brain Surgeon: An improved version of Optimal Brain Damage. Works by incrementally removing the neuron that causes the least decrease of training error.
4. Grid Search

   Brute force different number of neurons. Not practical for many real world problems.

5. Neuroevolution and Genetic Algorithms

   Utilize evolutionary and genetic algorithms that can generate or optimize a neural network's topology and parameters. These methods are relatively newer than the previous ones.

Resources

1. http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf
2. https://www.ijcsmc.com/docs/papers/November2014/V3I11201499a19.pdf
3. https://arxiv.org/pdf/1711.07655.pdf
4. http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/
5. https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464