GClasses
|
An artificial neural network. More...
#include <GNeuralNet.h>
Public Types | |
enum | TargetFunction { squared_error, cross_entropy, sign, physical } |
Public Member Functions | |
GNeuralNet (GRand &rand) | |
GNeuralNet (GDomNode *pNode, GLearnerLoader &ll) | |
Load from a text-format. | |
virtual | ~GNeuralNet () |
virtual GDomNode * | serialize (GDom *pDoc) |
Saves the model to a text file. | |
void | setActivationFunction (GActivationFunction *pSF, bool hold) |
Sets the activation function to use with all subsequently added layers. (Note that the activation function for the output layer is set when train or beginIncrementalLearning is called, so if you only wish to set the squshing function for the output layer, call this method after all hidden layers have been added, but before you call train.) If hold is true, then the neural network will hold on to this instance of the activation function and delete it when the neural network is deleted. | |
void | addLayer (size_t nNodes) |
Adds a hidden layer to the network. (The first hidden layer that you add will be adjacent to the input features. The last hidden layer that you add will be adjacent to the output layer.) | |
size_t | layerCount () |
Returns the number of layers in this neural network. (Every network has at least one output layer, plus all of the hidden layers that you add by calling addLayer. The input vector does not count as a layer, even though it may be common to visualize it as a layer.) | |
GNeuralNetLayer & | layer (size_t n) |
Returns a reference to the specified layer. | |
void | addNode (size_t layer) |
Adds a new node at the end of the specified layer. (The new node is initialized with small weights, so this operation should initially have little impact on predictions.) | |
void | dropNode (size_t layer, size_t node) |
Removes the specified node from the specified layer. (An exception will be thrown the layer only has one node.) | |
GBackProp * | backProp () |
Returns the backprop object associated with this neural net (if there is one) | |
void | setValidationPortion (double d) |
Set the portion of the data that will be used for validation. If the value is 0, then all of the data is used for both training and validation. | |
size_t | countWeights () |
Counts the number of weights in the network. (This value is not cached, so you should cache it rather than frequently call this method.) | |
void | perturbAllWeights (double deviation) |
Perturbs all weights in the network by a random normal offset with the specified deviation. | |
void | clipWeights (double max) |
Clips all non-bias weights to fall within the range [-max, max]. | |
void | decayWeights (double lambda, double gamma=1.0) |
Multiplies all non-bias weights by (1.0 - (learning_rate * lambda)), starting with the output layer, and ending with the first hidden layer. Typical values for lambda are small (like 0.001.) After each layer, the value of lambda is multiplied by gamma. (If gamma is greater than 1.0, then weights in hidden layers will decay faster, and if gamma is less than 1.0, then weights in hidden layers will decay slower.) It may be significant to note that if a regularizing penalty is added to the error of lambda times the sum-squared values of non-bias weights, then on-line weight updating works out to the same as decaying the weights after each application of back-prop. | |
void | decayWeightsSingleOutput (size_t output, double lambda) |
Just like decayWeights, except it only decays the weights in one of the output units. | |
double | learningRate () |
Returns the current learning rate. | |
void | setLearningRate (double d) |
Set the learning rate. | |
double | momentum () |
Returns the current momentum value. | |
void | setMomentum (double d) |
Momentum has the effect of speeding convergence and helping the gradient descent algorithm move past some local minimums. | |
double | improvementThresh () |
Returns the threshold ratio for improvement. | |
void | setImprovementThresh (double d) |
Specifies the threshold ratio for improvement that must be made since the last validation check for training to continue. (For example, if the mean squared error at the previous validation check was 50, and the mean squared error at the current validation check is 49, then training will stop if d is > 0.02.) | |
size_t | windowSize () |
Returns the number of epochs to perform before the validation data is evaluated to see if training should stop. | |
void | setWindowSize (size_t n) |
Sets the number of epochs that will be performed before each time the network is tested again with the validation set to determine if we have a better best-set of weights, and whether or not it's achieved the termination condition yet. (An epochs is defined as a single pass through all rows in the training set.) | |
void | setBackPropTargetFunction (TargetFunction eTF) |
Specify the target function to use for back-propagation. The default is squared_error. cross_entropy tends to be faster, and is well-suited for classification tasks. | |
TargetFunction | backPropTargetFunction () |
Returns the enumeration of the target function used for backpropagation. | |
virtual void | trainSparse (GSparseMatrix &features, GMatrix &labels) |
See the comment for GIncrementalLearner::trainSparse Assumes all attributes are continuous. | |
virtual void | clear () |
See the comment for GSupervisedLearner::clear. | |
size_t | trainWithValidation (GMatrix &trainFeatures, GMatrix &trainLabels, GMatrix &validateFeatures, GMatrix &validateLabels) |
Train the network until the termination condition is met. Returns the number of epochs required to train it. | |
void | releaseTrainingJunk () |
Some extra junk is allocated when training to make it efficient. This method is called when training is done to get rid of that extra junk. | |
GMatrix * | internalTraininGMatrix () |
Gets the internal training data set. | |
GMatrix * | internalValidationData () |
Gets the internal validation data set. | |
void | setWeights (const double *pWeights) |
Sets all the weights from an array of doubles. The number of doubles in the array can be determined by calling countWeights(). | |
void | copyWeights (GNeuralNet *pOther) |
Copy the weights from pOther. It is assumed (but not checked) that pOther has the same network structure as this neural network. | |
void | copyStructure (GNeuralNet *pOther) |
Copies the layers, nodes, and settings from pOther (but not the weights). beginIncrementalLearning must have been called on pOther so that it has a complete structure. | |
void | weights (double *pOutWeights) |
Serializes the network weights into an array of doubles. The number of doubles in the array can be determined by calling countWeights(). | |
void | forwardProp (const double *pInputs) |
Evaluates a feature vector. (The results will be in the nodes of the output layer.) | |
double | forwardPropSingleOutput (const double *pInputs, size_t output) |
This is the same as forwardProp, except it only propagates to a single output node. It returns the value that this node outputs. | |
void | copyPrediction (double *pOut) |
This method assumes forwardProp has been called. It copies the predicted vector into pOut. | |
double | sumSquaredPredictionError (const double *pTarget) |
This method assumes forwardProp has been called. It computes the sum squared prediction error with the specified target vector. | |
void | setErrorOnOutputLayer (const double *pTarget, TargetFunction eTargetFunction=squared_error) |
This method assumes that forwardProp has already been called. (Note that the predict method calls forwardProp). It computes the error values at each node in the output layer. After calling this method, it is typical to call backProp()->backpropagate(), to compute the error on the hidden nodes, and then to call backProp()->descendGradient to update the weights. pTarget contains the target values for the ouptut nodes. | |
void | setErrorSingleOutput (double target, size_t output, TargetFunction eTargetFunction=squared_error) |
This is teh same as setErrorOnOutputLayer, except that it only sets the error on a single output node. | |
void | autoTune (GMatrix &features, GMatrix &labels) |
Uses cross-validation to find a set of parameters that works well with the provided data. That is, this method will add a good number of hidden layers, pick a good momentum value, etc. | |
void | setUseInputBias (bool b) |
Specify whether to use an input bias. (The default is false.) This feature is used with generative-backpropagation, which adjusts inputs to create latent features. | |
bool | useInputBias () |
Returns whether this neural network utilizes an input bias. | |
bool | hasTrainingBegun () |
Returns true iff train or beginIncrementalTraining has been called. | |
void | swapNodes (size_t layer, size_t a, size_t b) |
Swaps two nodes in the specified layer. If layer specifies one of the hidden layers, then this will have no net effect on the output of the network. (Assumes this model is already trained.) | |
void | align (GNeuralNet &that) |
Swaps nodes in hidden layers of this neural network to align with those in that neural network, as determined using bipartite matching. (This might be done, for example, before averaging weights together.) | |
Static Public Member Functions | |
static void | test () |
Performs unit tests for this class. Throws an exception if there is a failure. | |
Protected Member Functions | |
double | validationSquaredError (GMatrix &features, GMatrix &labels) |
Measures the sum squared error against the specified dataset. | |
virtual void | trainInner (GMatrix &features, GMatrix &labels) |
See the comment for GSupervisedLearner::trainInner. | |
virtual void | predictInner (const double *pIn, double *pOut) |
See the comment for GSupervisedLearner::predictInner. | |
virtual void | predictDistributionInner (const double *pIn, GPrediction *pOut) |
See the comment for GSupervisedLearner::predictDistributionInner. | |
virtual bool | canImplicitlyHandleNominalFeatures () |
See the comment for GTransducer::canImplicitlyHandleNominalFeatures. | |
virtual bool | supportedFeatureRange (double *pOutMin, double *pOutMax) |
See the comment for GTransducer::supportedFeatureRange. | |
virtual bool | canImplicitlyHandleMissingFeatures () |
See the comment for GTransducer::canImplicitlyHandleMissingFeatures. | |
virtual bool | canImplicitlyHandleNominalLabels () |
See the comment for GTransducer::canImplicitlyHandleNominalLabels. | |
virtual bool | supportedLabelRange (double *pOutMin, double *pOutMax) |
See the comment for GTransducer::supportedFeatureRange. | |
virtual void | beginIncrementalLearningInner (sp_relation &pFeatureRel, sp_relation &pLabelRel) |
See the comment for GIncrementalLearner::beginIncrementalLearningInner. | |
virtual void | trainIncrementalInner (const double *pIn, const double *pOut) |
See the comment for GIncrementalLearner::trainIncrementalInner. | |
Protected Attributes | |
std::vector< GNeuralNetLayer > | m_layers |
GBackProp * | m_pBackProp |
size_t | m_internalFeatureDims |
size_t | m_internalLabelDims |
std::vector < GActivationFunction * > | m_activationFunctions |
GActivationFunction * | m_pActivationFunction |
double | m_learningRate |
double | m_momentum |
double | m_validationPortion |
double | m_minImprovement |
size_t | m_epochsPerValidationCheck |
TargetFunction | m_backPropTargetFunction |
bool | m_useInputBias |
Friends | |
class | GBackProp |
An artificial neural network.
GClasses::GNeuralNet::GNeuralNet | ( | GRand & | rand | ) |
GClasses::GNeuralNet::GNeuralNet | ( | GDomNode * | pNode, |
GLearnerLoader & | ll | ||
) |
Load from a text-format.
virtual GClasses::GNeuralNet::~GNeuralNet | ( | ) | [virtual] |
void GClasses::GNeuralNet::addLayer | ( | size_t | nNodes | ) |
Adds a hidden layer to the network. (The first hidden layer that you add will be adjacent to the input features. The last hidden layer that you add will be adjacent to the output layer.)
void GClasses::GNeuralNet::addNode | ( | size_t | layer | ) |
Adds a new node at the end of the specified layer. (The new node is initialized with small weights, so this operation should initially have little impact on predictions.)
void GClasses::GNeuralNet::align | ( | GNeuralNet & | that | ) |
Swaps nodes in hidden layers of this neural network to align with those in that neural network, as determined using bipartite matching. (This might be done, for example, before averaging weights together.)
Uses cross-validation to find a set of parameters that works well with the provided data. That is, this method will add a good number of hidden layers, pick a good momentum value, etc.
GBackProp* GClasses::GNeuralNet::backProp | ( | ) | [inline] |
Returns the backprop object associated with this neural net (if there is one)
TargetFunction GClasses::GNeuralNet::backPropTargetFunction | ( | ) | [inline] |
Returns the enumeration of the target function used for backpropagation.
virtual void GClasses::GNeuralNet::beginIncrementalLearningInner | ( | sp_relation & | pFeatureRel, |
sp_relation & | pLabelRel | ||
) | [protected, virtual] |
See the comment for GIncrementalLearner::beginIncrementalLearningInner.
Implements GClasses::GIncrementalLearner.
virtual bool GClasses::GNeuralNet::canImplicitlyHandleMissingFeatures | ( | ) | [inline, protected, virtual] |
See the comment for GTransducer::canImplicitlyHandleMissingFeatures.
Reimplemented from GClasses::GTransducer.
virtual bool GClasses::GNeuralNet::canImplicitlyHandleNominalFeatures | ( | ) | [inline, protected, virtual] |
See the comment for GTransducer::canImplicitlyHandleNominalFeatures.
Reimplemented from GClasses::GTransducer.
virtual bool GClasses::GNeuralNet::canImplicitlyHandleNominalLabels | ( | ) | [inline, protected, virtual] |
See the comment for GTransducer::canImplicitlyHandleNominalLabels.
Reimplemented from GClasses::GTransducer.
virtual void GClasses::GNeuralNet::clear | ( | ) | [inline, virtual] |
See the comment for GSupervisedLearner::clear.
Implements GClasses::GSupervisedLearner.
void GClasses::GNeuralNet::clipWeights | ( | double | max | ) |
Clips all non-bias weights to fall within the range [-max, max].
void GClasses::GNeuralNet::copyPrediction | ( | double * | pOut | ) |
This method assumes forwardProp has been called. It copies the predicted vector into pOut.
void GClasses::GNeuralNet::copyStructure | ( | GNeuralNet * | pOther | ) |
Copies the layers, nodes, and settings from pOther (but not the weights). beginIncrementalLearning must have been called on pOther so that it has a complete structure.
void GClasses::GNeuralNet::copyWeights | ( | GNeuralNet * | pOther | ) |
Copy the weights from pOther. It is assumed (but not checked) that pOther has the same network structure as this neural network.
size_t GClasses::GNeuralNet::countWeights | ( | ) |
Counts the number of weights in the network. (This value is not cached, so you should cache it rather than frequently call this method.)
void GClasses::GNeuralNet::decayWeights | ( | double | lambda, |
double | gamma = 1.0 |
||
) |
Multiplies all non-bias weights by (1.0 - (learning_rate * lambda)), starting with the output layer, and ending with the first hidden layer. Typical values for lambda are small (like 0.001.) After each layer, the value of lambda is multiplied by gamma. (If gamma is greater than 1.0, then weights in hidden layers will decay faster, and if gamma is less than 1.0, then weights in hidden layers will decay slower.) It may be significant to note that if a regularizing penalty is added to the error of lambda times the sum-squared values of non-bias weights, then on-line weight updating works out to the same as decaying the weights after each application of back-prop.
void GClasses::GNeuralNet::decayWeightsSingleOutput | ( | size_t | output, |
double | lambda | ||
) |
Just like decayWeights, except it only decays the weights in one of the output units.
void GClasses::GNeuralNet::dropNode | ( | size_t | layer, |
size_t | node | ||
) |
Removes the specified node from the specified layer. (An exception will be thrown the layer only has one node.)
void GClasses::GNeuralNet::forwardProp | ( | const double * | pInputs | ) |
Evaluates a feature vector. (The results will be in the nodes of the output layer.)
double GClasses::GNeuralNet::forwardPropSingleOutput | ( | const double * | pInputs, |
size_t | output | ||
) |
This is the same as forwardProp, except it only propagates to a single output node. It returns the value that this node outputs.
bool GClasses::GNeuralNet::hasTrainingBegun | ( | ) | [inline] |
Returns true iff train or beginIncrementalTraining has been called.
double GClasses::GNeuralNet::improvementThresh | ( | ) | [inline] |
Returns the threshold ratio for improvement.
GMatrix* GClasses::GNeuralNet::internalTraininGMatrix | ( | ) |
Gets the internal training data set.
GMatrix* GClasses::GNeuralNet::internalValidationData | ( | ) |
Gets the internal validation data set.
GNeuralNetLayer& GClasses::GNeuralNet::layer | ( | size_t | n | ) | [inline] |
Returns a reference to the specified layer.
size_t GClasses::GNeuralNet::layerCount | ( | ) | [inline] |
Returns the number of layers in this neural network. (Every network has at least one output layer, plus all of the hidden layers that you add by calling addLayer. The input vector does not count as a layer, even though it may be common to visualize it as a layer.)
double GClasses::GNeuralNet::learningRate | ( | ) | [inline] |
Returns the current learning rate.
double GClasses::GNeuralNet::momentum | ( | ) | [inline] |
Returns the current momentum value.
void GClasses::GNeuralNet::perturbAllWeights | ( | double | deviation | ) |
Perturbs all weights in the network by a random normal offset with the specified deviation.
virtual void GClasses::GNeuralNet::predictDistributionInner | ( | const double * | pIn, |
GPrediction * | pOut | ||
) | [protected, virtual] |
See the comment for GSupervisedLearner::predictDistributionInner.
Implements GClasses::GSupervisedLearner.
virtual void GClasses::GNeuralNet::predictInner | ( | const double * | pIn, |
double * | pOut | ||
) | [protected, virtual] |
See the comment for GSupervisedLearner::predictInner.
Implements GClasses::GSupervisedLearner.
void GClasses::GNeuralNet::releaseTrainingJunk | ( | ) |
Some extra junk is allocated when training to make it efficient. This method is called when training is done to get rid of that extra junk.
Saves the model to a text file.
Implements GClasses::GSupervisedLearner.
void GClasses::GNeuralNet::setActivationFunction | ( | GActivationFunction * | pSF, |
bool | hold | ||
) |
Sets the activation function to use with all subsequently added layers. (Note that the activation function for the output layer is set when train or beginIncrementalLearning is called, so if you only wish to set the squshing function for the output layer, call this method after all hidden layers have been added, but before you call train.) If hold is true, then the neural network will hold on to this instance of the activation function and delete it when the neural network is deleted.
void GClasses::GNeuralNet::setBackPropTargetFunction | ( | TargetFunction | eTF | ) | [inline] |
Specify the target function to use for back-propagation. The default is squared_error. cross_entropy tends to be faster, and is well-suited for classification tasks.
void GClasses::GNeuralNet::setErrorOnOutputLayer | ( | const double * | pTarget, |
TargetFunction | eTargetFunction = squared_error |
||
) |
This method assumes that forwardProp has already been called. (Note that the predict method calls forwardProp). It computes the error values at each node in the output layer. After calling this method, it is typical to call backProp()->backpropagate(), to compute the error on the hidden nodes, and then to call backProp()->descendGradient to update the weights. pTarget contains the target values for the ouptut nodes.
void GClasses::GNeuralNet::setErrorSingleOutput | ( | double | target, |
size_t | output, | ||
TargetFunction | eTargetFunction = squared_error |
||
) |
This is teh same as setErrorOnOutputLayer, except that it only sets the error on a single output node.
void GClasses::GNeuralNet::setImprovementThresh | ( | double | d | ) | [inline] |
Specifies the threshold ratio for improvement that must be made since the last validation check for training to continue. (For example, if the mean squared error at the previous validation check was 50, and the mean squared error at the current validation check is 49, then training will stop if d is > 0.02.)
void GClasses::GNeuralNet::setLearningRate | ( | double | d | ) | [inline] |
Set the learning rate.
void GClasses::GNeuralNet::setMomentum | ( | double | d | ) | [inline] |
Momentum has the effect of speeding convergence and helping the gradient descent algorithm move past some local minimums.
void GClasses::GNeuralNet::setUseInputBias | ( | bool | b | ) | [inline] |
Specify whether to use an input bias. (The default is false.) This feature is used with generative-backpropagation, which adjusts inputs to create latent features.
void GClasses::GNeuralNet::setValidationPortion | ( | double | d | ) | [inline] |
Set the portion of the data that will be used for validation. If the value is 0, then all of the data is used for both training and validation.
void GClasses::GNeuralNet::setWeights | ( | const double * | pWeights | ) |
Sets all the weights from an array of doubles. The number of doubles in the array can be determined by calling countWeights().
void GClasses::GNeuralNet::setWindowSize | ( | size_t | n | ) | [inline] |
Sets the number of epochs that will be performed before each time the network is tested again with the validation set to determine if we have a better best-set of weights, and whether or not it's achieved the termination condition yet. (An epochs is defined as a single pass through all rows in the training set.)
double GClasses::GNeuralNet::sumSquaredPredictionError | ( | const double * | pTarget | ) |
This method assumes forwardProp has been called. It computes the sum squared prediction error with the specified target vector.
virtual bool GClasses::GNeuralNet::supportedFeatureRange | ( | double * | pOutMin, |
double * | pOutMax | ||
) | [protected, virtual] |
See the comment for GTransducer::supportedFeatureRange.
Reimplemented from GClasses::GTransducer.
virtual bool GClasses::GNeuralNet::supportedLabelRange | ( | double * | pOutMin, |
double * | pOutMax | ||
) | [protected, virtual] |
See the comment for GTransducer::supportedFeatureRange.
Reimplemented from GClasses::GTransducer.
void GClasses::GNeuralNet::swapNodes | ( | size_t | layer, |
size_t | a, | ||
size_t | b | ||
) |
Swaps two nodes in the specified layer. If layer specifies one of the hidden layers, then this will have no net effect on the output of the network. (Assumes this model is already trained.)
static void GClasses::GNeuralNet::test | ( | ) | [static] |
Performs unit tests for this class. Throws an exception if there is a failure.
Reimplemented from GClasses::GSupervisedLearner.
virtual void GClasses::GNeuralNet::trainIncrementalInner | ( | const double * | pIn, |
const double * | pOut | ||
) | [protected, virtual] |
See the comment for GIncrementalLearner::trainIncrementalInner.
Implements GClasses::GIncrementalLearner.
virtual void GClasses::GNeuralNet::trainInner | ( | GMatrix & | features, |
GMatrix & | labels | ||
) | [protected, virtual] |
See the comment for GSupervisedLearner::trainInner.
Implements GClasses::GSupervisedLearner.
virtual void GClasses::GNeuralNet::trainSparse | ( | GSparseMatrix & | features, |
GMatrix & | labels | ||
) | [virtual] |
See the comment for GIncrementalLearner::trainSparse Assumes all attributes are continuous.
Implements GClasses::GIncrementalLearner.
size_t GClasses::GNeuralNet::trainWithValidation | ( | GMatrix & | trainFeatures, |
GMatrix & | trainLabels, | ||
GMatrix & | validateFeatures, | ||
GMatrix & | validateLabels | ||
) |
Train the network until the termination condition is met. Returns the number of epochs required to train it.
bool GClasses::GNeuralNet::useInputBias | ( | ) | [inline] |
Returns whether this neural network utilizes an input bias.
double GClasses::GNeuralNet::validationSquaredError | ( | GMatrix & | features, |
GMatrix & | labels | ||
) | [protected] |
Measures the sum squared error against the specified dataset.
void GClasses::GNeuralNet::weights | ( | double * | pOutWeights | ) |
Serializes the network weights into an array of doubles. The number of doubles in the array can be determined by calling countWeights().
size_t GClasses::GNeuralNet::windowSize | ( | ) | [inline] |
Returns the number of epochs to perform before the validation data is evaluated to see if training should stop.
friend class GBackProp [friend] |
std::vector<GActivationFunction*> GClasses::GNeuralNet::m_activationFunctions [protected] |
size_t GClasses::GNeuralNet::m_epochsPerValidationCheck [protected] |
size_t GClasses::GNeuralNet::m_internalFeatureDims [protected] |
size_t GClasses::GNeuralNet::m_internalLabelDims [protected] |
std::vector<GNeuralNetLayer> GClasses::GNeuralNet::m_layers [protected] |
double GClasses::GNeuralNet::m_learningRate [protected] |
double GClasses::GNeuralNet::m_minImprovement [protected] |
double GClasses::GNeuralNet::m_momentum [protected] |
GBackProp* GClasses::GNeuralNet::m_pBackProp [protected] |
bool GClasses::GNeuralNet::m_useInputBias [protected] |
double GClasses::GNeuralNet::m_validationPortion [protected] |