GClasses

GClasses::GNeuralNet Class Reference

An artificial neural network. More...

#include <GNeuralNet.h>

Inheritance diagram for GClasses::GNeuralNet:
GClasses::GIncrementalLearner GClasses::GSupervisedLearner GClasses::GTransducer

List of all members.

Public Types

enum  TargetFunction { squared_error, cross_entropy, sign, physical }

Public Member Functions

 GNeuralNet (GRand &rand)
 GNeuralNet (GDomNode *pNode, GLearnerLoader &ll)
 Load from a text-format.
virtual ~GNeuralNet ()
virtual GDomNodeserialize (GDom *pDoc)
 Saves the model to a text file.
void setActivationFunction (GActivationFunction *pSF, bool hold)
 Sets the activation function to use with all subsequently added layers. (Note that the activation function for the output layer is set when train or beginIncrementalLearning is called, so if you only wish to set the squshing function for the output layer, call this method after all hidden layers have been added, but before you call train.) If hold is true, then the neural network will hold on to this instance of the activation function and delete it when the neural network is deleted.
void addLayer (size_t nNodes)
 Adds a hidden layer to the network. (The first hidden layer that you add will be adjacent to the input features. The last hidden layer that you add will be adjacent to the output layer.)
size_t layerCount ()
 Returns the number of layers in this neural network. (Every network has at least one output layer, plus all of the hidden layers that you add by calling addLayer. The input vector does not count as a layer, even though it may be common to visualize it as a layer.)
GNeuralNetLayerlayer (size_t n)
 Returns a reference to the specified layer.
void addNode (size_t layer)
 Adds a new node at the end of the specified layer. (The new node is initialized with small weights, so this operation should initially have little impact on predictions.)
void dropNode (size_t layer, size_t node)
 Removes the specified node from the specified layer. (An exception will be thrown the layer only has one node.)
GBackPropbackProp ()
 Returns the backprop object associated with this neural net (if there is one)
void setValidationPortion (double d)
 Set the portion of the data that will be used for validation. If the value is 0, then all of the data is used for both training and validation.
size_t countWeights ()
 Counts the number of weights in the network. (This value is not cached, so you should cache it rather than frequently call this method.)
void perturbAllWeights (double deviation)
 Perturbs all weights in the network by a random normal offset with the specified deviation.
void clipWeights (double max)
 Clips all non-bias weights to fall within the range [-max, max].
void decayWeights (double lambda, double gamma=1.0)
 Multiplies all non-bias weights by (1.0 - (learning_rate * lambda)), starting with the output layer, and ending with the first hidden layer. Typical values for lambda are small (like 0.001.) After each layer, the value of lambda is multiplied by gamma. (If gamma is greater than 1.0, then weights in hidden layers will decay faster, and if gamma is less than 1.0, then weights in hidden layers will decay slower.) It may be significant to note that if a regularizing penalty is added to the error of lambda times the sum-squared values of non-bias weights, then on-line weight updating works out to the same as decaying the weights after each application of back-prop.
void decayWeightsSingleOutput (size_t output, double lambda)
 Just like decayWeights, except it only decays the weights in one of the output units.
double learningRate ()
 Returns the current learning rate.
void setLearningRate (double d)
 Set the learning rate.
double momentum ()
 Returns the current momentum value.
void setMomentum (double d)
 Momentum has the effect of speeding convergence and helping the gradient descent algorithm move past some local minimums.
double improvementThresh ()
 Returns the threshold ratio for improvement.
void setImprovementThresh (double d)
 Specifies the threshold ratio for improvement that must be made since the last validation check for training to continue. (For example, if the mean squared error at the previous validation check was 50, and the mean squared error at the current validation check is 49, then training will stop if d is > 0.02.)
size_t windowSize ()
 Returns the number of epochs to perform before the validation data is evaluated to see if training should stop.
void setWindowSize (size_t n)
 Sets the number of epochs that will be performed before each time the network is tested again with the validation set to determine if we have a better best-set of weights, and whether or not it's achieved the termination condition yet. (An epochs is defined as a single pass through all rows in the training set.)
void setBackPropTargetFunction (TargetFunction eTF)
 Specify the target function to use for back-propagation. The default is squared_error. cross_entropy tends to be faster, and is well-suited for classification tasks.
TargetFunction backPropTargetFunction ()
 Returns the enumeration of the target function used for backpropagation.
virtual void trainSparse (GSparseMatrix &features, GMatrix &labels)
 See the comment for GIncrementalLearner::trainSparse Assumes all attributes are continuous.
virtual void clear ()
 See the comment for GSupervisedLearner::clear.
size_t trainWithValidation (GMatrix &trainFeatures, GMatrix &trainLabels, GMatrix &validateFeatures, GMatrix &validateLabels)
 Train the network until the termination condition is met. Returns the number of epochs required to train it.
void releaseTrainingJunk ()
 Some extra junk is allocated when training to make it efficient. This method is called when training is done to get rid of that extra junk.
GMatrixinternalTraininGMatrix ()
 Gets the internal training data set.
GMatrixinternalValidationData ()
 Gets the internal validation data set.
void setWeights (const double *pWeights)
 Sets all the weights from an array of doubles. The number of doubles in the array can be determined by calling countWeights().
void copyWeights (GNeuralNet *pOther)
 Copy the weights from pOther. It is assumed (but not checked) that pOther has the same network structure as this neural network.
void copyStructure (GNeuralNet *pOther)
 Copies the layers, nodes, and settings from pOther (but not the weights). beginIncrementalLearning must have been called on pOther so that it has a complete structure.
void weights (double *pOutWeights)
 Serializes the network weights into an array of doubles. The number of doubles in the array can be determined by calling countWeights().
void forwardProp (const double *pInputs)
 Evaluates a feature vector. (The results will be in the nodes of the output layer.)
double forwardPropSingleOutput (const double *pInputs, size_t output)
 This is the same as forwardProp, except it only propagates to a single output node. It returns the value that this node outputs.
void copyPrediction (double *pOut)
 This method assumes forwardProp has been called. It copies the predicted vector into pOut.
double sumSquaredPredictionError (const double *pTarget)
 This method assumes forwardProp has been called. It computes the sum squared prediction error with the specified target vector.
void setErrorOnOutputLayer (const double *pTarget, TargetFunction eTargetFunction=squared_error)
 This method assumes that forwardProp has already been called. (Note that the predict method calls forwardProp). It computes the error values at each node in the output layer. After calling this method, it is typical to call backProp()->backpropagate(), to compute the error on the hidden nodes, and then to call backProp()->descendGradient to update the weights. pTarget contains the target values for the ouptut nodes.
void setErrorSingleOutput (double target, size_t output, TargetFunction eTargetFunction=squared_error)
 This is teh same as setErrorOnOutputLayer, except that it only sets the error on a single output node.
void autoTune (GMatrix &features, GMatrix &labels)
 Uses cross-validation to find a set of parameters that works well with the provided data. That is, this method will add a good number of hidden layers, pick a good momentum value, etc.
void setUseInputBias (bool b)
 Specify whether to use an input bias. (The default is false.) This feature is used with generative-backpropagation, which adjusts inputs to create latent features.
bool useInputBias ()
 Returns whether this neural network utilizes an input bias.
bool hasTrainingBegun ()
 Returns true iff train or beginIncrementalTraining has been called.
void swapNodes (size_t layer, size_t a, size_t b)
 Swaps two nodes in the specified layer. If layer specifies one of the hidden layers, then this will have no net effect on the output of the network. (Assumes this model is already trained.)
void align (GNeuralNet &that)
 Swaps nodes in hidden layers of this neural network to align with those in that neural network, as determined using bipartite matching. (This might be done, for example, before averaging weights together.)

Static Public Member Functions

static void test ()
 Performs unit tests for this class. Throws an exception if there is a failure.

Protected Member Functions

double validationSquaredError (GMatrix &features, GMatrix &labels)
 Measures the sum squared error against the specified dataset.
virtual void trainInner (GMatrix &features, GMatrix &labels)
 See the comment for GSupervisedLearner::trainInner.
virtual void predictInner (const double *pIn, double *pOut)
 See the comment for GSupervisedLearner::predictInner.
virtual void predictDistributionInner (const double *pIn, GPrediction *pOut)
 See the comment for GSupervisedLearner::predictDistributionInner.
virtual bool canImplicitlyHandleNominalFeatures ()
 See the comment for GTransducer::canImplicitlyHandleNominalFeatures.
virtual bool supportedFeatureRange (double *pOutMin, double *pOutMax)
 See the comment for GTransducer::supportedFeatureRange.
virtual bool canImplicitlyHandleMissingFeatures ()
 See the comment for GTransducer::canImplicitlyHandleMissingFeatures.
virtual bool canImplicitlyHandleNominalLabels ()
 See the comment for GTransducer::canImplicitlyHandleNominalLabels.
virtual bool supportedLabelRange (double *pOutMin, double *pOutMax)
 See the comment for GTransducer::supportedFeatureRange.
virtual void beginIncrementalLearningInner (sp_relation &pFeatureRel, sp_relation &pLabelRel)
 See the comment for GIncrementalLearner::beginIncrementalLearningInner.
virtual void trainIncrementalInner (const double *pIn, const double *pOut)
 See the comment for GIncrementalLearner::trainIncrementalInner.

Protected Attributes

std::vector< GNeuralNetLayerm_layers
GBackPropm_pBackProp
size_t m_internalFeatureDims
size_t m_internalLabelDims
std::vector
< GActivationFunction * > 
m_activationFunctions
GActivationFunctionm_pActivationFunction
double m_learningRate
double m_momentum
double m_validationPortion
double m_minImprovement
size_t m_epochsPerValidationCheck
TargetFunction m_backPropTargetFunction
bool m_useInputBias

Friends

class GBackProp

Detailed Description

An artificial neural network.


Member Enumeration Documentation

Enumerator:
squared_error 
cross_entropy 

(default) best for regression

sign 

best for classification

physical 

uses the sign of the error, as in the perceptron training rule


Constructor & Destructor Documentation

GClasses::GNeuralNet::GNeuralNet ( GRand rand)
GClasses::GNeuralNet::GNeuralNet ( GDomNode pNode,
GLearnerLoader ll 
)

Load from a text-format.

virtual GClasses::GNeuralNet::~GNeuralNet ( ) [virtual]

Member Function Documentation

void GClasses::GNeuralNet::addLayer ( size_t  nNodes)

Adds a hidden layer to the network. (The first hidden layer that you add will be adjacent to the input features. The last hidden layer that you add will be adjacent to the output layer.)

void GClasses::GNeuralNet::addNode ( size_t  layer)

Adds a new node at the end of the specified layer. (The new node is initialized with small weights, so this operation should initially have little impact on predictions.)

void GClasses::GNeuralNet::align ( GNeuralNet that)

Swaps nodes in hidden layers of this neural network to align with those in that neural network, as determined using bipartite matching. (This might be done, for example, before averaging weights together.)

void GClasses::GNeuralNet::autoTune ( GMatrix features,
GMatrix labels 
)

Uses cross-validation to find a set of parameters that works well with the provided data. That is, this method will add a good number of hidden layers, pick a good momentum value, etc.

GBackProp* GClasses::GNeuralNet::backProp ( ) [inline]

Returns the backprop object associated with this neural net (if there is one)

TargetFunction GClasses::GNeuralNet::backPropTargetFunction ( ) [inline]

Returns the enumeration of the target function used for backpropagation.

virtual void GClasses::GNeuralNet::beginIncrementalLearningInner ( sp_relation pFeatureRel,
sp_relation pLabelRel 
) [protected, virtual]
virtual bool GClasses::GNeuralNet::canImplicitlyHandleMissingFeatures ( ) [inline, protected, virtual]
virtual bool GClasses::GNeuralNet::canImplicitlyHandleNominalFeatures ( ) [inline, protected, virtual]
virtual bool GClasses::GNeuralNet::canImplicitlyHandleNominalLabels ( ) [inline, protected, virtual]

See the comment for GTransducer::canImplicitlyHandleNominalLabels.

Reimplemented from GClasses::GTransducer.

virtual void GClasses::GNeuralNet::clear ( ) [inline, virtual]

See the comment for GSupervisedLearner::clear.

Implements GClasses::GSupervisedLearner.

void GClasses::GNeuralNet::clipWeights ( double  max)

Clips all non-bias weights to fall within the range [-max, max].

void GClasses::GNeuralNet::copyPrediction ( double *  pOut)

This method assumes forwardProp has been called. It copies the predicted vector into pOut.

void GClasses::GNeuralNet::copyStructure ( GNeuralNet pOther)

Copies the layers, nodes, and settings from pOther (but not the weights). beginIncrementalLearning must have been called on pOther so that it has a complete structure.

void GClasses::GNeuralNet::copyWeights ( GNeuralNet pOther)

Copy the weights from pOther. It is assumed (but not checked) that pOther has the same network structure as this neural network.

size_t GClasses::GNeuralNet::countWeights ( )

Counts the number of weights in the network. (This value is not cached, so you should cache it rather than frequently call this method.)

void GClasses::GNeuralNet::decayWeights ( double  lambda,
double  gamma = 1.0 
)

Multiplies all non-bias weights by (1.0 - (learning_rate * lambda)), starting with the output layer, and ending with the first hidden layer. Typical values for lambda are small (like 0.001.) After each layer, the value of lambda is multiplied by gamma. (If gamma is greater than 1.0, then weights in hidden layers will decay faster, and if gamma is less than 1.0, then weights in hidden layers will decay slower.) It may be significant to note that if a regularizing penalty is added to the error of lambda times the sum-squared values of non-bias weights, then on-line weight updating works out to the same as decaying the weights after each application of back-prop.

void GClasses::GNeuralNet::decayWeightsSingleOutput ( size_t  output,
double  lambda 
)

Just like decayWeights, except it only decays the weights in one of the output units.

void GClasses::GNeuralNet::dropNode ( size_t  layer,
size_t  node 
)

Removes the specified node from the specified layer. (An exception will be thrown the layer only has one node.)

void GClasses::GNeuralNet::forwardProp ( const double *  pInputs)

Evaluates a feature vector. (The results will be in the nodes of the output layer.)

double GClasses::GNeuralNet::forwardPropSingleOutput ( const double *  pInputs,
size_t  output 
)

This is the same as forwardProp, except it only propagates to a single output node. It returns the value that this node outputs.

bool GClasses::GNeuralNet::hasTrainingBegun ( ) [inline]

Returns true iff train or beginIncrementalTraining has been called.

double GClasses::GNeuralNet::improvementThresh ( ) [inline]

Returns the threshold ratio for improvement.

GMatrix* GClasses::GNeuralNet::internalTraininGMatrix ( )

Gets the internal training data set.

GMatrix* GClasses::GNeuralNet::internalValidationData ( )

Gets the internal validation data set.

GNeuralNetLayer& GClasses::GNeuralNet::layer ( size_t  n) [inline]

Returns a reference to the specified layer.

size_t GClasses::GNeuralNet::layerCount ( ) [inline]

Returns the number of layers in this neural network. (Every network has at least one output layer, plus all of the hidden layers that you add by calling addLayer. The input vector does not count as a layer, even though it may be common to visualize it as a layer.)

double GClasses::GNeuralNet::learningRate ( ) [inline]

Returns the current learning rate.

double GClasses::GNeuralNet::momentum ( ) [inline]

Returns the current momentum value.

void GClasses::GNeuralNet::perturbAllWeights ( double  deviation)

Perturbs all weights in the network by a random normal offset with the specified deviation.

virtual void GClasses::GNeuralNet::predictDistributionInner ( const double *  pIn,
GPrediction pOut 
) [protected, virtual]
virtual void GClasses::GNeuralNet::predictInner ( const double *  pIn,
double *  pOut 
) [protected, virtual]
void GClasses::GNeuralNet::releaseTrainingJunk ( )

Some extra junk is allocated when training to make it efficient. This method is called when training is done to get rid of that extra junk.

virtual GDomNode* GClasses::GNeuralNet::serialize ( GDom pDoc) [virtual]

Saves the model to a text file.

Implements GClasses::GSupervisedLearner.

void GClasses::GNeuralNet::setActivationFunction ( GActivationFunction pSF,
bool  hold 
)

Sets the activation function to use with all subsequently added layers. (Note that the activation function for the output layer is set when train or beginIncrementalLearning is called, so if you only wish to set the squshing function for the output layer, call this method after all hidden layers have been added, but before you call train.) If hold is true, then the neural network will hold on to this instance of the activation function and delete it when the neural network is deleted.

void GClasses::GNeuralNet::setBackPropTargetFunction ( TargetFunction  eTF) [inline]

Specify the target function to use for back-propagation. The default is squared_error. cross_entropy tends to be faster, and is well-suited for classification tasks.

void GClasses::GNeuralNet::setErrorOnOutputLayer ( const double *  pTarget,
TargetFunction  eTargetFunction = squared_error 
)

This method assumes that forwardProp has already been called. (Note that the predict method calls forwardProp). It computes the error values at each node in the output layer. After calling this method, it is typical to call backProp()->backpropagate(), to compute the error on the hidden nodes, and then to call backProp()->descendGradient to update the weights. pTarget contains the target values for the ouptut nodes.

void GClasses::GNeuralNet::setErrorSingleOutput ( double  target,
size_t  output,
TargetFunction  eTargetFunction = squared_error 
)

This is teh same as setErrorOnOutputLayer, except that it only sets the error on a single output node.

void GClasses::GNeuralNet::setImprovementThresh ( double  d) [inline]

Specifies the threshold ratio for improvement that must be made since the last validation check for training to continue. (For example, if the mean squared error at the previous validation check was 50, and the mean squared error at the current validation check is 49, then training will stop if d is > 0.02.)

void GClasses::GNeuralNet::setLearningRate ( double  d) [inline]

Set the learning rate.

void GClasses::GNeuralNet::setMomentum ( double  d) [inline]

Momentum has the effect of speeding convergence and helping the gradient descent algorithm move past some local minimums.

void GClasses::GNeuralNet::setUseInputBias ( bool  b) [inline]

Specify whether to use an input bias. (The default is false.) This feature is used with generative-backpropagation, which adjusts inputs to create latent features.

void GClasses::GNeuralNet::setValidationPortion ( double  d) [inline]

Set the portion of the data that will be used for validation. If the value is 0, then all of the data is used for both training and validation.

void GClasses::GNeuralNet::setWeights ( const double *  pWeights)

Sets all the weights from an array of doubles. The number of doubles in the array can be determined by calling countWeights().

void GClasses::GNeuralNet::setWindowSize ( size_t  n) [inline]

Sets the number of epochs that will be performed before each time the network is tested again with the validation set to determine if we have a better best-set of weights, and whether or not it's achieved the termination condition yet. (An epochs is defined as a single pass through all rows in the training set.)

double GClasses::GNeuralNet::sumSquaredPredictionError ( const double *  pTarget)

This method assumes forwardProp has been called. It computes the sum squared prediction error with the specified target vector.

virtual bool GClasses::GNeuralNet::supportedFeatureRange ( double *  pOutMin,
double *  pOutMax 
) [protected, virtual]

See the comment for GTransducer::supportedFeatureRange.

Reimplemented from GClasses::GTransducer.

virtual bool GClasses::GNeuralNet::supportedLabelRange ( double *  pOutMin,
double *  pOutMax 
) [protected, virtual]

See the comment for GTransducer::supportedFeatureRange.

Reimplemented from GClasses::GTransducer.

void GClasses::GNeuralNet::swapNodes ( size_t  layer,
size_t  a,
size_t  b 
)

Swaps two nodes in the specified layer. If layer specifies one of the hidden layers, then this will have no net effect on the output of the network. (Assumes this model is already trained.)

static void GClasses::GNeuralNet::test ( ) [static]

Performs unit tests for this class. Throws an exception if there is a failure.

Reimplemented from GClasses::GSupervisedLearner.

virtual void GClasses::GNeuralNet::trainIncrementalInner ( const double *  pIn,
const double *  pOut 
) [protected, virtual]
virtual void GClasses::GNeuralNet::trainInner ( GMatrix features,
GMatrix labels 
) [protected, virtual]
virtual void GClasses::GNeuralNet::trainSparse ( GSparseMatrix features,
GMatrix labels 
) [virtual]

See the comment for GIncrementalLearner::trainSparse Assumes all attributes are continuous.

Implements GClasses::GIncrementalLearner.

size_t GClasses::GNeuralNet::trainWithValidation ( GMatrix trainFeatures,
GMatrix trainLabels,
GMatrix validateFeatures,
GMatrix validateLabels 
)

Train the network until the termination condition is met. Returns the number of epochs required to train it.

bool GClasses::GNeuralNet::useInputBias ( ) [inline]

Returns whether this neural network utilizes an input bias.

double GClasses::GNeuralNet::validationSquaredError ( GMatrix features,
GMatrix labels 
) [protected]

Measures the sum squared error against the specified dataset.

void GClasses::GNeuralNet::weights ( double *  pOutWeights)

Serializes the network weights into an array of doubles. The number of doubles in the array can be determined by calling countWeights().

size_t GClasses::GNeuralNet::windowSize ( ) [inline]

Returns the number of epochs to perform before the validation data is evaluated to see if training should stop.


Friends And Related Function Documentation

friend class GBackProp [friend]

Member Data Documentation