Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
This type implements a general-purpose feed-forward neural
network with an arbitrary number of input nodes, hidden layers,
and output nodes. It functions with arbitrary optimizers and hidden layer
types, just as long as these are subclasses of AbstractOptimizer and
AbstractLayer.
num_hidden_layers :
integer– Total number of hidden layers. Equal to `size(hidden_layers)`.
parameter_file_prefix :
character– Prefix of the file from which parameters will be read/written.
num_parameters :
integer– The total number of free parameters in the model. This should
always be set by the `constructor` procedure of the derived type.
num_parameters_mask :
integer– The total number of free parameters after the application of `mask`.
batchsize :
integer– How many samples can pass concurrently through the network.
num_forward_passes :
integer– How many forward passes have been performed since the last call
to `reset_prediction`.
num_backward_passes :
integer– How many backward passes have been performend since the last call to
`reset`.
grad :
real(DP) (dim: (:,:))– Gradients wrt the model input.
Dimension (num_input_nodes x batchsize).
output :
real(DP) (dim: (:,:))– The output of the model. Dimension (num_output_nodes x batchsize)
jacobian :
real(DP) (dim: (:,:,:))– The full Jacobian of the network.
Dimension (num_input_nodes x batchsize x num_output_nodes).
mask :
logical (dim: (:))– A logical array specifying, for each free parameter, whether it
should be optimized or not.
optimizer_idx :
integer– Index of the optimizer that is used to update the free parameters in
the list of all optimizers that were specified for the program.
This is helper variable to map between models and optimizers.
Nodes of different colours represent the following:
Solid arrows point from a derived type to the parent type which it
extends. Dashed arrows point from a derived type to the other
types it contains as a components, with a label listing the name(s) of
said component(s).
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a derived type to the parent type which it
extends. Dashed arrows point from a derived type to the other
types it contains as a components, with a label listing the name(s) of
said component(s).
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
self :
class(NeuralNetwork) [out]– The neural network that will be initialized.
name :
character [in]
num_input_nodes :
integer [in]– The size of the input layer, i.e. the number of input features.
input_node_activations :
character (dim: (num_input_nodes)) [in]– For each input node, a different activation function can be specified.
num_hidden_layers :
integer [in]– The number of hidden layers (that means not including input or output layer.)
num_nodes_per_hidden_layer :
integer (dim: (num_hidden_layers)) [in]– The size of each hidden layer.
hidden_layer_activations :
character (dim: (num_hidden_layers)) [in]– For each hidden layer, a letter specifying the activation function
that is used for all nodes. See `get_activation_function` for more
info.
num_output_nodes :
integer [in]– The size of the output layer. This determines how many values the network
can predict.
output_node_activations :
character (dim: (num_output_nodes)) [in]– For each output node, a different activation function can be specified.
parameter_file_prefix :
character [in]– The first part for the filename from which parameters are read.
batchsize :
integer [in]– How many samples can pass through the network at once.
lsaturation_check :
logical [in]– Whether node saturation is checked during a forward pass.
If True, this can have a very negative effect on performance.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
x :
real(DP) (dim: (self%num_input_nodes,self%batchsize)) [in]– Data that is passed forward through the network.
Data must be padded beforehand to fill the batchsize of `self`.
x :
real(DP) (dim: (self%num_input_nodes,self%batchsize)) [in]– The input data with dimension (num_input_nodes, batchsize).
dx_dq :
real(DP) (dim: (self%num_input_nodes,self%batchsize)) [in]– The gradient of the input vector with respect to one coordinate.
Data that is passed forward through the network.
Data must be padded beforehand to fill the batchsize of `self`.
The Jacobian contains all derivatives of the output neurons
with respect to the input neurons, i.e.
\(J \in\R^{(N_in, N_out)}\)
It is usually needed during inference or in sensitivity analysis.
For example, in a 2G we calculated dE/dG * dG/dxyz where dE/dG is
the Jacobian of the network.
On a technical level, the calculation of the Jacobian is implemented
as self%num_output_neuron backward passes through the entire network.
As we are not interested in the loss gradients, we start with a one-hot
encoding. This is simply a vector that is 1 for the current output node
in the loop and 0 otherwise.
The one-hot encoding ensures that we obtain exactly the gradient of
that output neuron. It is stored by self%backward in self%grad.
We transfer those gradients with dimension (num_input_nodes, batchsize)
to the `jacobian(:, :, iout) array and reset the whole network before
doing the next backward pass for the next output neuron.
It is easy to see that for a single output neuron this routine reduces to
a single call of self%backward with 1.0 as the loss_gradient of each
sample and a copy operation of `self%jacobian(:, :, 1) = self%grad.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
This routine initializes the parameters of the neural network based on
the given initialization methods. See the AbstractLayer class for
more information.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.