The AbstractOptimizer class defines a generic interface for optimizer components,
including:
- MPI-driven distributed gradient collection and parameter updates
- Customizable update routines for child classes through deferred methods
- Integrated support for gradient clipping mechanics
This abstract base class provides the necessary infrastructure to enable diverse
optimization techniques, such as stochastic gradient descent (SGD), Adam, and Kalman filter
methods, each implementing their logic in specialized subclasses.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a submodule to the (sub)module which it is
descended from. Dashed arrows point from a module or program unit to
modules which it uses.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
An optimizer is an object that takes a set of
optimizable parameters and their gradients and then updates those
parameters in the direction of the gradients.
The central routine is step and is defined as part of this
AbstractOptimizer class. step does two things:
1. Collect the provided parameters and their gradients from
all MPI processes.
2. update, i.e. optimize those parameters. This will trigger
the specific implementation of the update routine of the
actual optimizer being used.
The number of optimizers may be very different from the total
number of requested models. For example, users may want to update
all models with a single optimizer or, conversely, update each atomic
neural network of two HDNNs with separate optimizers. The code allows
this kind of experimentation and it is often very useful.
The mapping between optimizers and models is accomplished in mod_initialization
by calling set_optimizers on the models. Afterwards, each model
stores the index of its assigned optimizer(s) (the second dimension of
this array). Optimizers do not store the model index.
Nodes of different colours represent the following:
Solid arrows point from a derived type to the parent type which it
extends. Dashed arrows point from a derived type to the other
types it contains as a components, with a label listing the name(s) of
said component(s).
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a derived type to the parent type which it
extends. Dashed arrows point from a derived type to the other
types it contains as a components, with a label listing the name(s) of
said component(s).
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a derived type to the parent type which it
extends. Dashed arrows point from a derived type to the other
types it contains as a components, with a label listing the name(s) of
said component(s).
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Perform one update step using the instance of AbstractOptimizerMPI.
This is the main update routine that should be called to change
the parameters of a model object.
Before the routine is called, the following tasks must be done:
On root MPI process (this_image = 1): Call model%get_parameters().
Determine whether the model will contribute to the update
on this MPI process (update_local flag).
If 2., call model%get_gradients_wrt_parameters()
This routine will collect the gradients and loss values
from all processes into large arrays that are then passed on to
the update routines of the optimizer children. This requires
some MPI communication of gradients, loss values, and update flags.
After update has terminated, this routine will also broadcast
the updated parameters to all MPI processes.
After step has terminated, the following tasks must be done:
1. On all MPI processes, call model%set_parameters().
5. On all MPI processes, call model%reset().
parameters :
real(DP) (dim: (self%num_parameters)) [inout]– The model parameters on this MPI image. Parameters need only
be sent on the root MPI process, all other images are discarded.
They should always be identical, anyway.
gradients :
real(DP) (dim: (self%num_parameters)) [in]– The gradient of each parameter on this MPI image. This information
must be sent on each image for which `update_flag` is non-zero.
loss :
real(DP) [in]– The loss function value.
update_flag :
integer [in]– Flag whether this image/stream contributes to the update.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Parse optimizer hyperparameters from a single line of the format
optimizer 1 param1=value1 param2=value2 loss_function=mae
or
default_optimizer param1=value1 param2=value2 loss_function=mae
where all parameters are optional. If a parameter is not given,
the type-specific default will be applied.
Child classes need to implement initialize_defaults,
set_hyperparameter, and validate.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.
Nodes of different colours represent the following:
Solid arrows point from a procedure to one which it calls. Dashed
arrows point from an interface to procedures which implement that interface.
This could include the module procedures in a generic interface or the
implementation in a submodule of an interface in a parent module.
Where possible, edges connecting nodes are
given different colours to make them easier to distinguish in
large graphs.