Skip to content

mod_databatcher

Lines: 190 | Visibility: private

Implementation of a type for batching data.

This module provides the DataBatcher class, which handles the logic for organizing data indices into batches.

DataBatcher class, which handles the logic for organizing data indices into batches. It does not handle the physical data directly, but rather the indices pointing to that data.

It supports: 1. Naive batching: Grouping indices purely by count. 2. Grouped batching: Grouping indices based on species or types (e.g., creating batches that only contain specific chemical elements). 3. Splitting: Dividing datasets into training, testing, and validation sets. 4. Shuffling: Reproducible randomization using dedicated random number generators.

Dependencies

Uses

module~~mod_databatcher~~UsesGraph module~mod_databatcher mod_databatcher module~mod_array_helpers mod_array_helpers module~mod_databatcher->module~mod_array_helpers module~mod_mpi mod_mpi module~mod_databatcher->module~mod_mpi module~mod_precision mod_precision module~mod_databatcher->module~mod_precision module~mod_random mod_random module~mod_databatcher->module~mod_random module~mod_throw mod_throw module~mod_databatcher->module~mod_throw module~mod_array_helpers->module~mod_precision module~mod_array_helpers->module~mod_throw module~mod_mpi->module~mod_precision mpi_f08 mpi_f08 module~mod_mpi->mpi_f08 iso_c_binding iso_c_binding module~mod_precision->iso_c_binding iso_fortran_env iso_fortran_env module~mod_precision->iso_fortran_env module~mod_random->module~mod_precision module~mod_constants mod_constants module~mod_random->module~mod_constants module~mod_throw->module~mod_mpi module~mod_throw->module~mod_precision module~mod_throw->iso_c_binding module~mod_throw->iso_fortran_env module~mod_io mod_io module~mod_throw->module~mod_io module~mod_sort mod_sort module~mod_throw->module~mod_sort module~mod_constants->module~mod_precision module~mod_constants->iso_c_binding module~mod_io->module~mod_precision module~mod_io->iso_fortran_env module~mod_optional_args mod_optional_args module~mod_io->module~mod_optional_args module~mod_sort->module~mod_precision

Used By

module~~mod_databatcher~~UsedByGraph module~mod_databatcher mod_databatcher module~mod_create_batches mod_create_batches module~mod_create_batches->module~mod_databatcher module~mod_dataloader_file mod_dataloader_file module~mod_dataloader_file->module~mod_databatcher module~mod_forward_backward_batched mod_forward_backward_batched module~mod_forward_backward_batched->module~mod_databatcher module~mod_forward_backward_gradient_batched mod_forward_backward_gradient_batched module~mod_forward_backward_gradient_batched->module~mod_databatcher module~mod_forward_batched mod_forward_batched module~mod_forward_batched->module~mod_databatcher module~mod_forward_gradient_batched mod_forward_gradient_batched module~mod_forward_gradient_batched->module~mod_databatcher module~mod_predict_4g_charges_training_qeq_multiple_structures mod_predict_4g_charges_training_qeq_multiple_structures module~mod_predict_4g_charges_training_qeq_multiple_structures->module~mod_databatcher module~mod_structure_batch_iterator mod_structure_batch_iterator module~mod_predict_4g_charges_training_qeq_multiple_structures->module~mod_structure_batch_iterator module~mod_predict_4g_short_range_training_qeq_multiple_structures mod_predict_4g_short_range_training_qeq_multiple_structures module~mod_predict_4g_short_range_training_qeq_multiple_structures->module~mod_databatcher module~mod_predict_4g_short_range_training_qeq_multiple_structures->module~mod_structure_batch_iterator module~mod_predict_multiple_structures mod_predict_multiple_structures module~mod_predict_multiple_structures->module~mod_databatcher module~mod_predict_multiple_structures->module~mod_structure_batch_iterator module~mod_two_body mod_two_body module~mod_predict_multiple_structures->module~mod_two_body module~mod_post_processing_wrapper_single_structure mod_post_processing_wrapper_single_structure module~mod_predict_multiple_structures->module~mod_post_processing_wrapper_single_structure module~mod_structure_batch_iterator->module~mod_databatcher module~mod_calc_atomic_properties mod_calc_atomic_properties module~mod_structure_batch_iterator->module~mod_calc_atomic_properties module~mod_train_atomic_property_multiple_atoms mod_train_atomic_property_multiple_atoms module~mod_train_atomic_property_multiple_atoms->module~mod_databatcher module~mod_train_atomic_property_multiple_atoms->module~mod_create_batches module~mod_train_atomic_property_multiple_atoms->module~mod_forward_backward_batched module~mod_train_atomic_property_multiple_atoms->module~mod_forward_batched module~mod_train_test_split mod_train_test_split module~mod_train_test_split->module~mod_databatcher module~mod_calc_atomic_properties->module~mod_forward_batched module~mod_calculate_coulomb_matrix mod_calculate_coulomb_matrix module~mod_calculate_coulomb_matrix->module~mod_dataloader_file module~mod_cost_calculator mod_cost_calculator module~mod_cost_calculator->module~mod_dataloader_file module~mod_write_outfiles_training mod_write_outfiles_training module~mod_cost_calculator->module~mod_write_outfiles_training module~mod_dataset_statistics mod_dataset_statistics module~mod_dataset_statistics->module~mod_dataloader_file module~mod_dataset_statistics->module~mod_forward_batched module~mod_dataset_statistics_chi_all_chunks mod_dataset_statistics_chi_all_chunks module~mod_dataset_statistics_chi_all_chunks->module~mod_dataloader_file module~mod_dataset_statistics_chi_all_chunks->module~mod_dataset_statistics module~mod_feature_map_scaling_all_chunks mod_feature_map_scaling_all_chunks module~mod_feature_map_scaling_all_chunks->module~mod_dataloader_file module~mod_feature_map_scaling_all_chunks->module~mod_structure_batch_iterator module~mod_precalc_nl_deriv_features_single_chunk mod_precalc_nl_deriv_features_single_chunk module~mod_feature_map_scaling_all_chunks->module~mod_precalc_nl_deriv_features_single_chunk module~mod_prediction_statistics mod_prediction_statistics module~mod_feature_map_scaling_all_chunks->module~mod_prediction_statistics module~mod_output_files_and_cost mod_output_files_and_cost module~mod_output_files_and_cost->module~mod_dataloader_file module~mod_output_files_and_cost->module~mod_predict_multiple_structures module~mod_output_files_and_cost->module~mod_cost_calculator module~mod_output_files_and_cost_4g_charges mod_output_files_and_cost_4g_charges module~mod_output_files_and_cost_4g_charges->module~mod_dataloader_file module~mod_output_files_and_cost_4g_charges->module~mod_predict_4g_charges_training_qeq_multiple_structures module~mod_output_files_and_cost_4g_charges->module~mod_cost_calculator module~mod_output_files_and_cost_4g_short_range mod_output_files_and_cost_4g_short_range module~mod_output_files_and_cost_4g_short_range->module~mod_dataloader_file module~mod_output_files_and_cost_4g_short_range->module~mod_predict_4g_short_range_training_qeq_multiple_structures module~mod_output_files_and_cost_4g_short_range->module~mod_cost_calculator module~mod_predict_qeq_feature_calculator_multiple_structures mod_predict_qeq_feature_calculator_multiple_structures module~mod_output_files_and_cost_4g_short_range->module~mod_predict_qeq_feature_calculator_multiple_structures module~mod_precalc_coulomb_matrix_all_chunks mod_precalc_coulomb_matrix_all_chunks module~mod_precalc_coulomb_matrix_all_chunks->module~mod_dataloader_file module~mod_precalc_coulomb_matrix_all_chunks->module~mod_forward_batched module~mod_precalc_coulomb_matrix_all_chunks->module~mod_predict_4g_charges_training_qeq_multiple_structures module~mod_precalc_nl_deriv_features_single_chunk->module~mod_dataloader_file module~mod_precompute_features_all_chunks mod_precompute_features_all_chunks module~mod_precompute_features_all_chunks->module~mod_dataloader_file module~mod_precompute_features_all_chunks->module~mod_train_test_split module~mod_precompute_global_features_all_chunks mod_precompute_global_features_all_chunks module~mod_precompute_global_features_all_chunks->module~mod_dataloader_file module~mod_predict_ee4g_feature_calculator_multiple_structures mod_predict_ee4g_feature_calculator_multiple_structures module~mod_precompute_global_features_all_chunks->module~mod_predict_ee4g_feature_calculator_multiple_structures module~mod_precompute_global_features_all_chunks->module~mod_predict_qeq_feature_calculator_multiple_structures module~mod_predict_all_chunks mod_predict_all_chunks module~mod_predict_all_chunks->module~mod_dataloader_file module~mod_predict_all_chunks->module~mod_predict_multiple_structures module~mod_predict_all_chunks->module~mod_two_body module~mod_predict_ee4g_feature_calculator_multiple_structures->module~mod_dataloader_file module~mod_predict_mode mod_predict_mode module~mod_predict_mode->module~mod_dataloader_file module~mod_predict_mode->module~mod_predict_multiple_structures module~mod_predict_mode->module~mod_two_body module~mod_initialize mod_initialize module~mod_predict_mode->module~mod_initialize module~mod_write_outfiles mod_write_outfiles module~mod_predict_mode->module~mod_write_outfiles module~mod_predict_qeq_feature_calculator_multiple_structures->module~mod_dataloader_file module~mod_predict_qeq_feature_calculator_multiple_structures->module~mod_structure_batch_iterator module~mod_train_4g_atomic_charges_structure_batch mod_train_4g_atomic_charges_structure_batch module~mod_train_4g_atomic_charges_structure_batch->module~mod_create_batches module~mod_train_4g_atomic_charges_structure_batch->module~mod_dataloader_file module~mod_train_4g_atomic_charges_structure_batch->module~mod_forward_backward_batched module~mod_train_4g_atomic_charges_structure_batch->module~mod_forward_batched module~mod_train_energy_multiple_structures mod_train_energy_multiple_structures module~mod_train_energy_multiple_structures->module~mod_forward_backward_batched module~mod_train_energy_multiple_structures->module~mod_forward_batched module~mod_train_energy_multiple_structures->module~mod_initialize module~mod_train_forces_multiple_structures mod_train_forces_multiple_structures module~mod_train_forces_multiple_structures->module~mod_forward_backward_gradient_batched module~mod_train_forces_multiple_structures->module~mod_forward_gradient_batched module~mod_train_forces_multiple_structures->module~mod_train_energy_multiple_structures module~mod_train_forces_multiple_structures->module~mod_initialize module~mod_train_loop_4g_charges mod_train_loop_4g_charges module~mod_train_loop_4g_charges->module~mod_dataloader_file module~mod_train_loop_4g_charges->module~mod_cost_calculator module~mod_train_loop_4g_charges->module~mod_output_files_and_cost_4g_charges module~mod_train_loop_4g_charges->module~mod_train_4g_atomic_charges_structure_batch module~mod_train_loop_4g_charges->module~mod_initialize module~mod_train_loop_atomic_property mod_train_loop_atomic_property module~mod_train_loop_atomic_property->module~mod_dataloader_file module~mod_train_loop_atomic_property->module~mod_train_atomic_property_multiple_atoms module~mod_train_loop_atomic_property->module~mod_cost_calculator module~mod_train_loop_atomic_property->module~mod_output_files_and_cost module~mod_train_loop_atomic_property->module~mod_initialize module~mod_train_loop_short_range mod_train_loop_short_range module~mod_train_loop_short_range->module~mod_create_batches module~mod_train_loop_short_range->module~mod_dataloader_file module~mod_train_loop_short_range->module~mod_cost_calculator module~mod_train_loop_short_range->module~mod_output_files_and_cost module~mod_train_loop_short_range->module~mod_output_files_and_cost_4g_short_range module~mod_train_loop_short_range->module~mod_predict_ee4g_feature_calculator_multiple_structures module~mod_train_loop_short_range->module~mod_predict_qeq_feature_calculator_multiple_structures module~mod_train_loop_short_range->module~mod_train_energy_multiple_structures module~mod_train_loop_short_range->module~mod_train_forces_multiple_structures module~mod_train_loop_short_range->module~mod_initialize module~mod_train_mode mod_train_mode module~mod_train_mode->module~mod_dataloader_file module~mod_train_mode->module~mod_predict_4g_charges_training_qeq_multiple_structures module~mod_train_mode->module~mod_train_test_split module~mod_train_mode->module~mod_dataset_statistics module~mod_train_mode->module~mod_dataset_statistics_chi_all_chunks module~mod_train_mode->module~mod_feature_map_scaling_all_chunks module~mod_train_mode->module~mod_precalc_coulomb_matrix_all_chunks module~mod_train_mode->module~mod_precalc_nl_deriv_features_single_chunk module~mod_train_mode->module~mod_predict_all_chunks module~mod_train_mode->module~mod_predict_ee4g_feature_calculator_multiple_structures module~mod_train_mode->module~mod_predict_qeq_feature_calculator_multiple_structures module~mod_train_mode->module~mod_train_loop_4g_charges module~mod_train_mode->module~mod_train_loop_atomic_property module~mod_train_mode->module~mod_train_loop_short_range module~mod_train_mode->module~mod_two_body module~mod_train_mode->module~mod_initialize module~mod_train_mode->module~mod_write_outfiles_training module~mod_two_body->module~mod_dataloader_file program~runner RuNNer program~runner->module~mod_dataloader_file program~runner->module~mod_calculate_coulomb_matrix program~runner->module~mod_precompute_features_all_chunks program~runner->module~mod_precompute_global_features_all_chunks program~runner->module~mod_predict_mode program~runner->module~mod_train_mode program~runner->module~mod_two_body program~runner->module~mod_initialize module~mod_initialize->module~mod_two_body module~mod_post_processing_wrapper_single_structure->module~mod_calc_atomic_properties module~mod_post_processing_wrapper_single_structure->module~mod_two_body module~mod_predict_4g_no_force_trick_single_structure mod_predict_4g_no_force_trick_single_structure module~mod_predict_4g_no_force_trick_single_structure->module~mod_calc_atomic_properties module~mod_predict_single_structure mod_predict_single_structure module~mod_predict_single_structure->module~mod_calc_atomic_properties module~mod_predict_single_structure->module~mod_two_body module~mod_predict_single_structure->module~mod_post_processing_wrapper_single_structure module~mod_prediction_statistics->module~mod_calc_atomic_properties module~mod_runner_interface mod_runner_interface module~mod_runner_interface->module~mod_two_body module~mod_runner_interface_bindings mod_runner_interface_bindings module~mod_runner_interface_bindings->module~mod_calc_atomic_properties module~mod_runner_interface_bindings->module~mod_two_body module~mod_runner_interface_bindings->module~mod_initialize module~mod_runner_interface_bindings->module~mod_runner_interface module~mod_feature_calc_interface mod_feature_calc_interface module~mod_feature_calc_interface->module~mod_runner_interface module~mod_generic_interface mod_generic_interface module~mod_feature_calc_interface->module~mod_generic_interface module~mod_generic_interface->module~mod_predict_single_structure module~mod_generic_interface->module~mod_runner_interface module~mod_lammps_interface mod_lammps_interface module~mod_lammps_interface->module~mod_runner_interface module~mod_lammps_interface->module~mod_runner_interface_bindings module~mod_write_outfiles->module~mod_initialize module~mod_write_outfiles_training->module~mod_write_outfiles

Derived Types Summary

Name Description
DataBatcher

Procedures Summary

Name Type Description
initialize_databatcher Subroutine Initialize a new DataBatcher object. Sets up the internal random number generators.
batch_naive Subroutine Create batches from a given list of indices. This routine performs "naive" batching. It takes a...
batch_grouped Subroutine Create grouped batches from a given list of input samples. This routine is an extension of...
split Subroutine Split a list of indices into an arbitrary number of parts. This is typically used to create...

Derived Types Details

type DataBatcher

Components

Type-Bound Procedures

Name Attributes Binding
initialize public ⇒ initialize_databatcher
batch_naive private ⇒ batch_naive
batch_grouped private ⇒ batch_grouped
batch public ⇒ batch_naive⇒ batch_grouped
split public ⇒ split

Inherits

type~~databatcher~~InheritsGraph type~databatcher DataBatcher type~randomnumbergenerator RandomNumberGenerator type~databatcher->type~randomnumbergenerator rng_shuffle, rng_split

Inherited By

type~~databatcher~~InheritedByGraph type~databatcher DataBatcher type~filedataloader FileDataLoader type~filedataloader->type~databatcher batcher

Procedure Details

subroutine initialize_databatcher

subroutine initialize_databatcher( &
    self, &
    seed_shuffle, &
    seed_split &
)

Initialize a new DataBatcher object.

Sets up the internal random number generators.

Arguments

  • self : class(DataBatcher) [inout]
  • seed_shuffle : integer [in]– The seed used for shuffling indices within batches and shuffeling among batches.
  • seed_split : integer [in]– The seed used for splitting datasets (e.g. train/test splits).

Called By

proc~~initialize_databatcher~~CalledByGraph proc~initialize_databatcher mod_databatcher::DataBatcher%initialize_databatcher proc~initialize_file_dataloader mod_dataloader_file::FileDataLoader%initialize_file_dataloader proc~initialize_file_dataloader->proc~initialize_databatcher program~runner RuNNer program~runner->proc~initialize_file_dataloader

subroutine batch_naive

subroutine batch_naive( &
    self, &
    sample_indices, &
    batchsize, &
    fraction, &
    shuffle, &
    num_batches, &
    num_samples_per_batch, &
    batch_indices &
)

Create batches from a given list of indices.

This routine performs "naive" batching. It takes a 1D list of indices and separates them into smaller batches based solely on the batchsize. It does not discriminate based on the content of the indices (e.g., atom types).

Use this when every sample is structurally equivalent (e.g., batching whole atomic structures).

Arguments

  • self : class(DataBatcher) [inout]
  • sample_indices : integer (dim: (:)) [in]– The indices that will be batched.
  • batchsize : integer [in]– The maximum number of samples in one batch.
  • fraction : real(DP) [in]– The fraction of the input samples to use. If provided, only this percentage of indices will be batched.
  • shuffle : logical [in]– Whether to shuffle the indices before batching. Helpful when using `fraction` to get a random subset. If `shuffle` is false and `fraction` is given, the first samples in `sample_indices` are batched.
  • num_batches : integer [out]– The total number of batches created.
  • num_samples_per_batch : integer (dim: (:)) [out]– The number of samples in each specific batch (size: num_batches).
  • batch_indices : integer (dim: (:,:)) [out]– The actual indices assigned to each batch. Dimensions: batchsize x num_batches

Called By

proc~~batch_naive~~CalledByGraph proc~batch_naive mod_databatcher::DataBatcher%batch_naive none~batch mod_databatcher::DataBatcher%batch none~batch->proc~batch_naive proc~create_atom_batches mod_create_batches::create_atom_batches proc~create_atom_batches->none~batch proc~forward_and_jacobian_batched mod_forward_batched::forward_and_jacobian_batched proc~forward_and_jacobian_batched->none~batch proc~forward_backward_batched_multiple_models mod_forward_backward_batched::forward_backward_batched_multiple_models proc~forward_backward_batched_multiple_models->none~batch proc~forward_backward_batched_single_model mod_forward_backward_batched::forward_backward_batched_single_model proc~forward_backward_batched_single_model->none~batch proc~forward_backward_gradient_batched_multiple_models mod_forward_backward_gradient_batched::forward_backward_gradient_batched_multiple_models proc~forward_backward_gradient_batched_multiple_models->none~batch proc~forward_backward_gradient_batched_single_model mod_forward_backward_gradient_batched::forward_backward_gradient_batched_single_model proc~forward_backward_gradient_batched_single_model->none~batch proc~forward_batched_multiple_models mod_forward_batched::forward_batched_multiple_models proc~forward_batched_multiple_models->none~batch proc~forward_batched_single_model mod_forward_batched::forward_batched_single_model proc~forward_batched_single_model->none~batch proc~forward_gradient_batched_multiple_models mod_forward_gradient_batched::forward_gradient_batched_multiple_models proc~forward_gradient_batched_multiple_models->none~batch proc~forward_gradient_batched_single_model mod_forward_gradient_batched::forward_gradient_batched_single_model proc~forward_gradient_batched_single_model->none~batch proc~init~4 mod_structure_batch_iterator::StructureBatchIterator%init proc~init~4->none~batch proc~train_loop_4g_charges mod_train_loop_4g_charges::train_loop_4g_charges proc~train_loop_4g_charges->none~batch proc~train_4g_atomic_charges_structure_batch mod_train_4g_atomic_charges_structure_batch::train_4g_atomic_charges_structure_batch proc~train_loop_4g_charges->proc~train_4g_atomic_charges_structure_batch proc~output_files_and_cost_4g_charges mod_output_files_and_cost_4g_charges::output_files_and_cost_4g_charges proc~train_loop_4g_charges->proc~output_files_and_cost_4g_charges proc~train_loop_atomic_property mod_train_loop_atomic_property::train_loop_atomic_property proc~train_loop_atomic_property->none~batch proc~train_atomic_property_multiple_atoms mod_train_atomic_property_multiple_atoms::train_atomic_property_multiple_atoms proc~train_loop_atomic_property->proc~train_atomic_property_multiple_atoms proc~output_files_and_cost mod_output_files_and_cost::output_files_and_cost proc~train_loop_atomic_property->proc~output_files_and_cost proc~train_loop_short_range mod_train_loop_short_range::train_loop_short_range proc~train_loop_short_range->none~batch proc~create_force_batches mod_create_batches::create_force_batches proc~train_loop_short_range->proc~create_force_batches proc~predict_qeq_feature_calculator_multiple_structures mod_predict_qeq_feature_calculator_multiple_structures::predict_qeq_feature_calculator_multiple_structures proc~train_loop_short_range->proc~predict_qeq_feature_calculator_multiple_structures proc~train_loop_short_range->proc~output_files_and_cost proc~output_files_and_cost_4g_short_range mod_output_files_and_cost_4g_short_range::output_files_and_cost_4g_short_range proc~train_loop_short_range->proc~output_files_and_cost_4g_short_range proc~train_energy_multiple_structures mod_train_energy_multiple_structures::train_energy_multiple_structures proc~train_loop_short_range->proc~train_energy_multiple_structures proc~train_forces_multiple_structures mod_train_forces_multiple_structures::train_forces_multiple_structures proc~train_loop_short_range->proc~train_forces_multiple_structures interface~forward_backward_batched mod_forward_backward_batched::forward_backward_batched interface~forward_backward_batched->proc~forward_backward_batched_multiple_models interface~forward_backward_batched->proc~forward_backward_batched_single_model interface~forward_backward_gradient_batched mod_forward_backward_gradient_batched::forward_backward_gradient_batched interface~forward_backward_gradient_batched->proc~forward_backward_gradient_batched_multiple_models interface~forward_backward_gradient_batched->proc~forward_backward_gradient_batched_single_model interface~forward_batched mod_forward_batched::forward_batched interface~forward_batched->proc~forward_batched_multiple_models interface~forward_batched->proc~forward_batched_single_model interface~forward_gradient_batched mod_forward_gradient_batched::forward_gradient_batched interface~forward_gradient_batched->proc~forward_gradient_batched_multiple_models interface~forward_gradient_batched->proc~forward_gradient_batched_single_model proc~calc_atomic_properties mod_calc_atomic_properties::calc_atomic_properties proc~calc_atomic_properties->proc~forward_and_jacobian_batched proc~calc_atomic_properties->interface~forward_batched proc~create_force_batches->proc~create_atom_batches proc~predict_4g_charges_training_qeq_multiple_structures mod_predict_4g_charges_training_qeq_multiple_structures::predict_4g_charges_training_qeq_multiple_structures proc~predict_4g_charges_training_qeq_multiple_structures->proc~init~4 proc~predict_structure_batch mod_structure_batch_iterator::StructureBatchIterator%predict_structure_batch proc~predict_4g_charges_training_qeq_multiple_structures->proc~predict_structure_batch proc~predict_4g_short_range_training_qeq_multiple_structures mod_predict_4g_short_range_training_qeq_multiple_structures::predict_4g_short_range_training_qeq_multiple_structures proc~predict_4g_short_range_training_qeq_multiple_structures->proc~init~4 proc~predict_4g_short_range_training_qeq_multiple_structures->proc~predict_structure_batch proc~predict_multiple_structures mod_predict_multiple_structures::predict_multiple_structures proc~predict_multiple_structures->proc~init~4 proc~post_processing_wrapper_single_structure mod_post_processing_wrapper_single_structure::post_processing_wrapper_single_structure proc~predict_multiple_structures->proc~post_processing_wrapper_single_structure proc~predict_multiple_structures->proc~predict_structure_batch proc~predict_qeq_feature_calculator_multiple_structures->proc~init~4 proc~predict_qeq_feature_calculator_multiple_structures->proc~predict_structure_batch proc~prediction_statistics_all_chunks mod_feature_map_scaling_all_chunks::prediction_statistics_all_chunks proc~prediction_statistics_all_chunks->proc~init~4 proc~prediction_statistics_all_chunks->proc~predict_structure_batch proc~train_4g_atomic_charges_structure_batch->proc~create_atom_batches proc~train_4g_atomic_charges_structure_batch->interface~forward_backward_batched proc~train_4g_atomic_charges_structure_batch->interface~forward_batched proc~train_atomic_property_multiple_atoms->proc~create_atom_batches proc~train_atomic_property_multiple_atoms->interface~forward_backward_batched proc~train_atomic_property_multiple_atoms->interface~forward_batched proc~train_mode mod_train_mode::train_mode proc~train_mode->proc~train_loop_4g_charges proc~train_mode->proc~train_loop_atomic_property proc~train_mode->proc~train_loop_short_range proc~train_mode->proc~predict_qeq_feature_calculator_multiple_structures proc~feature_scaling_and_prediction_statistics_all_chunks mod_feature_map_scaling_all_chunks::feature_scaling_and_prediction_statistics_all_chunks proc~train_mode->proc~feature_scaling_and_prediction_statistics_all_chunks proc~precalc_coulomb_matrix_all_chunks mod_precalc_coulomb_matrix_all_chunks::precalc_coulomb_matrix_all_chunks proc~train_mode->proc~precalc_coulomb_matrix_all_chunks proc~predict_all_chunks mod_predict_all_chunks::predict_all_chunks proc~train_mode->proc~predict_all_chunks proc~dataset_statistics_chi_all_chunks mod_dataset_statistics_chi_all_chunks::dataset_statistics_chi_all_chunks proc~train_mode->proc~dataset_statistics_chi_all_chunks proc~feature_scaling_and_prediction_statistics_all_chunks->proc~prediction_statistics_all_chunks proc~output_files_and_cost->proc~predict_multiple_structures proc~output_files_and_cost_4g_charges->proc~predict_4g_charges_training_qeq_multiple_structures proc~output_files_and_cost_4g_short_range->proc~predict_4g_short_range_training_qeq_multiple_structures proc~post_processing_wrapper_single_structure->proc~calc_atomic_properties proc~precalc_coulomb_matrix_all_chunks->interface~forward_batched proc~precompute_global_features_all_chunks mod_precompute_global_features_all_chunks::precompute_global_features_all_chunks proc~precompute_global_features_all_chunks->proc~predict_qeq_feature_calculator_multiple_structures proc~predict_4g_no_force_trick_single_structure mod_predict_4g_no_force_trick_single_structure::predict_4g_no_force_trick_single_structure proc~predict_4g_no_force_trick_single_structure->proc~calc_atomic_properties proc~predict_all_chunks->proc~predict_multiple_structures proc~predict_mode mod_predict_mode::predict_mode proc~predict_mode->proc~predict_multiple_structures proc~predict_single_structure mod_predict_single_structure::predict_single_structure proc~predict_single_structure->proc~calc_atomic_properties proc~predict_single_structure->proc~post_processing_wrapper_single_structure proc~predict_structure_batch->proc~calc_atomic_properties proc~runner_interface_short_range mod_runner_interface_bindings::runner_interface_short_range proc~runner_interface_short_range->proc~calc_atomic_properties proc~runner_interface_short_range_4g mod_runner_interface_bindings::runner_interface_short_range_4g proc~runner_interface_short_range_4g->proc~calc_atomic_properties proc~train_energy_multiple_structures->interface~forward_backward_batched proc~train_energy_multiple_structures->interface~forward_batched proc~train_forces_multiple_structures->interface~forward_backward_gradient_batched proc~train_forces_multiple_structures->interface~forward_gradient_batched proc~train_forces_multiple_structures->proc~train_energy_multiple_structures proc~update_dataset_statistics_chi_multiple_structures mod_dataset_statistics::update_dataset_statistics_chi_multiple_structures proc~update_dataset_statistics_chi_multiple_structures->interface~forward_batched proc~update_prediction_statistics_multiple_structures mod_prediction_statistics::update_prediction_statistics_multiple_structures proc~update_prediction_statistics_multiple_structures->proc~calc_atomic_properties program~runner RuNNer program~runner->proc~train_mode program~runner->proc~precompute_global_features_all_chunks program~runner->proc~predict_mode proc~dataset_statistics_chi_all_chunks->proc~update_dataset_statistics_chi_multiple_structures proc~predict_structure_interface mod_generic_interface::predict_structure_interface proc~predict_structure_interface->proc~predict_single_structure

subroutine batch_grouped

subroutine batch_grouped( &
    self, &
    sample_indices, &
    batchsize, &
    fractions, &
    groups, &
    fraction_groups, &
    shuffle, &
    num_batches, &
    num_samples_per_batch, &
    batch_indices &
)

Create grouped batches from a given list of input samples.

This routine is an extension of batch_naive that respects group identities (e.g., atomic numbers).

Unlike naive batching, where any two samples can end up in the same batch, this routine ensures that a batch only contains samples belonging to the same group.

Arguments

  • self : class(DataBatcher) [inout]
  • sample_indices : integer (dim: (:)) [in]– The primary indices to be batched.
  • batchsize : integer (dim: (:)) [in]– An array of batch sizes. The index of this array corresponds to the group ID. Example: If batching atoms with atomic numbers 1 (H) and 8 (O), `batchsize` must be at least size 8. `batchsize(1)` sets the size for H batches, `batchsize(8)` for O batches.
  • fractions : real(DP) (dim: (:)) [in]– Percentage of samples to use per group. Follows the same indexing rules as `batchsize`.
  • groups : integer (dim: (size(sample_indices))) [in]– An array determining the group ID for each entry in `sample_indices`.
  • fraction_groups : integer (dim: (size(sample_indices))) [in]– Determines which group a sample belongs to when we apply the `fraction`.\ This is helpful if we want to select subset of samples based on their group, but still want to batch all of them together in group-agnostic batches. In this case `groups` may be set to the same value for all samples. Must only be given when `fractions` are given. In most cases, this is chosen to be the same as `groups`.
  • shuffle : logical [in]– Whether to shuffle indices before batching.
  • num_batches : integer [out]– Total number of batches created.
  • num_samples_per_batch : integer (dim: (:)) [out]– Number of samples in each batch.
  • batch_indices : integer (dim: (:,:)) [out]– The indices assigned to each batch.

Called By

proc~~batch_grouped~~CalledByGraph proc~batch_grouped mod_databatcher::DataBatcher%batch_grouped none~batch mod_databatcher::DataBatcher%batch none~batch->proc~batch_grouped proc~create_atom_batches mod_create_batches::create_atom_batches proc~create_atom_batches->none~batch proc~forward_and_jacobian_batched mod_forward_batched::forward_and_jacobian_batched proc~forward_and_jacobian_batched->none~batch proc~forward_backward_batched_multiple_models mod_forward_backward_batched::forward_backward_batched_multiple_models proc~forward_backward_batched_multiple_models->none~batch proc~forward_backward_batched_single_model mod_forward_backward_batched::forward_backward_batched_single_model proc~forward_backward_batched_single_model->none~batch proc~forward_backward_gradient_batched_multiple_models mod_forward_backward_gradient_batched::forward_backward_gradient_batched_multiple_models proc~forward_backward_gradient_batched_multiple_models->none~batch proc~forward_backward_gradient_batched_single_model mod_forward_backward_gradient_batched::forward_backward_gradient_batched_single_model proc~forward_backward_gradient_batched_single_model->none~batch proc~forward_batched_multiple_models mod_forward_batched::forward_batched_multiple_models proc~forward_batched_multiple_models->none~batch proc~forward_batched_single_model mod_forward_batched::forward_batched_single_model proc~forward_batched_single_model->none~batch proc~forward_gradient_batched_multiple_models mod_forward_gradient_batched::forward_gradient_batched_multiple_models proc~forward_gradient_batched_multiple_models->none~batch proc~forward_gradient_batched_single_model mod_forward_gradient_batched::forward_gradient_batched_single_model proc~forward_gradient_batched_single_model->none~batch proc~init~4 mod_structure_batch_iterator::StructureBatchIterator%init proc~init~4->none~batch proc~train_loop_4g_charges mod_train_loop_4g_charges::train_loop_4g_charges proc~train_loop_4g_charges->none~batch proc~train_4g_atomic_charges_structure_batch mod_train_4g_atomic_charges_structure_batch::train_4g_atomic_charges_structure_batch proc~train_loop_4g_charges->proc~train_4g_atomic_charges_structure_batch proc~output_files_and_cost_4g_charges mod_output_files_and_cost_4g_charges::output_files_and_cost_4g_charges proc~train_loop_4g_charges->proc~output_files_and_cost_4g_charges proc~train_loop_atomic_property mod_train_loop_atomic_property::train_loop_atomic_property proc~train_loop_atomic_property->none~batch proc~train_atomic_property_multiple_atoms mod_train_atomic_property_multiple_atoms::train_atomic_property_multiple_atoms proc~train_loop_atomic_property->proc~train_atomic_property_multiple_atoms proc~output_files_and_cost mod_output_files_and_cost::output_files_and_cost proc~train_loop_atomic_property->proc~output_files_and_cost proc~train_loop_short_range mod_train_loop_short_range::train_loop_short_range proc~train_loop_short_range->none~batch proc~create_force_batches mod_create_batches::create_force_batches proc~train_loop_short_range->proc~create_force_batches proc~predict_qeq_feature_calculator_multiple_structures mod_predict_qeq_feature_calculator_multiple_structures::predict_qeq_feature_calculator_multiple_structures proc~train_loop_short_range->proc~predict_qeq_feature_calculator_multiple_structures proc~train_loop_short_range->proc~output_files_and_cost proc~output_files_and_cost_4g_short_range mod_output_files_and_cost_4g_short_range::output_files_and_cost_4g_short_range proc~train_loop_short_range->proc~output_files_and_cost_4g_short_range proc~train_energy_multiple_structures mod_train_energy_multiple_structures::train_energy_multiple_structures proc~train_loop_short_range->proc~train_energy_multiple_structures proc~train_forces_multiple_structures mod_train_forces_multiple_structures::train_forces_multiple_structures proc~train_loop_short_range->proc~train_forces_multiple_structures interface~forward_backward_batched mod_forward_backward_batched::forward_backward_batched interface~forward_backward_batched->proc~forward_backward_batched_multiple_models interface~forward_backward_batched->proc~forward_backward_batched_single_model interface~forward_backward_gradient_batched mod_forward_backward_gradient_batched::forward_backward_gradient_batched interface~forward_backward_gradient_batched->proc~forward_backward_gradient_batched_multiple_models interface~forward_backward_gradient_batched->proc~forward_backward_gradient_batched_single_model interface~forward_batched mod_forward_batched::forward_batched interface~forward_batched->proc~forward_batched_multiple_models interface~forward_batched->proc~forward_batched_single_model interface~forward_gradient_batched mod_forward_gradient_batched::forward_gradient_batched interface~forward_gradient_batched->proc~forward_gradient_batched_multiple_models interface~forward_gradient_batched->proc~forward_gradient_batched_single_model proc~calc_atomic_properties mod_calc_atomic_properties::calc_atomic_properties proc~calc_atomic_properties->proc~forward_and_jacobian_batched proc~calc_atomic_properties->interface~forward_batched proc~create_force_batches->proc~create_atom_batches proc~predict_4g_charges_training_qeq_multiple_structures mod_predict_4g_charges_training_qeq_multiple_structures::predict_4g_charges_training_qeq_multiple_structures proc~predict_4g_charges_training_qeq_multiple_structures->proc~init~4 proc~predict_structure_batch mod_structure_batch_iterator::StructureBatchIterator%predict_structure_batch proc~predict_4g_charges_training_qeq_multiple_structures->proc~predict_structure_batch proc~predict_4g_short_range_training_qeq_multiple_structures mod_predict_4g_short_range_training_qeq_multiple_structures::predict_4g_short_range_training_qeq_multiple_structures proc~predict_4g_short_range_training_qeq_multiple_structures->proc~init~4 proc~predict_4g_short_range_training_qeq_multiple_structures->proc~predict_structure_batch proc~predict_multiple_structures mod_predict_multiple_structures::predict_multiple_structures proc~predict_multiple_structures->proc~init~4 proc~post_processing_wrapper_single_structure mod_post_processing_wrapper_single_structure::post_processing_wrapper_single_structure proc~predict_multiple_structures->proc~post_processing_wrapper_single_structure proc~predict_multiple_structures->proc~predict_structure_batch proc~predict_qeq_feature_calculator_multiple_structures->proc~init~4 proc~predict_qeq_feature_calculator_multiple_structures->proc~predict_structure_batch proc~prediction_statistics_all_chunks mod_feature_map_scaling_all_chunks::prediction_statistics_all_chunks proc~prediction_statistics_all_chunks->proc~init~4 proc~prediction_statistics_all_chunks->proc~predict_structure_batch proc~train_4g_atomic_charges_structure_batch->proc~create_atom_batches proc~train_4g_atomic_charges_structure_batch->interface~forward_backward_batched proc~train_4g_atomic_charges_structure_batch->interface~forward_batched proc~train_atomic_property_multiple_atoms->proc~create_atom_batches proc~train_atomic_property_multiple_atoms->interface~forward_backward_batched proc~train_atomic_property_multiple_atoms->interface~forward_batched proc~train_mode mod_train_mode::train_mode proc~train_mode->proc~train_loop_4g_charges proc~train_mode->proc~train_loop_atomic_property proc~train_mode->proc~train_loop_short_range proc~train_mode->proc~predict_qeq_feature_calculator_multiple_structures proc~feature_scaling_and_prediction_statistics_all_chunks mod_feature_map_scaling_all_chunks::feature_scaling_and_prediction_statistics_all_chunks proc~train_mode->proc~feature_scaling_and_prediction_statistics_all_chunks proc~precalc_coulomb_matrix_all_chunks mod_precalc_coulomb_matrix_all_chunks::precalc_coulomb_matrix_all_chunks proc~train_mode->proc~precalc_coulomb_matrix_all_chunks proc~predict_all_chunks mod_predict_all_chunks::predict_all_chunks proc~train_mode->proc~predict_all_chunks proc~dataset_statistics_chi_all_chunks mod_dataset_statistics_chi_all_chunks::dataset_statistics_chi_all_chunks proc~train_mode->proc~dataset_statistics_chi_all_chunks proc~feature_scaling_and_prediction_statistics_all_chunks->proc~prediction_statistics_all_chunks proc~output_files_and_cost->proc~predict_multiple_structures proc~output_files_and_cost_4g_charges->proc~predict_4g_charges_training_qeq_multiple_structures proc~output_files_and_cost_4g_short_range->proc~predict_4g_short_range_training_qeq_multiple_structures proc~post_processing_wrapper_single_structure->proc~calc_atomic_properties proc~precalc_coulomb_matrix_all_chunks->interface~forward_batched proc~precompute_global_features_all_chunks mod_precompute_global_features_all_chunks::precompute_global_features_all_chunks proc~precompute_global_features_all_chunks->proc~predict_qeq_feature_calculator_multiple_structures proc~predict_4g_no_force_trick_single_structure mod_predict_4g_no_force_trick_single_structure::predict_4g_no_force_trick_single_structure proc~predict_4g_no_force_trick_single_structure->proc~calc_atomic_properties proc~predict_all_chunks->proc~predict_multiple_structures proc~predict_mode mod_predict_mode::predict_mode proc~predict_mode->proc~predict_multiple_structures proc~predict_single_structure mod_predict_single_structure::predict_single_structure proc~predict_single_structure->proc~calc_atomic_properties proc~predict_single_structure->proc~post_processing_wrapper_single_structure proc~predict_structure_batch->proc~calc_atomic_properties proc~runner_interface_short_range mod_runner_interface_bindings::runner_interface_short_range proc~runner_interface_short_range->proc~calc_atomic_properties proc~runner_interface_short_range_4g mod_runner_interface_bindings::runner_interface_short_range_4g proc~runner_interface_short_range_4g->proc~calc_atomic_properties proc~train_energy_multiple_structures->interface~forward_backward_batched proc~train_energy_multiple_structures->interface~forward_batched proc~train_forces_multiple_structures->interface~forward_backward_gradient_batched proc~train_forces_multiple_structures->interface~forward_gradient_batched proc~train_forces_multiple_structures->proc~train_energy_multiple_structures proc~update_dataset_statistics_chi_multiple_structures mod_dataset_statistics::update_dataset_statistics_chi_multiple_structures proc~update_dataset_statistics_chi_multiple_structures->interface~forward_batched proc~update_prediction_statistics_multiple_structures mod_prediction_statistics::update_prediction_statistics_multiple_structures proc~update_prediction_statistics_multiple_structures->proc~calc_atomic_properties program~runner RuNNer program~runner->proc~train_mode program~runner->proc~precompute_global_features_all_chunks program~runner->proc~predict_mode proc~dataset_statistics_chi_all_chunks->proc~update_dataset_statistics_chi_multiple_structures proc~predict_structure_interface mod_generic_interface::predict_structure_interface proc~predict_structure_interface->proc~predict_single_structure

subroutine split

subroutine split( &
    self, &
    sample_indices, &
    num_parts, &
    fractions, &
    shuffle, &
    indices, &
    num_samples_per_part &
)

Split a list of indices into an arbitrary number of parts.

This is typically used to create training, testing, and validation sets from a global list of indices.

The upper limit for the size of each part is estimated based on fractions. If the sum of these limits exceeds the total number of samples due to rounding, the final part is shrunk to fit. This ensures all samples are accounted for, though the final fraction might deviate slightly for very small datasets.

Arguments

  • self : class(DataBatcher) [inout]
  • sample_indices : integer (dim: (:)) [in]– The list of indices to split.
  • num_parts : integer [in]– Number of subdivisions to create.
  • fractions : real(DP) (dim: (num_parts)) [in]– Target fraction of total samples for each part (must sum to 1.0).
  • shuffle : logical [in]– Whether to shuffle indices before splitting. Crucial for random train/test splits.
  • indices : integer (dim: (:,:)) [out]– The indices assigned to each part. Dimensions: max_part_size, num_parts
  • num_samples_per_part : integer (dim: (:)) [out]– Actual count of samples in each part.

Called By

proc~~split~2~~CalledByGraph proc~split~2 mod_databatcher::DataBatcher%split proc~train_test_split mod_train_test_split::train_test_split proc~train_test_split->proc~split~2 proc~precompute_features_all_chunks mod_precompute_features_all_chunks::precompute_features_all_chunks proc~precompute_features_all_chunks->proc~train_test_split proc~train_mode mod_train_mode::train_mode proc~train_mode->proc~train_test_split program~runner RuNNer program~runner->proc~precompute_features_all_chunks program~runner->proc~train_mode