Skip to content

The Continuous Integration Pipeline

Whenever a user pushes a commit to the Gitlab repository of the project, an automated Continuous Integration pipeline (CI pipeline) is triggered. Our CI pipeline fulfils many purposes that are grouped together in so-called stages. Each stage consists of multiple jobs, that often depend on each other.

The CI pipeline is configured in the .gitlab-ci.yml Please also take a look at the comments in this file for detailed information on the setup of the CI pipeline.

RuNNer Bot

Many jobs require cloning certain private repositories. For this purpose, we have set up a Gitlab user called "RuNNer Bot". His personal Gitlab token (necessary CI variable PROJECT_BOT_TOKEN) can be modified in the Gitlab settings of the project under 'CI/CD -> Variables'. It has to be updated once a year, due to Gitlab guidelines.

The pipeline stages

build stage

In the build stage, the code is built using both make and cmake, different combinations of compilers (ifx, ifort, gfortran) and libraries (MKL, OpenBLAS, etc.), and on Linux and MacOS platforms. The goals of this stage are:

  • make sure that the code compiles and can be linked on all platforms
  • generate libraries and executables for the current commit that can be used for testing in the upcoming stages.

For this reason, the generated executables are left behind as CI artifacts.

runner-ci-general container

Most build and test stages run in a Docker container called runner-ci-general. The image is very large (> 50 Gb), since it has all necessary dependencies to build RuNNer in a variety of setups (oneAPI, NVIDIA compilers, etc.). The image has already been pulled on all of our private Gitlab runners. Take a look at the README.md of the image repo for details on how to make changes to the image.

test stage

The testing stage

  • runs our collection of pFunit-based unit tests
  • runs a first set of simple regression and integration tests before we run a more extensive set of tests
  • runs a linter to enforce some general code style guidelines
  • checks that all keywords have been documented in alphabetical order

The unit tests run with multiple executables compiled in the build stage.

test-regression and test-regression-mpi stages

These stages are absolutely crucial. They run a comprehensive set of regression tests (comparison to results from earlier code versions) to ensure that we can reproduce past results up to a certain accuracy. For predictions, this accuracy is 1e-7 (for all properties), for training it is 1e-3.

The tests are the runner-examples in the RuNNer suite.

These tests do not run by default (except for merges to the main branch), because they take a significant amount of time. However, no branch can be merged without passing these checks at least once.

This stage also runs regression tests for the LAMMPS interface and runs all runnerase notebooks in the runnerase-usage repo.

deploy stage

This stage only runs for merges to main. It builds conda packages with RuNNer for different architectures, and deploys them together with the online documentation to Gitlab pages.

post-deploy-test-regression stage

Finally, after the conda packages have been deployed, they are tested once again against the full suite of regression tests in a production environment (clean mamba Docker container without any dependencies). This way, we can be sure to ship a program that just works.

Debugging the pipeline

It happens more often than one would hope that the pipeline gets stuck or fails. Here are a few tips what to do in those cases:

  1. Read the error log: This one is a no-brainer, but still: go to gitlab.com and look at the output for the failing job. This often already gives you a decent understanding of what went wrong.
  2. Pipeline editor: if Gitlab lists error messages about dependencies not being fulfilled or syntax errors in .gitlab-ci.yml, go to the pipeline editor. This online tool gives you realtime information about problems with the syntax of the file. Helps a lot!
  3. Debug locally: most jobs can be run locally (linting, regression tests, etc.). More often than not, this allows you to reproduce the error in an environment that you control. Fix it, then try again.
  4. Spin up the Docker container: if you cannot reproduce the behavior outside of the CI job, log in to one of our Gitlab runners (if you are a Behler group member), or contact us. Use docker run -it runner-ci-general/latest to spin up a clean container, and reproduce the problem there. You can copy files to the running container using docker cp.
  5. Do not run all jobs every time: our CI pipeline consists of many stages, each with many individual jobs. If you are only debugging one of them, comment or cancel the remaining jobs. Otherwise, the runners get clogged and that helps no one.