An example of multi-module Gitlab-CI pipeline

4 min readNov 10, 2020

This is a quick example of how to handle CI pipelines for a multi-module project. It is illustrated with Scala/Maven simple project, but, of course, the idea could be implemented in other languages/package managers.

🎬 Before we start

The purpose of this article is to show a manner of handling multi-module CI pipelines. The pipeline has 2 simple stages to make it easier to understand(Definitely, you can extend your pipeline as you need).

🗂️ Project structure

As you can see above, the project is composed of 3 modules:

common
healer
propagator

We tried to create a dependency between modules such as healer and propagator depend on common in order to make the example a bit trickier.

propagator module pom.xml — Propagator module pom.xml

📜 .gitlab-ci.yml

As the following diagram illustrates, there is a child-parent structure, a YAML file for every module with its specific logic(child) and a global one in the root folder(parent).

So let’s take a look at the parent .gitlab-ci.yml file where all the major logic is present. We’ll divide it into 3 parts for a clear explanation :

1st part

A very simple part, we define 2 stages(build and test), some variables to manage the maven dependencies caching:

MAVEN_OPTS: set the maven local repository to $CI_PROJECT_DIR/.m2/repository.
MAVEN_CLI_OPTS: some useful CLI options.
CACHE_KEY: set to the $CI_COMMIT_REF_SLUG, this will help us having a cache isolated by commit ID.

The next block of code defines a cache for the maven local repository and .sbt folder used for optimization(we’ll cover it later).

2nd part

This part covers our main interest, it starts with module YAML files inclusion :

If we dive into the child YAML file(common/.gitlab-ci.yml for instance):

we’ll see a definition of MODULE var and a magical 3 lines. The interesting keywords are only and changes(introduced in GitLab 11.4). Those help us say :

healer-module is only concerned with the healer folder, if there’s any change, trigger the pipeline.

After the inclusion part, we’ll find two templates:

.build-module : This will compile the module($MODULE) and all its dependent modules(thanks to the — also-make option). Then it creates an artifact archive(containing the build) available for the following jobs.
.test-module : Will test the previously compiled module(and other dependent module(s)).

3rd part

Now that we have the templates, we’ll define our jobs, it as simple as :

⚙️ Bonus : Compilation

Here we’ll discuss an optimization that was made in addition to(Caching and artifacts).
The optimization concerns the compilation(or should we say re-compilation).

So, in order to prevent re-compilation at both the build and test jobs. We used a scala-maven-plugin options to do an incremental compilation. It took us to set :

<plugin>
    <groupId>net.alchim31.maven</groupId>
    <artifactId>scala-maven-plugin</artifactId>
    <version>4.4.0</version>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
                <goal>testCompile</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <recompileMode>incremental</recompileMode>
        <secondaryCacheDir>${user.dir}/.sbt</secondaryCacheDir>
    </configuration>
</plugin>

The secondaryCacheDir points to the directory where to store the sbt compiler bridge (used by the plugin for managing the incremental compilation), and, this is why we cached the .sbt folder(in the first part).

👉 See the whole pom.xml file

Conclusion

We saw a brief example of maven multi-module CI pipeline. We need to keep in mind that this was possible due to :

only:changes feature(this is a possible solution, one could use workflow:rules)
— also-make Maven option: This lets us make the dependent modules, it could be designed with Gitlab-ci job dependencies, but our choice was to have a simple building process even locally.

👉 Get the Github project