CMake, Make, and the Pitfalls of Parallel Builds with Git Checkouts

When working on large CMake-based projects like Allen, the LHCb real-time trigger framework, it’s common to leverage parallelism via make -jN to reduce build times. However, this optimization can backfire when the build system involves cloning or checking out repositories during the build. In this post, we’ll explore a class of race conditions caused by this pattern, how we encountered it in Allen, and how to reliably work around it.

The Motivation: Allen

During a fresh standalone build of Allen, the build process performs multiple actions:

• Fetching parameter files (checkout_param_files)
• Cloning external projects like Gaudi and LHCb (checkout_gaudi, checkout_lhcb)
• Building the C++ and CUDA codebase

These checkouts are implemented as custom targets using CMake’s ExternalProject_Add. If you invoke:

make -j16

right after running cmake, the build fails more often than not with errors like:

make[2]: *** No rule to make target 'external/LHCb/Event/DAQEvent/src/RawBank.cpp', needed by 'stream/CMakeFiles/checkout_lhcb'.  Stop.

or even worse:

cd: Allen: No such file or directory

These are classic symptoms of a parallel race condition: dependent source files are accessed before the git clone is complete.

Why This Happens

CMake treats ExternalProject_Add targets as standalone. If you don’t explicitly serialize the dependencies in the build, make starts compiling code that expects external projects to already exist.

Here’s what’s going wrong:

1. make launches all targets in parallel.
2. Compilation targets start before checkout_lhcb or checkout_gaudi has finished cloning.
3. This leads to “missing source file” or “no rule to make target” errors.

Minimal Reproducible Example

Let’s create a simple case that mimics this failure.

cmake_minimum_required(VERSION 3.16)
project(ParallelCloneIssue)

include(ExternalProject)

ExternalProject_Add(MyDep
    GIT_REPOSITORY https://github.com/MohamedElashri/some-heavy-project.git
    PREFIX ${CMAKE_BINARY_DIR}/_deps
    TIMEOUT 30
    UPDATE_COMMAND ""
    CONFIGURE_COMMAND ""
    BUILD_COMMAND ""
    INSTALL_COMMAND ""
)

add_custom_target(mycode ALL
    COMMAND ${CMAKE_COMMAND} -E echo "Compiling my code..."
    DEPENDS MyDep
)

Then run:

cmake -S . -B build
cmake --build build -j8

It might work, or it might fail randomly if mycode doesn’t wait properly for the clone.

Workaround: Bootstrap Sequentially

The simplest workaround is to bootstrap external dependencies sequentially before any parallel make.

In Allen, this means:

make checkout_param_files checkout_gaudi checkout_lhcb -j1
make -j16

Or if you use a script:

echo "Bootstrapping parameter and external projects (single-threaded)..."
make checkout_param_files checkout_gaudi checkout_lhcb -j1

echo "Building the full project in parallel..."
make -j16

This guarantees the repositories are cloned and ready before compilation begins.

Additional Notes

• Always ensure DEPENDS is used correctly in add_custom_target() or add_dependencies() to force serialization.
• ExternalProject_Add is powerful, but dangerous in parallel builds.
• Avoid assuming that just because something works in make -j1, it’ll work in make -jN.

Conclusion

Parallel builds with CMake and Make can be extremely powerful, but they also amplify build system misconfigurations. When using ExternalProject_Add to fetch code from Git during the build, you must be careful to avoid race conditions. The fix is often simple: serialize the fetch phase before going parallel.

Allen, a real-world high-performance project, helped surface this issue in a reproducible way for me. If your build system includes git clone, remember: clone before you compile.