GNU Make Mastery Part 11: Parallel Builds & Performance

Race Conditions

Detecting Races

A race condition occurs when two parallel jobs read from or write to the same file — or when a job starts before a prerequisite it needs has been built. Races are non-deterministic: the build may succeed on one run and fail on the next.

# Run multiple times to expose flaky races:
for i in $(seq 1 10); do make clean && make -j$(nproc) || echo "FAILED on run $i"; done

# Use --shuffle (GNU Make 4.4+) to randomize build order and expose hidden deps:
make -j8 --shuffle

Fixing with Order-Only Prerequisites

# Problem: two rules write to build/ simultaneously before mkdir runs
$(BUILDDIR)/%.o: src/%.c
	$(CC) $(CFLAGS) -c $< -o $@     # fails if build/ doesn't exist yet

# Fix with order-only prerequisite (| separator):
# build/ is created before the rule, but its timestamp doesn't trigger rebuilds
$(BUILDDIR)/%.o: src/%.c | $(BUILDDIR)
	$(CC) $(CFLAGS) -MMD -MP -c $< -o $@

$(BUILDDIR):
	mkdir -p $@

# Race: generated header used before it's produced
$(BUILDDIR)/parser.o: src/parser.c | $(BUILDDIR)/grammar.h
	$(CC) $(CFLAGS) -c $< -o $@

$(BUILDDIR)/grammar.h: grammar.y
	bison --defines=$@ $<

.NOTPARALLEL and .WAIT

# Force serial execution of a specific target's prerequisites:
.NOTPARALLEL: generate-code

# GNU Make 4.4+: .WAIT inserts a sync point within a prerequisite list:
all: compile-stage .WAIT link-stage

# Force entire Makefile to be serial (last resort — kills all speedup):
.NOTPARALLEL:

Lazy vs Immediate Expansion

Recursive (=) variables are re-evaluated every time they're referenced. In large Makefiles with hundreds of $(shell ...) calls, this can add seconds of overhead.

# BAD — $(shell find ...) called EVERY time $(SRCS) is expanded:
SRCS = $(shell find src -name '*.c')

# GOOD — evaluated once at parse time with :=
SRCS := $(shell find src -name '*.c')

# VERY BAD in a rule — wildcard called in a recipe (correct place is makefile body):
%.o: %.c
	$(CC) $(shell pkg-config --cflags gtk+-3.0) -c $< -o $@    # re-runs pkg-config for every .o!

# GOOD — evaluate pkg-config once at parse time:
GTK_CFLAGS := $(shell pkg-config --cflags gtk+-3.0)
GTK_LIBS   := $(shell pkg-config --libs   gtk+-3.0)

%.o: %.c
	$(CC) $(CFLAGS) $(GTK_CFLAGS) -c $< -o $@

Profiling & Time Tracking

# Overall build time:
time make -j$(nproc)

# Per-recipe timing with make --trace (GNU Make 4.0+):
make --trace 2>&1 | head -60

# Use remake (enhanced make with profiling):
remake --profile       # writes profile to profile.json

# Ninja-style timing output (requires a wrapper):
# Set SHELL to a timing wrapper:
SHELL = /bin/bash
# Then add "time" prefix to expensive rules for spot checks

Try It — Parallel vs Sequential

Create a small project and compare -j1 (sequential) with -j$(nproc) (parallel) build times:

mkdir -p src

# Create 4 independent source files
for f in main utils parser network; do
cat > src/$f.c << EOF
#include <stdio.h>
void ${f}_init(void) { printf("$f ready\n"); }
EOF
done

# Add main() to main.c
cat > src/main.c << 'EOF'
#include <stdio.h>
extern void utils_init(void);
extern void parser_init(void);
extern void network_init(void);
int main(void) {
    utils_init(); parser_init(); network_init();
    printf("All modules loaded\n");
    return 0;
}
EOF

# Write a parallel-safe Makefile
cat > Makefile << 'EOF'
CC       := gcc
CFLAGS   := -Wall -Wextra -O2
SRCS     := $(wildcard src/*.c)
OBJS     := $(SRCS:.c=.o)
TARGET   := myapp

.PHONY: all clean info
all: $(TARGET)

$(TARGET): $(OBJS)
	$(CC) -o $@ $^

src/%.o: src/%.c
	$(CC) $(CFLAGS) -c $< -o $@

clean:
	rm -f $(OBJS) $(TARGET)

info:
	@echo "CC     = $(CC)"
	@echo "CFLAGS = $(CFLAGS)"
	@echo "SRCS   = $(SRCS)"
	@echo "OBJS   = $(OBJS)"
EOF

# Compare build times
make info
time make -j1          # sequential
make clean
time make -j$(nproc)   # parallel
make clean