Make and makefiles
n previous posts we have been compiling and linking projects using bash commands, we had to write all commands explicitly to build the projects. This is ok for small projects with few source files but oftentimes in large projects we need to compile hundreds of source files and link them to different libraries which makes the build complex. This is what make was invented for.
The GNU make utility was written in 1977 by Stuart Feldman at Bell Labs and its purpose was to automate the build process, replacing manual shell scripts for compiling and linking large projects. Other (more modern) build systems for C++ projects are SCons, CMake, Bazel, and Ninja. Even though make is old is still widely used in the industry, specially for small projects.
In this post we will learn how make works as always illustrating it with an example. The entire example can be found in my github repository blogging-code, in the subdirectory cpp-makefile.
TLDR
With the filestructure defned in the following section, the makefile can be
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# Compiler and flags
CXX = g++
CXXFLAGS = -std=c++17 -Iinclude
# Directories
SRC_DIR = src
INCLUDE_DIR = include
BUILD_DIR = build
OBJ_DIR = $(BUILD_DIR)/obj
BIN_DIR = $(BUILD_DIR)/bin
# Target executable name
TARGET = $(BIN_DIR)/main
# # Find all source files and corresponding object files
SRCS = $(wildcard $(SRC_DIR)/*.cpp)
OBJS = $(patsubst $(SRC_DIR)/%.cpp, $(OBJ_DIR)/%.o, $(SRCS))
# Default target
all:
@echo "Available options:"
@echo " build - Build the project"
@echo " clean - Remove all build files"
@echo " help - Show this message"
# Build target
build: $(TARGET)
# Rule to build the executable
$(TARGET): $(OBJS)
@mkdir -p $(BIN_DIR)
$(CXX) $(CXXFLAGS) -o $@ $^
# Rule to build object files
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.cpp
@mkdir -p $(OBJ_DIR)
$(CXX) $(CXXFLAGS) -c $< -o $@
# Clean up build files
clean:
rm -rf $(BUILD_DIR)
# Help target
help: all
# Phony targets
.PHONY: all clean build help
That can be executed with
1
2
3
4
make
make clean
make build
./build/bin/main
that displays the helper, builds the program and executes the main
File structure
As always for our example we define here the file structure and contents so that you can copy-paste the example and run it yourself. The file structure is
1
2
3
4
5
6
7
.
├── Makefile
├── include
│ └── matmul.h
└── src
├── main.cpp
└── matmul.cpp
with matmul.h:
1
2
3
4
5
6
7
#ifndef MATMUL_H
#define MATMUL_H
void matmul(const int* A, const int* B, int* C, int M, int N, int K);
void printmatrix(const int* A, int M, int N);
#endif
and matmul.cpp:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#include <iostream>
#include "matmul.h"
// Matrices are indexed row-major in this example. E.g. if A is [M x N]
// If i,j are the row and column indices, the element A[i, j] is
// A[i, j] = A[i * N + j] // if row-index
// A[i, j] = A[j * M + i] // if column-index
void matmul(const int* A, const int* B, int* C, int M, int N, int K){
// Matrix multiplication, C[M x K] = A[M x N] * B[N x K]
// Multiplication is $\sum_n A[m, n] * B[n, k]$
for(int m=0; m<M; m++){
for(int k=0; k<K; k++){
C[m * K + k] = 0;
for(int n=0; n<N; n++){
C[m * K + k] += A[m * N + n] * B[n * K + k];
}
}
}
}
void printmatrix(const int* A, int M, int N) {
for (int i = 0; i < M; ++i) {
for (int j = 0; j < N; ++j) {
std::cout << A[i * N + j] << " ";
}
std::cout << "\n";
}
}
and finally main.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#include <iostream>
#include "matmul.h"
int main(void){
// A[M x N]
int M = 2;
int N = 3;
int* A = new int[M * N];
for(int i=0; i < M * N; i++){
A[i] = i;
}
std::cout << std::endl << "A:" << std::endl;
printmatrix(A, M, N);
// B[N x K]
int K = 4;
int* B = new int[N * K];
for(int i=0; i < N * K; i++){
B[i] = i;
}
std::cout << std::endl << "B:" << std::endl;
printmatrix(B, N, K);
// C[M x K]
int* C = new int[M * K];
matmul(A, B, C, M, N, K);
std::cout << std::endl << "C = A x B: " << std::endl;
printmatrix(C, M, K);
return 0;
}
We have used this example before, this is just a matrix multiplication example. The idea is to compile the main as an executable using make.
Make basics
The process of compilation can be thought of as a graph: to generate a executable or library we need first to compile all the source files to objects and then link all the objects into the executable (or library). In more complex projects there could be even more compilation steps. Make has also the advantage of compiling only the files that have changed, this is crucial for large projects as it could take several minutes to compile the code again from scratch while there is no need if the source file hasn’t changed. Check the compilation time of OpenCV in a previous post.
To run make we create a file named Makefile with the compilation instructions and then run make:
1
2
touch Makefile
make
Since the Makefile is empty (there are no rules) make will complain with make: *** No targets. Stop.. Let’s add a rule in the file
1
2
hello:
echo "Hi there"
Now run make hello and it will print into your screen the “Hi there”. In essence the makefile contains rules, a rule has the following syntax
1
2
3
4
targets: prerequisites
command
command
command
The targets are normally files to be compiled (object files) and the prerequisites the source files, but before jumping to that, let’s understand the essence of the graph calculation. Modify the makefile to contain
1
2
3
4
5
6
7
8
9
10
11
calculate_1:
echo "calculate_1"
calculate_2:
echo "calculate_2"
calculate_3: calculate_1
echo "calculate_3"
calculate_4: calculate_1 calculate_2
echo "calculate_4"
Running make calculate_1 will just print “calculate_1”, similarly for make calculate_2. These two rules don’t depend on any other rule. However if we run make calculate_3 it will print first “calculate_1” and then “calculate_3” as the third calculation depends on the first (by the prerequisites in the rule). A similar case will happen in make calculate_4, this time it will print first “calculate_1” and then “calculate_2” before printing “calculate_4”. This describes the nature of makefile, you can nest this as much as you want to generate a direct acyclic graph of your bash commands.
But make is more than just instructions, it is intrinsically linked to files. Let me explain this with an example. Create a file “calculate_1” and try to run the calculate_1 rule from the previous makefile.
1
2
touch calculate_1
make calculate_1
you will be prompted with make: 'calculate_1' is up to date.. Indeed!, make interprets that since there is a file in this directory named calculate_1 it has already been “compiled” and there is nothing to do for this rule. This is very useful when you have compilation errors in certain files, the files that are compiled successfully won’t be compiled again if you re-run make.
Simple make: build and run the example
Let’s write a simple makefile to compile and link the program:
1
2
3
4
5
6
7
8
matmul.o:
g++ -std=c++17 -Iinclude -c src/matmul.cpp -o matmul.o
main.o:
g++ -std=c++17 -Iinclude -c src/main.cpp -o main.o
compile: matmul.o main.o
g++ matmul.o main.o -o main
Now run
1
2
make compile
./main
to compile and run the program. See that we have specified a graph here. To run compile we need to have the files matmul.o and main.o. See that the files will be generated in the current directory, we can modify that by writing the path in the rule build/obj/matmul.o for instance.
Automatic variables
There are some special characters defined in the make documentation called automatic variables. These are very useful but rarely explained through examples. Before jumping to a complete makefile we’ll explain some of them
1
2
3
4
5
6
output.txt: input1.txt input2.txt input1.txt
echo "Target: $@" > $@
echo "First prerequisite: $<" >> $@
echo "Updated prerequisites: $?" >> $@
echo "All prerequisites (unique): $^" >> $@
echo "All prerequisites (with duplicates): $+" >> $@
input1.txt, input2.txt, and input1.txt are listed as prerequisites. Notice that input1.txt is repeated to demonstrate $^ (unique) vs. $+ (duplicates included).
$@: Refers to the target,output.txt.$<: Refers to the first prerequisite,input1.txt.$?: Lists all prerequisites that are newer than the target. This is dynamic and depends on file timestamps.$^: Lists all unique prerequisites (input1.txt input2.txt).$+: Lists all prerequisites, including duplicates (input1.txt input2.txt input1.txt).
As an example create the prerequisites with
1
2
echo Hello from input1! > input1.txt
echo Hello from input2! > input2.txt
and run make output.txt, the result in output.txt will be:
1
2
3
4
5
Target: output.txt
First prerequisite: input1.txt
Updated prerequisites: input1.txt input2.txt
All prerequisites (unique): input1.txt input2.txt
All prerequisites (with duplicates): input1.txt input2.txt input1.txt
We will use some of these automatic variables to write a more robust makefile.
A more complete makefile
Let’s write a proper makefile this time, now that we understand the basics.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# Compiler and flags
CXX = g++
CXXFLAGS = -std=c++17 -Iinclude
# Directories
SRC_DIR = src
INCLUDE_DIR = include
BUILD_DIR = build
OBJ_DIR = $(BUILD_DIR)/obj
BIN_DIR = $(BUILD_DIR)/bin
# Target executable name
TARGET = $(BIN_DIR)/main
# # Find all source files and corresponding object files
SRCS = $(wildcard $(SRC_DIR)/*.cpp)
OBJS = $(patsubst $(SRC_DIR)/%.cpp, $(OBJ_DIR)/%.o, $(SRCS))
# Default target
all: $(TARGET)
# Rule to build the executable
$(TARGET): $(OBJS)
@mkdir -p $(BIN_DIR)
$(CXX) $(CXXFLAGS) -o $@ $^
# Rule to build object files
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.cpp
@mkdir -p $(OBJ_DIR)
$(CXX) $(CXXFLAGS) -c $< -o $@
# Clean up build files
clean:
rm -rf $(BUILD_DIR)
# Phony targets
.PHONY: all clean
I know, this is a lot… let’s explain line by line.
In the first two lines we define the compiler g++ and the flags used to compile the files -std=c++17 -Iinclude, the C++ standard 17 and the include directories.
Next we define the paths, src, include and the build directories as usual build/obj and build/bin (this time we won’t compile any library).
The target is the variable target, the file build/bin/main. This is the main rule we are going to execute.
The sources and object files could be specified explicitly with
1
2
SRCS = src/matmul.cpp src/main.cpp
OBJS = build/obj/matmul.o build/obj/main.o
but, it’s better to use the wildcard and patsubst commands. The first command finds all the files in the SRC_DIR that end with .cpp, the second is used to substitute the names of the files ending with .cpp to end with .o, so we construct the object paths and names. This is very convenient if our sources are in the same directory, if we have a nested directory we can do this operation several times, one per path.
Next the all rule. This one is the rule that is going to be executed when calling make without specifying any rule. The default target.
The $(TARGET) rule is (by substituting the variable) build/bin/main but this rule has the requirements of the object files $(OBJ) (that’s build/obj/matmul.o build/obj/main.o). To build the objects first, the target needs to build $(OBJ_DIR)/%.o, that expands all the objects paths. This rule creates the objects directory first and then compiles the objects (recall the -c flag for compilation to object). The command $< , as explained previously, represents the all the input (source files, prerequisites in this case) and $@ the name of the target output.
Going back to the rule $(TARGET) after the objects have been compiled, we create first the binary directory and then link the objects. For that the -o flag with $@ representing the input (build/bin/main) and $^ all the prerequisites (all the object files).
Finally we write a clean rule that removes the build dir. See that we define .PHONY targets to be all and clean. A phony target is not associated with any actual file; instead, it represents an action or a command. By declaring a target as phony, you ensure that make will always execute the associated recipe, regardless of whether a file with the same name as the target exists in the filesystem. Why use phony then? if a file named clean exists in the directory, running make clean without .PHONY would check the timestamp of the clean file and conclude that the target is “up to date.” This prevents the clean recipe from running. Declaring clean as a phony target ensures the recipe is executed regardless of such a file’s presence.
Another nice addition is a helper menu. Make the default all to print the targets on screen. To do that let me modify part of the makefile. For simplicity I include the entire new makefile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# Compiler and flags
CXX = g++
CXXFLAGS = -std=c++17 -Iinclude
# Directories
SRC_DIR = src
INCLUDE_DIR = include
BUILD_DIR = build
OBJ_DIR = $(BUILD_DIR)/obj
BIN_DIR = $(BUILD_DIR)/bin
# Target executable name
TARGET = $(BIN_DIR)/main
# # Find all source files and corresponding object files
SRCS = $(wildcard $(SRC_DIR)/*.cpp)
OBJS = $(patsubst $(SRC_DIR)/%.cpp, $(OBJ_DIR)/%.o, $(SRCS))
# Default target
all:
@echo "Available options:"
@echo " build - Build the project"
@echo " clean - Remove all build files"
@echo " help - Show this message"
# Build target
build: $(TARGET)
# Rule to build the executable
$(TARGET): $(OBJS)
@mkdir -p $(BIN_DIR)
$(CXX) $(CXXFLAGS) -o $@ $^
# Rule to build object files
$(OBJ_DIR)/%.o: $(SRC_DIR)/%.cpp
@mkdir -p $(OBJ_DIR)
$(CXX) $(CXXFLAGS) -c $< -o $@
# Clean up build files
clean:
rm -rf $(BUILD_DIR)
# Help target
help: all
# Phony targets
.PHONY: all clean build help
Again run
1
2
3
4
make
make clean
make build
./build/bin/main
to display the help, clean the directory and build again (compile) the directory.
Wrap up
We showed how to compile an executable using make and a makefile. This tutorial could be expanded with building libraries (static, dynamic) and linking them. But this is just an extension of the logic presented here. Developers still use makefiles, they are easy to comprehend, widely adopted and efficient but cmake (and other tools like bazel) is used for bigger projects, we’ll show how to use this tool in a follow up post. But till then, congratulations! Now you can build C++ programs with make!.