--- layout: post title: DistCC and CCache on Linux --- ![xkcd knows this](https://imgs.xkcd.com/comics/compiling.png) After a project reaches a certain size, compiling the code becomes a task in and of itself and is a source of wasted time the world over. Recently I started looking at tools to reduce the amount of time I waste waiting on code to compile and I found a few things that can help with that. ### ccache The first tool I found was something called [ccache](https://ccache.dev), and it works almost like a web browser cache, i.e. it stores the output of every compile operation I run in my home directory, and when I run the same job again it recalls the same result. All one needs to do to get it working is to either prefix every call to the compiler with `ccache` or, and this is what I recommend, prefix your PATH environment variable with `/usr/lib/ccache/bin`. I just have a line in my `.zshenv` file that does just that: export PATH="/usr/lib/ccache/bin:$PATH" Now be warned, the first run with a large project will take *longer* than a run without ccache enabled. The reason for this is that it has to build its internal cache which takes some amount of time between calls to the compiler. You can check the ccache statistics with the command `ccache -s`. Here's an example: $ ccache -s cache directory /home/dholman/.ccache primary config /home/dholman/.ccache/ccache.conf secondary config (readonly) /etc/ccache.conf stats updated Fri Nov 6 13:31:48 2020 cache hit (direct) 16980 cache hit (preprocessed) 2373 cache miss 76011 cache hit rate 20.29 % called for link 2154 called for preprocessing 2603 multiple source files 2 compiler produced stdout 4 compiler produced empty output 652 compile failed 657 preprocessor error 1541 bad compiler arguments 260 unsupported source language 10 autoconf compile/link 3087 unsupported code directive 123 could not write to output file 110 no input file 2858 cleanups performed 65 files in cache 6387 cache size 586.9 MB max cache size 20.0 GB ### distcc The next tool was a bit more fiddly to get working and may or may not work for everyone, so be warned. Also, in order to get any benefit out of this, you need a separate computer attached to the same network as your development workstation. Distcc, according to the [website](https://distcc.github.io), is a distributed compiler. That's not *really* true, as it functions as an extension to the existing compiler already running on my machine. In my case I have a Dell R710 dual Xeon system running in the basement, and I have distcc set up as a daemon listening on TCP port 3632 on that system. Here's an excerpt from `/etc/default/distcc` allowing all hosts on my network to use it: # # Which networks/hosts should be allowed to connect to the daemon? # You can list multiple hosts/networks separated by spaces. # Networks have to be in CIDR notation, f.e. 192.168.1.0/24 # Hosts are represented by a single IP Adress # # ALLOWEDNETS="127.0.0.1" ALLOWEDNETS="127.0.0.1 192.168.1.0/24" The next step was to install distcc on my workstation and configure it to use the server downstairs as a slave: # --- /etc/distcc/hosts ----------------------- # See the "Hosts Specification" section of # "man distcc" for the format of this file. # # By default, just test that it works in loopback mode. 192.168.1.26/24,cpp,lzo With all of this configured I could then build a project with `make -j CC=distcc`. Here's a few results from some of my personal projects and other things. Forge game engine: $ time make -j20 CC=distcc [ 23%] Building C object CMakeFiles/forge.dir/src/ui/button.c.o [ 23%] Building C object CMakeFiles/forge.dir/src/data/stack.c.o [ 23%] Building C object CMakeFiles/forge.dir/src/data/list.c.o [ 30%] Building C object CMakeFiles/forge.dir/src/ui/spinner.c.o [ 38%] Building C object CMakeFiles/forge.dir/src/ui/text.c.o [ 46%] Building C object CMakeFiles/forge.dir/src/engine.c.o [ 53%] Building C object CMakeFiles/forge.dir/src/ui/rect.c.o [ 61%] Building C object CMakeFiles/forge.dir/src/graphics.c.o [ 69%] Building C object CMakeFiles/forge.dir/src/input.c.o [ 76%] Building C object CMakeFiles/forge.dir/src/entity.c.o [ 92%] Building C object CMakeFiles/forge.dir/src/sprite.c.o [ 92%] Building C object CMakeFiles/forge.dir/src/tmx.c.o [100%] Linking C shared library libforge.so [100%] Built target forge make -j20 CC=distcc 0.03s user 0.02s system 97% cpu 0.046 total Linux kernel with default config: $ time make -j20 CC=distcc ... Kernel: arch/x86/boot/bzImage is ready (#1) make -j20 CC=distcc 463.45s user 116.23s system 120% cpu 8:01.29 total Astute readers might notice that these results seem a little far-fetched for *just* distcc alone, and they would be right. Because I set ccache to be called *before* the compiler, distcc gets hooked into the call of ccache. I do not recommend doing it the other way around and neither do the developers of ccache and distcc. I also recommend *not* setting up distcc to be called on *every* call to the compiler, as distributing source code over TCP/IP like this can be a very time consuming process on sub-gigabit speed networks. One other thing to note is that the compiler versions *must* match, or all calls to distcc will fail. One can look at what distcc is currently doing behind the scenes with `distccmon-text `, or if a GUI interface is preferred, you can use `distccmon-gnome`. ![distcc monitor](/assets/distcc.png)