From 3dc3136872ef219d18645c3114c315b50c63035d Mon Sep 17 00:00:00 2001 From: Danny Holman Date: Tue, 17 Nov 2020 00:03:48 -0600 Subject: Create new post 2020/11/17 Add new post for 2020/11/17. This post describes my experiences with distcc and ccache and getting them set up on my systems. --- _posts/2020-11-16-Distcc-ccache-setup.md | 139 +++++++++++++++++++++++++++++++ assets/distcc.png | Bin 0 -> 78579 bytes 2 files changed, 139 insertions(+) create mode 100644 _posts/2020-11-16-Distcc-ccache-setup.md create mode 100644 assets/distcc.png diff --git a/_posts/2020-11-16-Distcc-ccache-setup.md b/_posts/2020-11-16-Distcc-ccache-setup.md new file mode 100644 index 0000000..8dc74c6 --- /dev/null +++ b/_posts/2020-11-16-Distcc-ccache-setup.md @@ -0,0 +1,139 @@ +--- +layout: post +title: DistCC and CCache on Linux +--- + +![xkcd knows this](https://imgs.xkcd.com/comics/compiling.png) + +After a project reaches a certain size, compiling the code becomes a task in and +of itself and is a source of wasted time the world over. Recently I started +looking at tools to reduce the amount of time I waste waiting on code to compile +and I found a few things that can help with that. + +### ccache + +The first tool I found was something called [ccache](https://ccache.dev), and it +works almost like a web browser cache, i.e. it stores the output of every +compile operation I run in my home directory, and when I run the same job again +it recalls the same result. + +All one needs to do to get it working is to either prefix every call to the +compiler with `ccache` or, and this is what I recommend, prefix your PATH +environment variable with `/usr/lib/ccache/bin`. I just have a line in my +`.zshenv` file that does just that: + + export PATH="/usr/lib/ccache/bin:$PATH" + +Now be warned, the first run with a large project will take *longer* than a +run without ccache enabled. The reason for this is that it has to build its +internal cache which takes some amount of time between calls to the compiler. + +You can check the ccache statistics with the command `ccache -s`. Here's an +example: + + $ ccache -s + cache directory /home/dholman/.ccache + primary config /home/dholman/.ccache/ccache.conf + secondary config (readonly) /etc/ccache.conf + stats updated Fri Nov 6 13:31:48 2020 + cache hit (direct) 16980 + cache hit (preprocessed) 2373 + cache miss 76011 + cache hit rate 20.29 % + called for link 2154 + called for preprocessing 2603 + multiple source files 2 + compiler produced stdout 4 + compiler produced empty output 652 + compile failed 657 + preprocessor error 1541 + bad compiler arguments 260 + unsupported source language 10 + autoconf compile/link 3087 + unsupported code directive 123 + could not write to output file 110 + no input file 2858 + cleanups performed 65 + files in cache 6387 + cache size 586.9 MB + max cache size 20.0 GB + +### distcc + +The next tool was a bit more fiddly to get working and may or may not work for +everyone, so be warned. Also, in order to get any benefit out of this, you need +a separate computer attached to the same network as your development +workstation. + +Distcc, according to the [website](https://distcc.github.io), is a distributed +compiler. That's not *really* true, as it functions as an extension to the +existing compiler already running on my machine. In my case I have a Dell R710 +dual Xeon system running in the basement, and I have distcc set up as a daemon +listening on TCP port 3632 on that system. Here's an excerpt from +`/etc/default/distcc` allowing all hosts on my network to use it: + + # + # Which networks/hosts should be allowed to connect to the daemon? + # You can list multiple hosts/networks separated by spaces. + # Networks have to be in CIDR notation, f.e. 192.168.1.0/24 + # Hosts are represented by a single IP Adress + # + # ALLOWEDNETS="127.0.0.1" + + ALLOWEDNETS="127.0.0.1 192.168.1.0/24" + +The next step was to install distcc on my workstation and configure it to use +the server downstairs as a slave: + + # --- /etc/distcc/hosts ----------------------- + # See the "Hosts Specification" section of + # "man distcc" for the format of this file. + # + # By default, just test that it works in loopback mode. + 192.168.1.26/24,cpp,lzo + +With all of this configured I could then build a project with +`make -j CC=distcc`. Here's a few results from some of +my personal projects and other things. + +Forge game engine: + + $ time make -j20 CC=distcc + [ 23%] Building C object CMakeFiles/forge.dir/src/ui/button.c.o + [ 23%] Building C object CMakeFiles/forge.dir/src/data/stack.c.o + [ 23%] Building C object CMakeFiles/forge.dir/src/data/list.c.o + [ 30%] Building C object CMakeFiles/forge.dir/src/ui/spinner.c.o + [ 38%] Building C object CMakeFiles/forge.dir/src/ui/text.c.o + [ 46%] Building C object CMakeFiles/forge.dir/src/engine.c.o + [ 53%] Building C object CMakeFiles/forge.dir/src/ui/rect.c.o + [ 61%] Building C object CMakeFiles/forge.dir/src/graphics.c.o + [ 69%] Building C object CMakeFiles/forge.dir/src/input.c.o + [ 76%] Building C object CMakeFiles/forge.dir/src/entity.c.o + [ 92%] Building C object CMakeFiles/forge.dir/src/sprite.c.o + [ 92%] Building C object CMakeFiles/forge.dir/src/tmx.c.o + [100%] Linking C shared library libforge.so + [100%] Built target forge + make -j20 CC=distcc 0.03s user 0.02s system 97% cpu 0.046 total + +Linux kernel with default config: + + $ time make -j20 CC=distcc + ... + Kernel: arch/x86/boot/bzImage is ready (#1) + make -j20 CC=distcc 463.45s user 116.23s system 120% cpu 8:01.29 total + +Astute readers might notice that these results seem a little far-fetched for +*just* distcc alone, and they would be right. Because I set ccache to be called +*before* the compiler, distcc gets hooked into the call of ccache. I do not +recommend doing it the other way around and neither do the developers of ccache +and distcc. I also recommend *not* setting up distcc to be called on *every* +call to the compiler, as distributing source code over TCP/IP like this can +be a very time consuming process on sub-gigabit speed networks. One other thing +to note is that the compiler versions *must* match, or all calls to distcc will +fail. + +One can look at what distcc is currently doing behind the scenes with +`distccmon-text `, or if a GUI interface is preferred, you can +use `distccmon-gnome`. + +![distcc monitor](/assets/distcc.png) diff --git a/assets/distcc.png b/assets/distcc.png new file mode 100644 index 0000000..eb48a13 Binary files /dev/null and b/assets/distcc.png differ -- cgit v1.2.3