distcc compared to other systems

Summary

The main advantages of distcc compared to other tools are:

  • distcc is free software under the GNU GPL.
  • distcc is simple to install and configure.
  • distcc can be installed by a non-root user, without requiring a kernel patch or any change to installed software or filesystems.
  • distcc has very little impact on machines helping with the build.
  • distcc doesn't require a shared filesystem.
  • distcc can optionally use strongly encrypted and authenticated ssh channels for communication. In this mode it is safe to use on untrusted networks.
  • distcc can use machines with different processor architectures (i386, ppc, ...) or operating systems (Cygwin, Linux, BSD), if appropriate cross-compilers are installed. distcc runs on many operating systems and architectures.
  • distcc works reasonably even over wireless networks, when the LZO compression option is turned on.

The disadvantages of distcc are:

  • distcc requires that the same compiler version be installed on all participating machines. (Every other distributed build system requires this too.)
  • Only C/C++/ObjC/ObjC++ compilation can be distributed, not linking or other tasks. (Some clustering systems can distribute any task.)
  • distcc comes with no warranty (as does most software), though it has been successfully used by many people for many years.

OpenMOSIX

OpenMOSIX is a kernel-level single-image clustering system for Linux. It can be used to spread software builds or any other parallel task across multiple machines.

OpenMOSIX requires a kernel patch. All the machines must be of the same architecture and running similar kernels.

MOSIX and OpenMOSIX give you essentially a "single system image" cluster, which looks like a fast computer. distcc just looks like a fast compiler.

Some people have reported that distcc is much faster than OpenMOSIX for software builds in some situations. Short-lived, CPU-intensive compiler processes may be a very difficult load for a design based on process migration.

Other differences:

  • You don't need root to install distcc
  • Installing distcc doesn't require a kernel patch or reboot; presumably it can't destabilize anything else on the machine.
  • distcc copes reasonably gracefully if a machine goes away abruptly or the network goes down; it will just keep compiling locally.
  • distcc works (at least in theory) on any POSIX-like operating system, not just Linux, and on any processor. It has been reported to work on Linux, FreeBSD, HP-UX, Solaris, AIX and Cygwin, and you can spread load across machines using different systems. OpenMOSIX works only on Linux, and (perhaps?) only on x86 and IA-64.
  • distcc volunteers can have different processor architectures or even operating systems to the machines they're serving. (You do need to install a cross compiler though.)
  • distcc is only about 13000 lines of C, so you may find it easier to hack or patch.

Barnowl writes:

Currently openMosix (oM) only runs on Linux and currently only on x86 and IA64 arch. distcc works like a champ when used combined with oM. While oM does help some with short run jobs, its forte is big computation jobs so it works better for things like video edits or DNA decoding. If you just want to speed up C compiles with out a lot of work or custom kernels distcc is the way to go.

Cook

Cook is a Make replacement written by Peter Miller and distributed under the GNU GPL. Cook can do parallel builds and has built-in support for distributing builds across machines on a LAN.

Using Cook requires you to convert your Makefiles to a cookbook, though there is a make2cook program to help with this.

Cook runs commands over rsh by default, but it seems like you could use ssh instead. Cook uses rup to calculate load on different machines.

It seems like Cook requires a shared filesystem and the same tools on all machines. There is a mechanism to group machines into classes and run particular commands on particular classes of machine.

pvmgmake

pvmgmake is a fork of GNU Make which can spread tasks across multiple machines using the PVM protocol.

pvmgmake requires that all the machines see the same filesystem, and have the same tools installed.

XCode

Apple XCode Distributed Build is based on distcc.

Apple add patches for Rendezvous-based discovery of servers and transport of precompiled headers. These features may come back into the mainstream distribution in the future. Apple also add GUI to configure distcc, and support for some Apple-specific gcc features.

Apple's tools only work on Mac OS X.

The Apple version at the moment does not seem to support LZO or SSH.

Copyright © 2002–2004 Martin Pool. distcc is a trademark of Martin Pool.

Send comments to distcc(at)lists.samba.org, or talk to us on #distcc on irc.oftc.net.