On multi-developer software projects,
you can sometimes speed up every developer's builds a lot by
allowing them to share the derived files that they build.
&SCons; makes this easy, as well as reliable.
Specifying the Shared Cache Directory
To enable sharing of derived files,
use the &CacheDir; function
in any &SConscript; file:
CacheDir('/usr/local/build_cache')
Note that the directory you specify must already exist
and be readable and writable by all developers
who will be sharing derived files.
It should also be in some central location
that all builds will be able to access.
In environments where developers are using separate systems
(like individual workstations) for builds,
this directory would typically be
on a shared or NFS-mounted file system.
Here's what happens:
When a build has a &CacheDir; specified,
every time a file is built,
it is stored in the shared cache directory
along with its MD5 build signature.
Actually, the MD5 signature is used as the name of the file
in the shared cache directory in which the contents are stored.
On subsequent builds,
before an action is invoked to build a file,
&SCons; will check the shared cache directory
to see if a file with the exact same build
signature already exists.
If so, the derived file will not be built locally,
but will be copied into the local build directory
from the shared cache directory,
like so:
% scons -Q
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q
Retrieved `hello.o' from cache
Retrieved `hello' from cache
Keeping Build Output Consistent
One potential drawback to using a shared cache
is that the output printed by &SCons;
can be inconsistent from invocation to invocation,
because any given file may be rebuilt one time
and retrieved from the shared cache the next time.
This can make analyzing build output more difficult,
especially for automated scripts that
expect consistent output each time.
If, however, you use the --cache-show option,
&SCons; will print the command line that it
would have executed
to build the file,
even when it is retrieving the file from the shared cache.
This makes the build output consistent
every time the build is run:
% scons -Q
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q --cache-show
cc -o hello.o -c hello.c
cc -o hello hello.o
The trade-off, of course, is that you no longer
know whether or not &SCons;
has retrieved a derived file from cache
or has rebuilt it locally.
Not Using the Shared Cache for Specific Files
You may want to disable caching for certain
specific files in your configuration.
For example, if you only want to put
executable files in a central cache,
but not the intermediate object files,
you can use the &NoCache;
function to specify that the
object files should not be cached:
env = Environment()
obj = env.Object('hello.c')
env.Program('hello.c')
CacheDir('cache')
NoCache('hello.o')
Then when you run &scons; after cleaning
the built targets,
it will recompile the object file locally
(since it doesn't exist in the shared cache directory),
but still realize that the shared cache directory
contains an up-to-date executable program
that can be retrieved instead of re-linking:
% scons -Q
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q
cc -o hello.o -c hello.c
Retrieved `hello' from cache
Disabling the Shared Cache
Retrieving an already-built file
from the shared cache
is usually a significant time-savings
over rebuilding the file,
but how much of a savings
(or even whether it saves time at all)
can depend a great deal on your
system or network configuration.
For example, retrieving cached files
from a busy server over a busy network
might end up being slower than
rebuilding the files locally.
In these cases, you can specify
the --cache-disable
command-line option to tell &SCons;
to not retrieve already-built files from the
shared cache directory:
% scons -Q
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q
Retrieved `hello.o' from cache
Retrieved `hello' from cache
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q --cache-disable
cc -o hello.o -c hello.c
cc -o hello hello.o
Populating a Shared Cache With Already-Built Files
Sometimes, you may have one or more derived files
already built in your local build tree
that you wish to make available to other people doing builds.
For example, you may find it more effective to perform
integration builds with the cache disabled
(per the previous section)
and only populate the shared cache directory
with the built files after the integration build
has completed successfully.
This way, the cache will only get filled up
with derived files that are part of a complete, successful build
not with files that might be later overwritten
while you debug integration problems.
In this case, you can use the
the --cache-force option
to tell &SCons; to put all derived files in the cache,
even if the files already exist in your local tree
from having been built by a previous invocation:
% scons -Q --cache-disable
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q --cache-disable
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q --cache-force
scons: `.' is up to date.
% scons -Q
scons: `.' is up to date.
Notice how the above sample run
demonstrates that the --cache-disable
option avoids putting the built
hello.o
and
hello files in the cache,
but after using the --cache-force option,
the files have been put in the cache
for the next invocation to retrieve.
Minimizing Cache Contention: the --random Option
If you allow multiple builds to update the
shared cache directory simultaneously,
two builds that occur at the same time
can sometimes start "racing"
with one another to build the same files
in the same order.
If, for example,
you are linking multiple files into an executable program:
Program('prog',
['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])
&SCons; will normally build the input object files
on which the program depends in their normal, sorted order:
% scons -Q
cc -o f1.o -c f1.c
cc -o f2.o -c f2.c
cc -o f3.o -c f3.c
cc -o f4.o -c f4.c
cc -o f5.o -c f5.c
cc -o prog f1.o f2.o f3.o f4.o f5.o
But if two such builds take place simultaneously,
they may each look in the cache at nearly the same
time and both decide that f1.o
must be rebuilt and pushed into the shared cache directory,
then both decide that f2.o
must be rebuilt (and pushed into the shared cache directory),
then both decide that f3.o
must be rebuilt...
This won't cause any actual build problems--both
builds will succeed,
generate correct output files,
and populate the cache--but
it does represent wasted effort.
To alleviate such contention for the cache,
you can use the --random command-line option
to tell &SCons; to build dependencies
in a random order:
% scons -Q --random
cc -o f3.o -c f3.c
cc -o f1.o -c f1.c
cc -o f5.o -c f5.c
cc -o f2.o -c f2.c
cc -o f4.o -c f4.c
cc -o prog f1.o f2.o f3.o f4.o f5.o
Multiple builds using the --random option
will usually build their dependencies in different,
random orders,
which minimizes the chances for a lot of
contention for same-named files
in the shared cache directory.
Multiple simultaneous builds might still race to try to build
the same target file on occasion,
but long sequences of inefficient contention
should be rare.
Note, of course,
the --random option
will cause the output that &SCons; prints
to be inconsistent from invocation to invocation,
which may be an issue when
trying to compare output from different build runs.
If you want to make sure dependencies will be built
in a random order without having to specify
the --random on very command line,
you can use the &SetOption; function to
set the random option
within any &SConscript; file:
Program('prog',
['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])
SetOption('random', 1)
Program('prog',
['f1.c', 'f2.c', 'f3.c', 'f4.c', 'f5.c'])