The &SCons; project has paid particular attention from day one to the
development process. One of the first internal documents produced was
a set of Developer's Guidelines to provide a loose framework for what
we were trying to accomplish and how we would go about accomplishing
it. These Guidelines cover things like:
&SCons; will be written to Python version 1.5.2 (to ensure
usability by a wide install base).
How &SCons; is be tested: which infrastructure modules to use,
what platforms to test on, etc.
Expectations for developers (subscribe to the mailing list,
encouraged to register at SourceForge).
Brief outline of how to use the change management systems (Aegis and
CVS) for &SCons; development;.
Establishing these guidelines up front had two purposes: 1)
Demonstrate the seriousness of the project to anyone wondering about
joining the effort; 2) Give potential developers an idea up front as
to whether their development style would mesh with the rest of the
project.
Aegis
One of the most important aspects of the &SCons; development process
is the use of Peter Miller's Aegis change management system. I
had been using Aegis for personal projects for several years, and
found its development methodology vastly improved the quality of my
programming. I was consequently committed to using it for &SCons;
development.
Aegis provides a number of things, including:
A flexible source code control and branching model.
A defined process with separate development, review and
integration steps.
A distributed development model based on distribution of atomic
change sets.
The single most important reason for using Aegis, however, is its
management of automated tests as part of the development process.
Testing, Testing, Testing
The &SCons; project has made extensive use of automated tests from day
one, taking inspiration mostly from Aegis, partly from the eXtreme
Programming model, and with a little home-brew scripting for glue.
Testing Criteria
The underlying criteria for testing changes to the &SCons; code
are taken from Aegis:
Every change must have one or more new or modified tests
checked in along with the code.
The new code being checked in must pass all of the new and/or
modified tests.
The old, already checked-in code in must
fail all of the new and/or modified
tests.
The new code being checked in must pass all unmodified,
already checked-in tests.
In practice, these restrictions can be overridden as necessary--for
example, when changing comments or documentation.
The criterion that surprises many people is having the old code
fail the tests in the change. This makes sure that the new tests
or modified tests really do exercise the bug fix or feature being
added by the change.
Together, these criteria ensure that every newly checked-in
version &SCons; conforms to defined behavior, as defined by
the tests. Whenever a bug is found, its fix is checked in with
a new or modified test that guarantees the bug will not recur
in the future. We have already built up a regression test base
of almost 90 tests that cover the vast majority of &SCons;'
functionality.
Testing Infrastructure
Testing standards are no good if they're too much of a burden for
developers, who will at best work around or ignore the testing
requirements, or at worst stop contributing code and go join a
project that's more fun. To this end, good testing infrastructure
that makes it easy to write tests is crucial.
&SCons; development uses two development methodologies, one for
the individual modules in the build engine, and the other for
end-to-end tests of the &SCons; script.
For the build engine modules, we use PyUnit. Every change to a
build engine module must have a change to its corresponding unit
tests, which live side-by-side in a separate file that imports
module. As we build up a large body of unit tests, this ensures
that the build engine will perform correctly whenever someone uses
it in some application other than the &SCons; script itself.
For end-to-end script tests, we have developed two modules to make
writing tests easy. The first, TestCmd.py,
is a generic module for
testing commands or scripts (in any language, not just Python).
The second module, TestScons.py,
is a subclass of the generic
TestCmd.py module.
TestScons.py
takes care of initialization and
displaying error conditions
specific to testing &SCons;.
In practice, simple tests only
need to initialize a test object, use the object to write some
input files, run &SCons;, and then check whatever criteria
determine whether the test passed or failed. A complete test of
the &Program; method, for example, looks like this:
test = TestSCons.TestSCons()
test.write('SConstruct',
"""env = Environment()
env.Program(target = 'foo', source = 'foo.c')
""")
test.write('foo.c',
"""
int
main(int argc, char *argv[])
{
argv[argc++] = "-"; /* dummy use of args */
printf("foo.c successfully compiled\\n");
exit (0);
}
""")
test.run(arguments = 'foo') # runs SCons
test.run(program = test.workpath('foo'))
test.fail_test(test.stdout() != "foo.c successfully compiled\n")
test.pass_test()
SourceForge
Registration of the &SCons; project was approved at SourceForge on
29 June 2001. Within a week, the initial code base was checked in,
mailing lists were created, and the web site was set up. We started
making use of the task-list manager to track what we had to finish
for initial release.
The obvious complication was how to use
structured testing methodology of Aegis when SourceForge uses
CVS for source control. Not using the SourceForge CVS tree would
have had two significant disadvantages: one, missing out on the
archiving and central location in the event of disaster; two, people
coming to the SourceForge project page wouldn't be able to browse
the source. The latter was particularly important in
the early stages of development, in order to avoid any impression
that this was Yet Another Project that starts with a bang and then
dwindles as the initial enthusiasm starts to wear off.
The solution was to use the SourceForge CVS repository for read-only
access to the source. &SCons; developers are welcome to use CVS for
their development, but the changes are not
committed to the SourceForge repository. Instead, patches are sent
to the integrator for processing through Aegis. When the change
has been integrated into the Aegis repository, a home-brew
script translates the Aegis change into a virtual shell script
of commands that copy the necessary files from Aegis and check them
in to CVS at SourceForge.
(In practice, write access is not actually disabled for registered
developers, but if they do make any changes directly at SourceForge,
they can be overwritten at the next Aegis update.)