summaryrefslogtreecommitdiffstats
path: root/doc/python10/process.xml
blob: 6b049be77974359148b4da3f249162986ba9bf51 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
<para>

  The &SCons; project has paid particular attention from day one to the
  development process. One of the first internal documents produced was
  a set of Developer's Guidelines to provide a loose framework for what
  we were trying to accomplish and how we would go about accomplishing
  it. These Guidelines cover things like:

</para>

<itemizedlist>

  <listitem>
    <para>

      &SCons; will be written to Python version 1.5.2 (to ensure
      usability by a wide install base).

    </para>
  </listitem>

  <listitem>
    <para>

      How &SCons; is be tested: which infrastructure modules to use,
      what platforms to test on, etc.

    </para>
  </listitem>

  <listitem>
    <para>

      Expectations for developers (subscribe to the mailing list,
      encouraged to register at SourceForge).

    </para>
  </listitem>

  <listitem>
    <para>

      Brief outline of how to use the change management systems (Aegis and
      CVS) for &SCons; development;.

    </para>
  </listitem>

</itemizedlist>

<para>

  Establishing these guidelines up front had two purposes: 1)
  Demonstrate the seriousness of the project to anyone wondering about
  joining the effort; 2) Give potential developers an idea up front as
  to whether their development style would mesh with the rest of the
  project.

</para>

<section>
  <title>Aegis</title>

  <para>

    One of the most important aspects of the &SCons; development process
    is the use of Peter Miller's Aegis change management system. I
    had been using Aegis for personal projects for several years, and
    found its development methodology vastly improved the quality of my
    programming. I was consequently committed to using it for &SCons;
    development.

  </para>

  <para>

    Aegis provides a number of things, including:

  </para>

  <itemizedlist>

    <listitem>
      <para>

        A flexible source code control and branching model.

      </para>
    </listitem>

    <listitem>
      <para>

        A defined process with separate development, review and
        integration steps.

      </para>
    </listitem>

    <listitem>
      <para>

        A distributed development model based on distribution of atomic
        change sets.

      </para>
    </listitem>

  </itemizedlist>

  <para>

    The single most important reason for using Aegis, however, is its
    management of automated tests as part of the development process.

  </para>

</section>

<section>
  <title>Testing, Testing, Testing</title>

  <para>

    The &SCons; project has made extensive use of automated tests from day
    one, taking inspiration mostly from Aegis, partly from the eXtreme
    Programming model, and with a little home-brew scripting for glue.

  </para>

  <section>
    <title>Testing Criteria</title>

    <para>

      The underlying criteria for testing changes to the &SCons; code
      are taken from Aegis:

    </para>

    <itemizedlist>

      <listitem>
        <para>

          Every change must have one or more new or modified tests
          checked in along with the code.

        </para>
      </listitem>

      <listitem>
        <para>

          The new code being checked in must pass all of the new and/or
          modified tests.

        </para>
      </listitem>

      <listitem>
        <para>

          The <emphasis>old</emphasis>, already checked-in code in must
          <emphasis>fail</emphasis> all of the new and/or modified
          tests.

        </para>
      </listitem>

      <listitem>
        <para>

          The new code being checked in must pass all unmodified,
          already checked-in tests.

        </para>
      </listitem>

    </itemizedlist>

    <para>

      In practice, these restrictions can be overridden as necessary--for
      example, when changing comments or documentation.

    </para>

    <para>

      The criterion that surprises many people is having the old code
      fail the tests in the change. This makes sure that the new tests
      or modified tests really do exercise the bug fix or feature being
      added by the change.

    </para>

    <para>

      Together, these criteria ensure that every newly checked-in
      version &SCons; conforms to defined behavior, as defined by
      the tests. Whenever a bug is found, its fix is checked in with
      a new or modified test that guarantees the bug will not recur
      in the future. We have already built up a regression test base
      of almost 90 tests that cover the vast majority of &SCons;'
      functionality.

    </para>

  </section>

  <section>
    <title>Testing Infrastructure</title>

    <para>

      Testing standards are no good if they're too much of a burden for
      developers, who will at best work around or ignore the testing
      requirements, or at worst stop contributing code and go join a
      project that's more fun. To this end, good testing infrastructure
      that makes it easy to write tests is crucial.

    </para>

    <para>

      &SCons; development uses two development methodologies, one for
      the individual modules in the build engine, and the other for
      end-to-end tests of the &SCons; script.

    </para>

    <para>

      For the build engine modules, we use PyUnit. Every change to a
      build engine module must have a change to its corresponding unit
      tests, which live side-by-side in a separate file that imports
      module. As we build up a large body of unit tests, this ensures
      that the build engine will perform correctly whenever someone uses
      it in some application other than the &SCons; script itself.

    </para>

    <para>

      For end-to-end script tests, we have developed two modules to make
      writing tests easy. The first, <filename>TestCmd.py</filename>,
      is a generic module for
      testing commands or scripts (in any language, not just Python).

      The second module, <filename>TestScons.py</filename>,
      is a subclass of the generic
      <filename>TestCmd.py</filename> module.
      <filename>TestScons.py</filename>
      takes care of initialization and
      displaying error conditions
      specific to testing &SCons;.

    </para>

    <para>
      
      In practice, simple tests only
      need to initialize a test object, use the object to write some
      input files, run &SCons;, and then check whatever criteria
      determine whether the test passed or failed. A complete test of
      the &Program; method, for example, looks like this:

    </para>

    <programlisting>
      test = TestSCons.TestSCons()

      test.write('SConstruct',
      """env = Environment()
      env.Program(target = 'foo', source = 'foo.c')
      """)

      test.write('foo.c',
      """
      int
      main(int argc, char *argv[])
      {
          argv[argc++] = "-"; /* dummy use of args */
          printf("foo.c successfully compiled\\n");
          exit (0);
      }
      """)

      test.run(arguments = 'foo') # runs SCons

      test.run(program = test.workpath('foo'))

      test.fail_test(test.stdout() != "foo.c successfully compiled\n")

      test.pass_test()
    </programlisting>

  </section>

</section>

<section>
  <title>SourceForge</title>

  <para>

    Registration of the &SCons; project was approved at SourceForge on
    29 June 2001.  Within a week, the initial code base was checked in,
    mailing lists were created, and the web site was set up. We started
    making use of the task-list manager to track what we had to finish
    for initial release.

  </para>

  <para>

    The obvious complication was how to use
    structured testing methodology of Aegis when SourceForge uses
    CVS for source control. Not using the SourceForge CVS tree would
    have had two significant disadvantages: one, missing out on the
    archiving and central location in the event of disaster; two, people
    coming to the SourceForge project page wouldn't be able to browse
    the source.  The latter was particularly important in
    the early stages of development, in order to avoid any impression
    that this was Yet Another Project that starts with a bang and then
    dwindles as the initial enthusiasm starts to wear off.

  </para>

  <para>

    The solution was to use the SourceForge CVS repository for read-only
    access to the source. &SCons; developers are welcome to use CVS for
    their development, but the changes are <emphasis>not</emphasis>
    committed to the SourceForge repository. Instead, patches are sent
    to the integrator for processing through Aegis. When the change
    has been integrated into the Aegis repository, a home-brew
    script translates the Aegis change into a virtual shell script
    of commands that copy the necessary files from Aegis and check them
    in to CVS at SourceForge.

  </para>

  <para>

    (In practice, write access is not actually disabled for registered
    developers, but if they do make any changes directly at SourceForge,
    they can be overwritten at the next Aegis update.)

  </para>

</section>