summaryrefslogtreecommitdiffstats
path: root/doc/design/overview.sgml
blob: 04671be0ad2b07f75a3c298d50e5eabfb15626c4 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
<!--

  Copyright (c) 2001 Steven Knight

  Permission is hereby granted, free of charge, to any person obtaining
  a copy of this software and associated documentation files (the
  "Software"), to deal in the Software without restriction, including
  without limitation the rights to use, copy, modify, merge, publish,
  distribute, sublicense, and/or sell copies of the Software, and to
  permit persons to whom the Software is furnished to do so, subject to
  the following conditions:

  The above copyright notice and this permission notice shall be included
  in all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
  KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
  WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
  NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
  LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
  OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
  WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

-->

<section id="sect-architecture">
 <title>Architecture</title>

 <para>

   The heart of &SCons; is its <emphasis>Build Engine</emphasis>.
   The &SCons; Build Engine is a Python module
   that manages dependencies between
   external objects
   such as files or database records.
   The Build Engine is designed to
   be interface-neutral
   and easily embeddable in any
   software system that needs dependency
   analysis between updatable objects.

 </para>

 <para>

   The key parts of the Build Engine architecture
   are captured in the following quasi-UML diagram:

 </para>

<REMARK>
Including this figure makes our PDF build blow up.
The figure, however,
is left over from the Software Carpentry contest
and is therefore old, out-of-date, and needs to be redone anyway.
This is where it will go, anyway...
</REMARK>

   <!--
   YARG!  THIS MAKES THE PDF BUILD BLOW UP.  HELP!
   <figure>
    <title>&SCons; Architecture</title>
    <graphic fileref="engine.jpg">
   </figure>
   -->

 <para>

   The point of &SCons; is to manage
   dependencies between arbitrary external objects.
   Consequently, the Build Engine does not restrict or specify
   the nature of the external objects it manages,
   but instead relies on subclass of the &Node;
   class to interact with the external system or systems
   (file systems, database management systems)
   that maintain the objects being examined or updated.

 </para>

 <para>

   The Build Engine presents to the software system in
   which it is embedded
   a Python API for specifying source (input) and target (output) objects,
   rules for building/updating objects,
   rules for scanning objects for dependencies, etc. 
   Above its Python API,
   the Build Engine is completely
   interface-independent,
   and can be encapsulated by any other software
   that supports embedded Python.

 </para>

 <para>

   Software that chooses to use the Build Engine
   for dependency management
   interacts with it
   through <emphasis>Construction Environments</emphasis>.
   A Construction Environment consists
   of a dictionary of environment variables,
   and one or more associated
   &Scanner; objects
   and &Builder; objects.
   The Python API is used to
   form these associations.

 </para>

 <para>

   A &Scanner; object specifies
   how to examine a type of source object
   (C source file, database record)
   for dependency information.
   A &Scanner; object may use
   variables from the associated
   Construction Environment
   to modify how it scans an object:
   specifying a search path for included files,
   which field in a database record to consult,
   etc.

 </para>

 <para>

   A &Builder; object specifies
   how to update a type of target object:
   executable program, object file, database field, etc.
   Like a &Scanner; object,
   a &Builder; object may use
   variables from the associated
   Construction Environment
   to modify how it builds an object:
   specifying flags to a compiler,
   using a different update function,
   etc.

 </para>

 <para>

   &Scanner; and &Builder; objects will return one or more
   &Node; objects that represent external objects.
   &Node; objects are the means by which the
   Build Engine tracks dependencies:
   A &Node; may represent a source (input) object that
   should already exist,
   or a target (output) object which may be built,
   or both.
   The &Node; class is sub-classed to 
   represent external objects of specific type:
   files, directories, database fields or records, etc.
   Because dependency information, however,
   is tracked by the top-level &Node; methods and attributes,
   dependencies can exist
   between nodes representing different external object types.
   For example,
   building a file could be made
   dependent on the value of a given
   field in a database record,
   or a database table could depend
   on the contents of an external file.

 </para>

 <para>

   The Build Engine uses a &Job; class (not displayed)
   to manage the actual work of updating external target objects:
   spawning commands to build files,
   submitting the necessary commands to update a database record,
   etc.
   The &Job; class has sub-classes
   to handle differences between spawning
   jobs in parallel and serially.

 </para>

 <para>

   The Build Engine also uses a
   &Signature; class (not displayed)
   to maintain information about whether
   an external object is up-to-date.
   Target objects with out-of-date signatures
   are updated using the appropriate
   &Builder; object.

 </para>

   <!-- BEGIN HTML -->

   <!--
   Details on the composition, methods,
   and attributes of these classes
   are available in the  A HREF="internals.html" Internals /A  page.
   -->

   <!-- END HTML -->

</section>



<section id="sect-engine">
 <title>Build Engine</title>

 <para>

   More detailed discussion of some of the
   Build Engine's characteristics:

 </para>

 <section>
  <title>Python API</title>

   <para>

   The Build Engine can be embedded in any other software
   that supports embedding Python:
   in a GUI,
   in a wrapper script that
   interprets classic <filename>Makefile</filename> syntax,
   or in any other software that
   can translate its dependency representation
   into the appropriate calls to the Build Engine API.
   <!--<xref linkend="chap-native">--> describes in detail
   the specification for a "Native Python" interface
   that will drive the &SCons; implementation effort.

   </para>

 </section>

 <section>
 <title>Single-image execution</title>

   <para>

   When building/updating the objects,
   the Build Engine operates as a single executable
   with a complete Directed Acyclic Graph (DAG)
   of the dependencies in the entire build tree.
   This is in stark contrast to the
   commonplace recursive use of Make
   to handle hierarchical directory-tree builds.

   </para>

 </section>

 <section>
 <title>Dependency analysis</title>

   <para>

   Dependency analysis is carried out via digital signatures
   (a.k.a. "fingerprints").
   Contents of object are examined and reduced
   to a number that can be stored and compared to
   see if the object has changed.
   Additionally, &SCons; uses the same
   signature technique on the command-lines that
   are executed to update an object.
   If the command-line has changed since the last time,
   then the object must be rebuilt.

   </para>

 </section>

 <section>
 <title>Customized output</title>

   <para>

   The output of Build Engine is customizable
   through user-defined functions.
   This could be used to print additional desired
   information about what &SCons; is doing,
   or tailor output to a specific build analyzer,
   GUI, or IDE.

   </para>

 </section>

 <section>
 <title>Build failures</title>

   <para>

   &SCons; detects build failures via the exit status from the tools
   used to build the target files.  By default, a failed exit status
   (non-zero on UNIX systems) terminates the build with an appropriate
   error message.  An appropriate class from the Python library will
   interpret build-tool failures via an OS-independent API.

   </para>

   <para>

   If multiple tasks are executing in a parallel build, and one tool
   returns failure, &SCons; will not initiate any further build tasks,
   but allow the other build tasks to complete before terminating.

   </para>

   <para>

   A <option>-k</option> command-line option may be used to ignore
   errors and continue building other targets.  In no case will a target
   that depends on a failed build be rebuilt.

   </para>

 </section>

</section>



<section id="sect-interfaces">
 <title>Interfaces</title>

 <para>

   As previously described,
   the &SCons; Build Engine
   is interface-independent above its Python API,
   and can be embedded in any software system
   that can translate its dependency requirements
   into the necessary Python calls.

 </para>

 <para>

   The "main" &SCons; interface
   for implementation purposes,
   uses Python scripts as configuration files.
   Because this exposes the Build Engine's Python API to the user,
   it is current called the "Native Python" interface.

 </para>

 <para>

   This section will also discuss
   how &SCons; will function in the context
   of two other interfaces:
   the &Makefile; interface of the classic &Make; utility,
   and a hypothetical graphical user interface (GUI).

 </para>

 <section>
  <title>Native Python interface</title>

  <para>

   The Native Python interface is intended to be the primary interface
   by which users will know &SCons;--that is,
   it is the interface they will use
   if they actually type &SCons; at a command-line prompt.

  </para>

  <para>

   In the Native Python interface, &SCons; configuration files are simply
   Python scripts that directly invoke methods from the Build Engine's
   Python API to specify target files to be built, rules for building
   the target files, and dependencies.  Additional methods, specific to
   this interface, are added to handle functionality that is specific to
   the Native Python interface:  reading a subsidiary configuration file;
   copying target files to an installation directory; etc.

  </para>

  <para>

   Because configuration files are Python scripts, Python flow control
   can be used to provide very flexible manipulation of objects and
   dependencies.  For example, a function could be used to invoke a common
   set of methods on a file, and called iteratively over an array of
   files.

  </para>

  <para>

   As an additional advantage, syntax errors in &SCons; Native Python
   configuration files will be caught by the Python parser.  Target-building
   does not begin until after all configuration files are read, so a syntax
   error will not cause a build to fail half-way.

  </para>

 </section>

 <section>
  <title>Makefile interface</title>

  <para>

   An alternate &SCons; interface would provide backwards
   compatibility with the classic &Make utility.
   This would be done by embedding the &SCons; Build Engine
   in a Python script that can translate existing
   &Makefile;s into the underlying calls to the
   Build Engine's Python API
   for building and tracking dependencies.
   Here are approaches to solving some of the issues
   that arise from marrying these two pieces:

  </para>

  <itemizedlist>

   <listitem>
   <para>
   &Makefile; suffix rules can be translated
   into an appropriate &Builder; object
   with suffix maps from the Construction Environment.
   </para>
   </listitem>

   <listitem>
   <para>
   Long lists of static dependences
   appended to a &Makefile; by
   various <command>"make depend"</command> schemes
   can be preserved
   but supplemented by
   the more accurate dependency information
   provided by &Scanner; objects.
   </para>
   </listitem>

   <listitem>
   <para>
   Recursive invocations of &Make;
   can be avoided by reading up
   the subsidiary &Makefile; instead.
   </para>
   </listitem>

  </itemizedlist>

  <para>

   Lest this seem like too outlandish an undertaking,
   there is a working example of this approach:
   Gary Holt's &Makepp; utility
   is a Perl script that provides
   admirably complete parsing of complicated &Makefile;s
   around an internal build engine inspired,
   in part, by the classic <application>Cons</application> utility.

  </para>

 </section>

 <section>
  <title>Graphical interfaces</title>

  <para>

   The &SCons; Build Engine
   is designed from the ground up to be embedded
   into multiple interfaces.
   Consequently, embedding the dependency capabilities
   of &SCons; into graphical interface
   would be a matter of mapping the
   GUI's dependency representation
   (either implicit or explicit)
   into corresponding calls to the Python API
   of the &SCons; Build Engine.

  </para>

  <para>

   Note, however, that this proposal leaves the problem of
   designed a good graphical interface
   for representing software build dependencies
   to people with actual GUI design experience...

  </para>

 </section>

</section>