1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
# __COPYRIGHT__
This directory contains timing configurations for SCons.
Each configuration exists in a subdirectory. The controlling script
is named TimeSCons-run.py for the configuration. The TimeSCons-run.py
scripts use TestSCons.TimeSCons, a subclass of TestSCons.TestSCons (both
defined in ../QMTest/TestSCons.py), to manage execution of the timing
runs.
Unlike the TestSCons.TestSCons base class, the TestSCons.TimeSCons
subclass copies the contents of its containing directory to the temporary
working directory. (It avoids copying the .svn directory, and any files
or directories that start with the string "TimeSCons-".) This allows
the timing configuration files to be checked in directly to our source
code management system, instead of requiring that they be created from
in-line data inside the script.
The simplest-possible TimeSCons-run.py script would look like:
import TestSCons
TestSCons.TimeSCons().main()
The above script would end up executing a SConstruct file configuration
in a temporary directory. The main() method is the standard interface
for a timing run. See its docstring for precisely what it does.
Although the TestSCons.TimeSCons subclass copies its directory contents to
a temporary working directory for the timing run, because it is a subclass
of TestSCons.TestSCons, it *can* also create files or directories from
in-line data. This is typically done when it's necessary to create
hundreds of identical input files or directories before running the
timing test, to avoid cluttering our SCM system with hundreds of otherwise
meaningless files.
STRUCTURE OF A TIMING CONFIGURATION
===================================
A timing configuration should be in a subdirectory and should contain
at least the following three files:
TimeSCons-run.py
The script that controls the timing run. It looks a lot
like the test scripts (since it uses the same infrastructure)
except that you instantiate TestSCons.TimeSCons object, not a
TestSCons.TestSCons object.
Typically you want to initialize the object with a "variables"
dict containing one or more parameters whose values control the
scale of the configuration. This would typically be the number
of source files, directories to scan, etc. The TimeSCons-run.py
script can then use the value of those variables to generate that
many copies of input source files, or directories, or what have,
from in-line data, instead of having to check in a couple hundred
files for a large configuration.
These variables get passed to the timed SCons invocation as
ARGUMENT= arguments on the command line, so the SConstruct
file can use it to loop through the right number of files /
directories / what have you.
SConstruct
This is the actual SCons configuration that gets tested. It has
access to the variable(s) that control the configuration as
ARGUMENTS from the command line.
It's possible for the SConstruct file to do additional set up of
input files and the like, but in general that should be kept to
a minimum. We want what the SConscript file does to be dominated
by the actual execution we're timing, not initialization stuff,
so most initialization is better left in TimeSCons-run.py.
config.js
This gives our buildbot information about the timing configuration
(specifically, the title) for display.
Note that it's perfectly acceptable to check in additional files that
may be necessary for your configuration. They'll get copied to the
temporary directory used to run the timing.
RUNNING YOUR TIMING CONFIGURATION
=================================
Because the TimeSCons.py module is a subclass of the whole TestSCons
hierarchy, you use a normal runtest.py invocation to run the timings
configuration:
$ python runtest.py timings/Configuration/TimeSCons-run.py
This runs the entire timing configuration, which actually runs SCons
itself three times:
1) First, with the --help option, to exit immediately after
reading the SConscript file(s). This allows us to get a
rough independent measurement of how much startup cost is
involved in this configuration, so that the amount can be
discounted from the
2) A full build.
3) An rebuild of the full build, which is presumably up-to-date.
When you execute runtest.py from the command line, the output of
each SCons run is printed on standard output. (Note this means
that the output can get pretty large if the timing configuration
involves thousands of files.)
The collected memory and time statistics for each run are printed
on standard output, each with the prefix "TRACE:". These are the
lines that the buildbot grabs to collect the timing statistics for
the graphs available on the web site.
CALIBRATING YOUR TIMING CONFIGURATION
=====================================
One goal we have for timing configurations is that they should take
about 10 seconds to run on our buildbot timing system, which is an older,
slower system than most.
Per above, you presumably defined one or more variables that control the
"size" of your configuration: the number of input files, directories,
etc. The timing infrastructure actually reports the value of these
variables in a way that lets us automate the process of adjusting the
variable values to run within a specified amount of time.
The bin/calibrate.py will run your configuration repeatedly, adjusting
the value(s) of the variable(s) that control your configuration until
it gets three successive runs that take between 9.5 and 10.0 seconds
(by default, options let you adjust the range):
$ python bin/calibrate.py timings/MyNewTimingConfiguration/TimeSCons-run.py
run 1: 3.124: TARGET_COUNT=50
run 2: 11.936: TARGET_COUNT=160
run 3: 9.175: TARGET_COUNT=134
run 4: 10.489: TARGET_COUNT=146
run 5: 9.798: TARGET_COUNT=139
run 6: 9.695: TARGET_COUNT=139
run 7: 9.670: TARGET_COUNT=139
$
If you have multiple variables, it will adjust *all* of the variables
on each run. In other words, the proportion between your variables will
remain (relatively) constant.
Of course, this needs to be run on a quiet system for the numbers
to converge. And what you really need to do before committing a
configuration is run bin/calibrate.py on the actual system that runs
our Buildbot timings. For that, see Bill Deegan or Steven Knight.
Once you have "good" values for your variables, put them in your
TimeSCons-run.py and you should be good to go. Note that we've started a
convention of also pasting the run output from calibrate.py into comments
in the TimeSCons-run.py, just to preserve some of the historical context
that led to certain values being chosen.
ADDING A NEW TIMING CONFIGURATION
=================================
In addition to creating a subdirectory with at least the pieces listed
above in the "STRUCTURE" section and "CALIBRATING" your variable(s),
you need to update the following file in this directory:
index.html
Add an entry to the test_map dictionary for the subdirectory
you just created.
That should be it before checkin. After checkin, one of the Buildbot
administrators (currently Bill Deegan or Steven Knight) needs to update
and restart the Buildbot master so that it will start executing the
build step to run the new timing configuration.
|
