diff options
author | Steven Knight <knight@baldmt.com> | 2007-09-19 12:52:29 (GMT) |
---|---|---|
committer | Steven Knight <knight@baldmt.com> | 2007-09-19 12:52:29 (GMT) |
commit | d43c3fa04d636344c9d38794182970914a85ce0b (patch) | |
tree | 0c824d7694f6a54b5fae780cfca8d04cb02b85bc /src/engine | |
parent | eecd55b0591926f04dc812088702587ab1d6cc10 (diff) | |
download | SCons-d43c3fa04d636344c9d38794182970914a85ce0b.zip SCons-d43c3fa04d636344c9d38794182970914a85ce0b.tar.gz SCons-d43c3fa04d636344c9d38794182970914a85ce0b.tar.bz2 |
Merged revisions 2302-2362,2364-2452 via svnmerge from
http://scons.tigris.org/svn/scons/branches/core
................
r2311 | stevenknight | 2007-08-17 07:51:31 -0500 (Fri, 17 Aug 2007) | 2 lines
Fix the CHECKPOINT= help text.
................
r2313 | stevenknight | 2007-08-17 13:12:13 -0500 (Fri, 17 Aug 2007) | 2 lines
Make sure the --debug=time works when -h is specified, too.
................
r2320 | stevenknight | 2007-08-18 08:54:49 -0500 (Sat, 18 Aug 2007) | 3 lines
Don't execute any Configure() actions while reading SConscript files
when -c or -h or -H are specified.
................
r2321 | stevenknight | 2007-08-20 08:48:57 -0500 (Mon, 20 Aug 2007) | 2 lines
Cleaner test failure if before-and-after PostScript files don't match.
................
r2322 | stevenknight | 2007-08-20 11:02:57 -0500 (Mon, 20 Aug 2007) | 2 lines
Remove function definitions that make some Java versions fail.
................
r2354 | stevenknight | 2007-08-20 14:26:13 -0500 (Mon, 20 Aug 2007) | 3 lines
Relax test conditions for two scripts that have differing behavior
under different versions of Java + SWIG (+ gcc...?).
................
r2355 | stevenknight | 2007-08-20 14:29:36 -0500 (Mon, 20 Aug 2007) | 2 lines
Fix the SConstruct file so it doesn't die if Subversion isn't installed.
................
r2356 | stevenknight | 2007-08-20 22:26:35 -0500 (Mon, 20 Aug 2007) | 3 lines
Update to latest Test{Cmd,Common}.py, with better avoidance of
race conditions on temporary file and directory names.
................
r2373 | stevenknight | 2007-08-27 10:47:21 -0500 (Mon, 27 Aug 2007) | 2 lines
Windows portability fixes in tests.
................
r2377 | stevenknight | 2007-08-27 12:33:47 -0500 (Mon, 27 Aug 2007) | 2 lines
Add a null command list to suppress the baseline build when testing.
................
r2380 | stevenknight | 2007-08-27 16:33:42 -0500 (Mon, 27 Aug 2007) | 616 lines
Merged revisions 1503-1543,1545-1546,1548-1558,1560-1562,1564-1886,1888-1909,1911-1941,1943,1945-1955,1957-1960,1962-1963,1965-1999,2001-2019,2021-2037,2039-2061,2063-2067,2069-2146,2148-2150,2152-2156,2158-2199,2201-2208,2210-2212,2214-2238,2240,2242-2288,2290-2291,2293-2305,2307-2364,2366-2373,2375-2377,2379 via svnmerge from
http://scons.tigris.org/svn/scons/branches/sigrefactor
........
r1550 | stevenknight | 2006-07-27 21:40:08 -0500 (Thu, 27 Jul 2006) | 3 lines
First cut at storing csig separately in the NodeInfo,
but still using bsigs for up-to-date checks.
........
r1551 | stevenknight | 2006-07-28 07:07:27 -0500 (Fri, 28 Jul 2006) | 2 lines
Refactor the use of dictify() to avoid __builtin__ issues with zip on 1.5.
........
r1552 | stevenknight | 2006-07-28 10:05:18 -0500 (Fri, 28 Jul 2006) | 3 lines
Have msvsTests.py print why it's not executing tests
when on a non-win32 system.
........
r1553 | stevenknight | 2006-07-28 11:28:37 -0500 (Fri, 28 Jul 2006) | 3 lines
Refactor when NodeInfo objects get instantiated (when requested
explicitly, not as a side effect of creating a BuildInfo object).
........
r1554 | stevenknight | 2006-07-28 11:42:53 -0500 (Fri, 28 Jul 2006) | 3 lines
Refactor test/{Source,Target}Signatures.py to make changing their
semantics easier.
........
r1555 | stevenknight | 2006-07-28 13:18:35 -0500 (Fri, 28 Jul 2006) | 6 lines
Eliminate use of build signatures, changing interfaces and semantics
of SourceSignature() and TargetSignatures(), accordingly. Make use
of content signatures the default behavior. Get rid of most uses
the Sig/*.py calculator modules and passing around a "calc" argument
everywhere.
........
r1556 | stevenknight | 2006-07-28 15:14:03 -0500 (Fri, 28 Jul 2006) | 3 lines
Fix use of CacheDir() with Alias and Value Nodes.
Refactor sub-tests in test/CacheDir.py into separate test scripts.
........
r1557 | stevenknight | 2006-07-28 21:40:31 -0500 (Fri, 28 Jul 2006) | 2 lines
Finish the necessary coding to remove the no-longer-needed Sig/*.py modules.
........
r1560 | stevenknight | 2006-07-29 05:52:27 -0500 (Sat, 29 Jul 2006) | 3 lines
Move current_*() methods for the different flavors of file comparison into
(e.g.) the Node.FS.File class itself, not its FileNodeInfo class.
........
r1562 | stevenknight | 2006-07-29 08:46:23 -0500 (Sat, 29 Jul 2006) | 2 lines
Fix the default returned Boolean value of changed_{content,timestamp}().
........
r1564 | stevenknight | 2006-07-29 10:51:24 -0500 (Sat, 29 Jul 2006) | 3 lines
Fix the --debug=explain option. Move test/explain.py to
test/option/debug-explain.py.
........
r1565 | stevenknight | 2006-07-29 16:09:31 -0500 (Sat, 29 Jul 2006) | 2 lines
Refactor Taskmaster to visit source file Nodes in its walk.
........
r1566 | stevenknight | 2006-07-29 17:50:38 -0500 (Sat, 29 Jul 2006) | 4 lines
Merge Node.FS._cur2() into Node.FS.is_up_to_date().
Use a Node.FS.*.make_ready() to handle disambiguation.
Split the guts of has_src_builder() into a separate find_src_builder() method.
........
r1567 | stevenknight | 2006-07-29 17:55:36 -0500 (Sat, 29 Jul 2006) | 2 lines
Make the sconsign script print the stored action information.
........
r1568 | stevenknight | 2006-07-29 17:59:03 -0500 (Sat, 29 Jul 2006) | 2 lines
Refactor for test readability.
........
r1569 | stevenknight | 2006-07-29 20:32:26 -0500 (Sat, 29 Jul 2006) | 2 lines
Get rid of the now-unnecessary "module" argument to SConsign methods.
........
r1570 | stevenknight | 2006-07-29 22:05:47 -0500 (Sat, 29 Jul 2006) | 3 lines
Move content signature calcuation to an explicit make_ready()
function. Collect NodeInfo field updating in the base class.
........
r1571 | stevenknight | 2006-07-30 06:26:17 -0500 (Sun, 30 Jul 2006) | 2 lines
Eliminte the no-longer-necessary is_pseudo_derived() method.
........
r1572 | stevenknight | 2006-07-30 07:53:40 -0500 (Sun, 30 Jul 2006) | 2 lines
Add tgt_sig_type arguments to various changed() methods' calling arguments.
........
r1573 | stevenknight | 2006-07-30 08:17:43 -0500 (Sun, 30 Jul 2006) | 2 lines
More informative output from test/SideEffect.py when it fails.
........
r1574 | stevenknight | 2006-07-30 09:53:11 -0500 (Sun, 30 Jul 2006) | 3 lines
Restore the ability to fall back gracefully to timestamps if the running
version of Python has no md5 module available.
........
r1575 | stevenknight | 2006-08-02 20:21:04 -0500 (Wed, 02 Aug 2006) | 3 lines
Fix a failing Qt test by calling disambiguate() before make_ready()
for all of the targets in a list, not just the first.
........
r1646 | stevenknight | 2006-10-17 17:21:58 -0500 (Tue, 17 Oct 2006) | 6 lines
Move all the scons.org stuff from the scons source tree itself to a
directory next to the trunk, and delete the copies from the branches.
There's a lot of stuff there (what with all of the documentation of
the different versions) and it's ridiculous to make everyone sync it
just to work on the code.
........
r1888 | stevenknight | 2007-04-17 14:57:47 -0500 (Tue, 17 Apr 2007) | 2 lines
More efficient/cleaner code for dictifying --debug=explain info.
........
r1889 | stevenknight | 2007-04-17 15:55:00 -0500 (Tue, 17 Apr 2007) | 3 lines
More efficient taskmaster: filter the list once for anything that hasn't
been built, then extract a sub-list of anything that hasn't been visited.
........
r1894 | stevenknight | 2007-04-25 13:07:29 -0500 (Wed, 25 Apr 2007) | 4 lines
Push conversion of individual NodeInfo objects into the objects themselves
(instead of having the FileBuildInfo class assume that every dependency
is a file system Entry of some kind).
........
r1895 | stevenknight | 2007-05-04 23:49:13 -0500 (Fri, 04 May 2007) | 3 lines
Store content signatures for Aliases so we can really use them
as dependencies.
........
r1896 | stevenknight | 2007-05-07 16:37:06 -0500 (Mon, 07 May 2007) | 3 lines
Move the TargetSignatures.py test into a subdirectory, which we can
populate with other tests (of the new settings, etc.).
........
r1897 | stevenknight | 2007-05-07 17:36:04 -0500 (Mon, 07 May 2007) | 3 lines
Fix use of TargetSignatures('content') to override settings of
SourceSignatures('timestamp').
........
r1900 | stevenknight | 2007-05-14 18:48:39 -0500 (Mon, 14 May 2007) | 5 lines
More specific sconsign signature tests: check for an actual 32-digit-long
hex string when an MD5 checksum is expected, so we can correctly detect
test failures if "None" shows up instead. (This will be used for getting
avoiding calculating the content signature in 'timestamp' mode.)
........
r1949 | stevenknight | 2007-05-30 22:45:25 -0500 (Wed, 30 May 2007) | 7 lines
Refactor the get_{csig,timestamp,size}() methods so they use the Memoizer
pattern, instead of the the NodeInfo object, to cache the results.
This won't be the final form (we have to reach and stuff a csig value
in the cache slot to make --implicit-cache work) but it's a step that
should make it easier to work on other in-progress modifications.
........
r1957 | stevenknight | 2007-06-01 10:18:15 -0500 (Fri, 01 Jun 2007) | 14 lines
Shift updating a Node's signature/timestamp/size information until after
the Node has been built (or visited, if it's a leaf Node). This therefore
changes the semantics of the .visited() method, so that it's called for
*every* Node visited, not just leaf Nodes.
This adds a Taskmaster method for updating a Node's status without
calling the .built() and .visited() methods, which is used by the
CleanTask subclass to avoid writing .sconsign info when cleaning (-c).
Interesting ripple effect in the SConf.py module: we have to manually
override the .store_info() method on the Nodes we create, so that they
don't write out their normal .sconsign information. The SConf subsystem
writes out its own by hand.
........
r1962 | stevenknight | 2007-06-01 13:12:24 -0500 (Fri, 01 Jun 2007) | 5 lines
Fix the ability to catch {IO,OS}Errors during build preparation and
display the actual Node that triggered the error. (Which Node actually
triggers this case, and therefore the displayed error message, changed
after we delayed content evaluation until *after* a Node is built.)
........
r1975 | stevenknight | 2007-06-04 15:21:09 -0500 (Mon, 04 Jun 2007) | 3 lines
Small refactoring: use {new,get}_ninfo() instead of instantiating
self.NodeInfo() directly.
........
r1976 | stevenknight | 2007-06-04 16:14:12 -0500 (Mon, 04 Jun 2007) | 5 lines
Similar to the previous change, call new_binfo() instead of instantiating
the BuildInfo() type directly. Change new_binfo() so it doesn't
automatically attach a NodeInfo object (because sometimes we want the
existing NodeInfo, and sometimes we want a completely new one).
........
r1977 | stevenknight | 2007-06-04 16:35:44 -0500 (Mon, 04 Jun 2007) | 3 lines
The body of DirFile.write() was all indented underneath a "if self.dirty:"
block. Invert the test and outdent the body.
........
r1978 | stevenknight | 2007-06-04 16:52:14 -0500 (Mon, 04 Jun 2007) | 2 lines
Memoize the stored information we get from the .sconsign file.
........
r1979 | stevenknight | 2007-06-05 14:17:32 -0500 (Tue, 05 Jun 2007) | 6 lines
Delay merging a Node's build information into the .sconsign entry until
the entry is being written out. This will allow us to include information
(such as a content signature) that gets fetched for the first time *after*
a Node is built. (The code here isn't the prettiest at the moment, but
this checkpoints an implementation that passes all the tests.)
........
r1980 | stevenknight | 2007-06-06 10:06:56 -0500 (Wed, 06 Jun 2007) | 4 lines
Refactor to have the store_info() method pull the binfo from
the Node (instead of having the Node passing it in).
Add a do_not_store_info() method to override the behavior.
........
r1981 | stevenknight | 2007-06-06 11:18:29 -0500 (Wed, 06 Jun 2007) | 3 lines
Refactor the unit test invocation so we can run multiple test
methods from each class.
........
r1982 | stevenknight | 2007-06-06 11:48:00 -0500 (Wed, 06 Jun 2007) | 7 lines
Move responsibility for merging buildinfo entries from Node.FS to the
SConsign module, through a new Base.store_info() method. Return the
existing Base.set_entry() method to actually setting the entry, and use
the new Base.store_info() method for things that need delayed storage
and merging. Adds some commented-out prototype code for delaying
the buildinfo entry merge until later.
........
r1983 | stevenknight | 2007-06-06 22:57:14 -0500 (Wed, 06 Jun 2007) | 4 lines
Better testing for parallel-build exceptions: re-order the file names
so that the input files are evaluated before the output files, increasing
the likelihood of all three being built by simultaneous threads.
........
r1984 | stevenknight | 2007-06-06 22:57:55 -0500 (Wed, 06 Jun 2007) | 2 lines
Have Trace() flush after write so interleaving stays correct.
........
r1985 | stevenknight | 2007-06-06 23:01:23 -0500 (Wed, 06 Jun 2007) | 11 lines
Pull the NodeInfo objects out of the BuildInfo object and just deal
with them as co-equal attributes of a Node.
.sconsign files now have a place-holder SConsignEntry class that holds a
Node's separate binfo and ninfo objects. This will also be more flexible
if we ever add more information in the future.
Since this changes the format (but not content) of the .sconsign file
entries, we will need to double back and figure out how to make the
transition from pre-BSR .sconsign files before this goes live.
........
r1986 | stevenknight | 2007-06-07 00:08:21 -0500 (Thu, 07 Jun 2007) | 2 lines
Store the computed csig in the NodeInfo structure.
........
r1987 | stevenknight | 2007-06-07 18:59:31 -0500 (Thu, 07 Jun 2007) | 12 lines
Track the content signature directly in the NodeInfo object, not in the
_memo dictionary.
Wipe out the existing NodeInfo after the Node has
been built so the content signature gets recalculated correctly.
Change the NodeInfoBase.update() method so that the default behavior is
to always update the specified fields, not just ones that exist. This
was necessary to get Alias and Value Nodes to work with this.
Remove left-over comments.
........
r1988 | stevenknight | 2007-06-08 17:09:24 -0500 (Fri, 08 Jun 2007) | 5 lines
Invert the logic (is_up_to_date => changed) when gathering Configure
Node states, and eliminate (most) Boolean logic that short-circuits
some of the tests. (This area isn't critical, so making sure the code
is readable is more important than pure performance.)
........
r1990 | stevenknight | 2007-06-09 17:14:32 -0500 (Sat, 09 Jun 2007) | 7 lines
Refactor the Node.changed() method so it always evaluates every
dependency, instead of returning early as soon as it finds a difference
(like a different number of old and new dependencies). This will be
necessary to delay evaluation of Node information (content signatures)
until it's actually needed (instead of evaluating it up front in case
it's needed later).
........
r2001 | stevenknight | 2007-06-11 11:20:29 -0500 (Mon, 11 Jun 2007) | 3 lines
Use SCons.compat in the refactored SConsign.py so use of True/False/etc.
works on Python 1.5 and 2.1.
........
r2002 | stevenknight | 2007-06-11 11:57:56 -0500 (Mon, 11 Jun 2007) | 3 lines
Have the "sconsign" script handle Configure context .sconsign information,
which comes from different structures that have no NodeInfo objects.
........
r2003 | stevenknight | 2007-06-11 15:08:52 -0500 (Mon, 11 Jun 2007) | 8 lines
Avoid generating content signatures unless they're actually used:
Don't automatically create a content signature after a File Node is
built/visited, and delay the .sconsign merge of the information until
just before the information will be written. This means we'll only write
out information that's actually fetched and used as part of deciding
whether or not to rebuild any targets.
........
r2004 | stevenknight | 2007-06-13 00:05:52 -0500 (Wed, 13 Jun 2007) | 4 lines
Fetch the {Source,Target}Signatures() values inside the
Node.FS.changed_since_last_build() method, not in Node.Node.changed().
This is slightly less efficient, but cleaner architecturally.
........
r2005 | stevenknight | 2007-06-13 10:18:17 -0500 (Wed, 13 Jun 2007) | 6 lines
Use zip() to collect the children and previous signatures info more
efficiently when deciding if a node must be rebuilt. (The comment in
the previous version of the source code about not using zip() because
of Python 1.5 name space problems was due to the way the SConfTests.py
module tried to reset the world's state.)
........
r2006 | stevenknight | 2007-06-13 10:59:16 -0500 (Wed, 13 Jun 2007) | 4 lines
Add a NotImplementedError base class implementation of
changed_since_last_build(), with doc string commentary about why the
method is called through the dependency Node, not the target Node.
........
r2019 | stevenknight | 2007-06-18 12:26:06 -0500 (Mon, 18 Jun 2007) | 4 lines
Remove the unnecessary cut-and-paste "import SCons.Action" in the
changed_since_last_build() method, which was a left-over cut and paste.
SCons.Action was already imported up top.
........
r2021 | stevenknight | 2007-06-18 18:31:09 -0500 (Mon, 18 Jun 2007) | 2 lines
Make sure all "is up to date" messages get printed when -j is used.
........
r2022 | stevenknight | 2007-06-19 16:26:22 -0500 (Tue, 19 Jun 2007) | 3 lines
Refactor the __checkClass() and must_be_a_Dir() methods into a more
general and more efficient must_be_same() method.
........
r2024 | stevenknight | 2007-06-19 19:16:09 -0500 (Tue, 19 Jun 2007) | 3 lines
More clean up: change various self.fs.Entry() calls to calls through
the bound directory.Entry() method.
........
r2026 | stevenknight | 2007-06-19 22:10:04 -0500 (Tue, 19 Jun 2007) | 10 lines
Refactor lookup of Node.FS nodes as follows:
* Completely get rid of _doLookup(), which was old, complicated code
that implemented a top-down, entry-by-entry search for path names.
* Use a new FS._lookup() method, which normalizes names to absolute
paths, with a new RootDir._lookup_abs() method, which handles the
actual lookup. We plan to use the latter method for fast(er) internal
lookups of already-normalized paths.
* Create a new separate FS.get_root() method, just to avoid clutter
in the RootDir._lookup_abs() method.
........
r2027 | stevenknight | 2007-06-20 13:29:19 -0500 (Wed, 20 Jun 2007) | 3 lines
Remove the debug-explain.py test from sigrefactor, branches/core
split this up into multiple sub-tests.
........
r2028 | stevenknight | 2007-06-20 13:38:28 -0500 (Wed, 20 Jun 2007) | 2 lines
Handle a line-ending mismatch on Windows.
........
r2029 | stevenknight | 2007-06-20 13:39:05 -0500 (Wed, 20 Jun 2007) | 2 lines
Support the ability to -d when path names have Windows \ separators.
........
r2030 | stevenknight | 2007-06-20 13:55:11 -0500 (Wed, 20 Jun 2007) | 2 lines
Windows portability: TestSConsign._obj instead of a hard-coded '.o' suffix.
........
r2031 | stevenknight | 2007-06-20 15:30:06 -0500 (Wed, 20 Jun 2007) | 2 lines
Python 1.5 namespace portability.
........
r2032 | stevenknight | 2007-06-20 16:38:28 -0500 (Wed, 20 Jun 2007) | 2 lines
Fix tests affected by the -d fix for runtest.py.
........
r2039 | stevenknight | 2007-06-21 12:17:06 -0500 (Thu, 21 Jun 2007) | 2 lines
Ignore *.pyo files, too, now that one of the tests causes us to generate them.
........
r2041 | stevenknight | 2007-06-21 12:36:30 -0500 (Thu, 21 Jun 2007) | 7 lines
In the .sconsign file, store the paths to all of the dependencies relative
to the top-level SConstruct directory, not the target's directory.
This allows us to be much more efficient when writing the .sconsign file
(since we can just store the already-computed path) and reading the
.sconsign file (since we can use the quick, normalized-lookup method to
translate the string into the correct Node).
........
r2042 | stevenknight | 2007-06-22 13:25:26 -0500 (Fri, 22 Jun 2007) | 2 lines
Remove left-over debug code (instrumenting os.path.normpath()).
........
r2043 | stevenknight | 2007-06-22 13:28:35 -0500 (Fri, 22 Jun 2007) | 8 lines
Actually use the new Dir._lookup_abs() method for fast .sconsign lookups
relative to the top-level SConstruct directory and make it work correctly.
(I had commented out the call to binfo.prepare_dependencies() and
forgot to un-comment it.) This simplifies things by getting rid of the
target node we attached to a BuildInfo (purely so we could do relative
path lookups) and the corresponding conversions into and out of the
.sconsign file.
........
r2044 | stevenknight | 2007-06-22 19:31:00 -0500 (Fri, 22 Jun 2007) | 5 lines
Delay fetching the {Source,Target}Signature() values from an environment
until they're actually needed. (If the performance here is "close
enough", then we may not have to look at more complicated loop-invariant
schemes.)
........
r2045 | stevenknight | 2007-06-22 22:06:19 -0500 (Fri, 22 Jun 2007) | 4 lines
Replace the Environment.our_deepcopy() function with a
Util.semi_deepcopy() function, rewritten to use the more efficient
dispatch-table approach that the normal Python copy.py module uses.
........
r2050 | stevenknight | 2007-06-24 23:49:06 -0500 (Sun, 24 Jun 2007) | 5 lines
Make the latest signature refactoring portable to Windows, and between
platforms, by always storing the dependency paths with POSIX separators.
Handle multiple drive letters by tracking a separate "labspath" attribute
for the lookup path relative to the root directory's drive letter.
........
r2065 | stevenknight | 2007-06-27 11:02:48 -0500 (Wed, 27 Jun 2007) | 2 lines
Branch files that were left out of the last merge.
........
r2066 | stevenknight | 2007-06-27 11:03:05 -0500 (Wed, 27 Jun 2007) | 2 lines
Branch files that were left out of the last merge.
........
r2067 | stevenknight | 2007-06-27 11:03:24 -0500 (Wed, 27 Jun 2007) | 2 lines
Branch files that were left out of the last merge.
........
r2069 | stevenknight | 2007-06-27 19:08:50 -0500 (Wed, 27 Jun 2007) | 2 lines
Remove an unnecessary target.has_builder() call in changed_since_last_build().
........
r2070 | stevenknight | 2007-06-27 19:25:47 -0500 (Wed, 27 Jun 2007) | 3 lines
Refactor changed_since_last_build() to avoid code duplication and simplify
(?) the structure--probably negligible performance impact.
........
r2071 | stevenknight | 2007-06-29 16:21:37 -0500 (Fri, 29 Jun 2007) | 3 lines
Add the Decider() function, globally, and settable per construction
environment or per Node. This hasn't yet been optimized.
........
r2072 | stevenknight | 2007-06-29 18:01:53 -0500 (Fri, 29 Jun 2007) | 2 lines
Memoize Executor.get_build_env().
........
r2073 | stevenknight | 2007-06-30 01:04:43 -0500 (Sat, 30 Jun 2007) | 5 lines
Constructing a dictionary to map signature types ('build', 'MD5', etc.)
to a dependency's methods ends up being less efficient than just using
a series of if-elif statements. Refactor the default_decider_function()
accordingly.
........
r2074 | stevenknight | 2007-07-01 08:59:35 -0500 (Sun, 01 Jul 2007) | 4 lines
Now that the objects returned by Node.FS.get_stored_implicit() already
convert the .sconsign strings to Nodes, don't have the --implicit-cache
loop run them through the source_factory method.
........
r2075 | stevenknight | 2007-07-02 09:46:21 -0500 (Mon, 02 Jul 2007) | 4 lines
Eliminate an if-test for the intialized default construction environment
for internal calls of SCons.Defaults.DefaultEnvironment(), and some
external calls as well.
........
r2076 | stevenknight | 2007-07-02 10:17:58 -0500 (Mon, 02 Jul 2007) | 3 lines
More efficient use of the changed_since_last_build() method from
the default construction environment.
........
r2077 | stevenknight | 2007-07-02 12:26:34 -0500 (Mon, 02 Jul 2007) | 6 lines
Move the decision about whether a given dependent Node has a Builder
or not back to when the Builder is actually set, by splitting the
decision path in two; one for sources and one for targets. (Setting a
construction environment's decider sets both of them, which may or may
not be what we want in the long term.))
........
r2079 | stevenknight | 2007-07-02 13:14:40 -0500 (Mon, 02 Jul 2007) | 4 lines
Now explicity set the source decider function for a construction
environment when the SourceSignatures() method is called, avoiding
the delayed-evaluation tests.
........
r2081 | stevenknight | 2007-07-02 13:55:58 -0500 (Mon, 02 Jul 2007) | 6 lines
And now explicity set the target decider function for a construction
environment when the TargetSignatures() method is called, avoiding even
more delayed-evaluation tests.
This also renames the cslb_*() functions to decide_*().
........
r2084 | stevenknight | 2007-07-03 07:13:38 -0500 (Tue, 03 Jul 2007) | 3 lines
Memoize the Node.get_build_env() value, too, since we're now callng
that repeatedly.
........
r2085 | stevenknight | 2007-07-03 09:50:25 -0500 (Tue, 03 Jul 2007) | 5 lines
Get rid of the str_to_nodes() function within the
FileBuildInfo.prepare_dependencies() method that checks for Alias lookups,
in favor of a loop that uses new str_to_nodes() methods in the individual
*NodeInfo classes to convert the strings to Nodes more directly.
........
r2086 | stevenknight | 2007-07-03 10:07:04 -0500 (Tue, 03 Jul 2007) | 3 lines
Format the FileBuildInfo entries by zipping the children and their
NodeInfo signatures, not by fetching each one by index.
........
r2089 | stevenknight | 2007-07-03 20:20:15 -0500 (Tue, 03 Jul 2007) | 4 lines
Remove the _add_child() checks that verify that the argument is a list
and contains all Nodes. It's internal, so any necessary verification
can happen in the methods that call it.
........
r2101 | stevenknight | 2007-07-08 16:13:56 -0500 (Sun, 08 Jul 2007) | 3 lines
Add string aliases for the canned Decider functions we'll support out of the
box: timestamp-match (a.k.a. make), timestamp-newer, MD5 (a.k.a. content).
........
r2115 | stevenknight | 2007-07-10 21:00:14 -0500 (Tue, 10 Jul 2007) | 2 lines
Refactor test/option-q.py into two separate subtests.
........
r2144 | stevenknight | 2007-07-16 01:04:25 -0500 (Mon, 16 Jul 2007) | 3 lines
Fix -q exiting with a non-zero exit status when a file with no builder
is specified on the command line.
........
r2314 | stevenknight | 2007-08-17 13:46:27 -0500 (Fri, 17 Aug 2007) | 3 lines
Capture a test case to make sure get_csig() can be called from
within a function Action. (Damyan Pepper)
........
r2315 | stevenknight | 2007-08-17 15:05:54 -0500 (Fri, 17 Aug 2007) | 3 lines
Uncomment the test for content signatures with directories as sources,
which the signature refactoring makes work.
........
r2316 | stevenknight | 2007-08-17 16:33:25 -0500 (Fri, 17 Aug 2007) | 2 lines
Make sure that CacheDir() works even when timestamp signatures are used.
........
r2317 | stevenknight | 2007-08-17 16:51:02 -0500 (Fri, 17 Aug 2007) | 2 lines
Change line endings from DOS to UNIX.
........
r2318 | stevenknight | 2007-08-18 07:13:09 -0500 (Sat, 18 Aug 2007) | 4 lines
Add a new MD5-timestamp decider function that assumes that if the
timestamp hasn't changed, then the content hasn't changed either, and
it should then just re-use the content signature from the last run.
........
r2319 | stevenknight | 2007-08-18 07:41:59 -0500 (Sat, 18 Aug 2007) | 3 lines
Add a test script to verify that action changes cause rebuilds even
when file decisions are configured for timestamps.
........
r2360 | stevenknight | 2007-08-23 08:07:16 -0500 (Thu, 23 Aug 2007) | 2 lines
Fix the test by resetting the content3.in file to the correct timestamp.
........
r2361 | stevenknight | 2007-08-23 12:02:04 -0500 (Thu, 23 Aug 2007) | 3 lines
Use the new hashlib module for our MD5 signature calculations, with the
introduction of a compatibility module for pre-2.5 Python versions.
........
r2362 | stevenknight | 2007-08-23 14:02:56 -0500 (Thu, 23 Aug 2007) | 2 lines
Make targets implicitly depend on the commands used to build them.
........
r2364 | stevenknight | 2007-08-24 00:07:46 -0500 (Fri, 24 Aug 2007) | 2 lines
Add code to convert what we can from the old .sconsign file entries.
........
r2366 | stevenknight | 2007-08-24 19:40:09 -0500 (Fri, 24 Aug 2007) | 2 lines
Add a $IMPLICIT_COMMAND_DEPENDENCIES variable.
........
r2367 | stevenknight | 2007-08-24 19:43:58 -0500 (Fri, 24 Aug 2007) | 2 lines
Add mention of $IMPLICIT_COMMAND_DEPENDENCIES to the release notes.
........
r2375 | stevenknight | 2007-08-27 11:43:04 -0500 (Mon, 27 Aug 2007) | 2 lines
Windows portability fixes in tests.
........
r2376 | stevenknight | 2007-08-27 12:19:52 -0500 (Mon, 27 Aug 2007) | 2 lines
Use string.join() instead of a more modern string object method.
........
r2379 | stevenknight | 2007-08-27 12:40:02 -0500 (Mon, 27 Aug 2007) | 3 lines
Add version ID's to the classes stored in a .sconsign file, for easier
future conversion as we change schemas.
........
................
r2390 | stevenknight | 2007-08-27 17:32:00 -0500 (Mon, 27 Aug 2007) | 4 lines
Remove the old TargetSignatures.py test, which was
overlooked in the merge. (Its functionality got moved to
test/TargetSignatures/build-content.py.)
................
r2391 | stevenknight | 2007-08-28 10:47:15 -0500 (Tue, 28 Aug 2007) | 2 lines
Don't die if an old .sconsign entry has no .ninfo attribute.
................
r2392 | stevenknight | 2007-08-28 15:52:56 -0500 (Tue, 28 Aug 2007) | 2 lines
Fix an O(N^2) search in Tool.install.add_targets_to_INSTALLED_FILES().
................
r2393 | stevenknight | 2007-08-28 21:28:30 -0500 (Tue, 28 Aug 2007) | 2 lines
Add support for an ensure_suffix Builder keyword argument.
................
r2394 | stevenknight | 2007-08-28 21:37:04 -0500 (Tue, 28 Aug 2007) | 3 lines
Separate the failure tests when trying to connect to an X server,
so we can tell why it's failing...
................
r2395 | stevenknight | 2007-08-29 11:46:34 -0500 (Wed, 29 Aug 2007) | 3 lines
Look for a raw exit status of 1, or the shifted exit status, so we're
not dependent on how the test infrastructure hands it to us.
................
r2396 | stevenknight | 2007-08-29 18:12:39 -0500 (Wed, 29 Aug 2007) | 2 lines
Disable universal_newlines when calling subprocess.
................
r2397 | stevenknight | 2007-08-29 18:13:29 -0500 (Wed, 29 Aug 2007) | 3 lines
Add a Progress() hook to support display of per-Node progress while
walking the DAG.
................
r2398 | stevenknight | 2007-08-29 19:43:12 -0500 (Wed, 29 Aug 2007) | 7 lines
Add a test case (courtesy Greg Noel) to characterize how we handle
mid-build changes to implicit dependency files (we currently don't detect
the change and rebuild all dependents).
Move two other test scripts related to implicit dependency behavior
into a common subdirectory.
................
r2399 | stevenknight | 2007-08-30 10:32:50 -0500 (Thu, 30 Aug 2007) | 4 lines
Add a $JAVABOOTCLASSPATH variable.
Commonize $JAVA*PATH expansion with a utility class.
(Greg Ward)
................
r2400 | stevenknight | 2007-08-30 10:59:56 -0500 (Thu, 30 Aug 2007) | 3 lines
Make sure extra auxiliary files generated by LaTeX packages are deleted
by scons -c. (Matthias Troffaes)
................
r2401 | stevenknight | 2007-08-30 18:43:24 -0500 (Thu, 30 Aug 2007) | 3 lines
Issue 1589: when Cloning a construction environment, apply set variables
both before and after calling tools (like during intialization).
................
r2402 | stevenknight | 2007-08-30 18:52:02 -0500 (Thu, 30 Aug 2007) | 3 lines
Issue 1555: add Windows64 support for the Intel C compiler.
(Gary Oberbrunner)
................
r2403 | stevenknight | 2007-08-30 19:57:24 -0500 (Thu, 30 Aug 2007) | 4 lines
Issue 1493: have ParseConfig(), MergeFlags() and ParseFlags() handle
output from *-config commands with quoted arguments (path names that
contain spaces).
................
r2410 | stevenknight | 2007-09-04 00:28:46 -0500 (Tue, 04 Sep 2007) | 4 lines
Add a shlex.split() compatability wrapper for pre-Python 2.3 versions.
Unfortunately, it doesn't behave exactly like later versions (it doesn't
detect mid-token quotes), but we're not going to worry about that.
................
r2411 | stevenknight | 2007-09-04 12:56:36 -0500 (Tue, 04 Sep 2007) | 3 lines
Add a Sig.py module that generates a warning if user code tries to
"import SCons.Sig" directly.
................
r2412 | stevenknight | 2007-09-04 20:53:09 -0500 (Tue, 04 Sep 2007) | 3 lines
Make the Return() function return immediately. Add a stop= keyword
argument in case anyone really needs the old behavior.
................
r2413 | stevenknight | 2007-09-05 10:38:40 -0500 (Wed, 05 Sep 2007) | 4 lines
Refactor initialization of different pieces (the underlying
InstallBuilder, the Install() and InstallAs() wrappers) into separate
blocks.
................
r2414 | stevenknight | 2007-09-05 11:39:25 -0500 (Wed, 05 Sep 2007) | 4 lines
Move the definitions of the wrapper functions to the global namespace
(nesting them and delaying intialization of global variables doesn't
save anything and just complicates reading the code).
................
r2415 | stevenknight | 2007-09-05 13:31:27 -0500 (Wed, 05 Sep 2007) | 3 lines
When Cloning a construction environment, re-bind any methods added by
the AddMethod() method to the new construction environment.
................
r2416 | stevenknight | 2007-09-06 10:04:18 -0500 (Thu, 06 Sep 2007) | 7 lines
Add a MethodWrapper class that handles generic association of a
callable and an object that wants to use the callable as a "method."
Provide hooks for centralized cloning of the different "methods" on to
a new, copied underlying environment (used by the env.Clone() method).
Make BuilderWrapper a subclass of MethodWrapper. Modify env.Clone()
for the new, combined way of handling these "methods."
................
r2417 | stevenknight | 2007-09-06 14:11:35 -0500 (Thu, 06 Sep 2007) | 2 lines
Refactor test failure conditions to use test.must_match().
................
r2418 | stevenknight | 2007-09-06 14:28:52 -0500 (Thu, 06 Sep 2007) | 9 lines
Restore the ability to use the Install() and InstallAs() builders without
having to specify the 'install' tool directly.
Add a ToolInitializer class that handles delayed intialization of tool
modules when added to a construction environment as a "method."
Add an env.RemoveModule() method that takes care of removing an added
MethodWrapper instance from the environment, deleting it from the
list that gets copied on cloning (and changing the name of that list
back to "added_methods").
................
r2419 | stevenknight | 2007-09-07 06:27:33 -0500 (Fri, 07 Sep 2007) | 3 lines
Issue 1725: avoid race conditions when pushing a file to CacheDir().
(Carsten Koch)
................
r2420 | pscholl | 2007-09-07 09:13:28 -0500 (Fri, 07 Sep 2007) | 7 lines
* delete the push_emitter() function. Direct manipulation of the source and
target lists seem the way to go for packaging. Fix all packagers using this
function.
* fix the internationalization testcase
................
r2421 | pscholl | 2007-09-07 09:14:12 -0500 (Fri, 07 Sep 2007) | 3 lines
* new testcase for building multiple rpm packages from one scons call.
................
r2422 | stevenknight | 2007-09-07 14:12:22 -0500 (Fri, 07 Sep 2007) | 4 lines
Capture a test script with a particular reported test case for multiple
Package() calls spread across different SConscript subdirectories.
(Use case courtesy Andrew Smith.)
................
r2423 | stevenknight | 2007-09-07 17:40:25 -0500 (Fri, 07 Sep 2007) | 4 lines
Fix use of exitstatfunc on Action objects by getting rid of the ability
to override exitstatfunc when calling a function (which we were only using
internally to basically suppress whatever was set on the Action object).
................
r2424 | stevenknight | 2007-09-10 20:06:59 -0500 (Mon, 10 Sep 2007) | 3 lines
Make sure the library dependencies show up in --tree=derived output.
(The Microsoft toolchain used to fail this on SCons 0.97 and earlier.)
................
r2425 | stevenknight | 2007-09-10 22:25:27 -0500 (Mon, 10 Sep 2007) | 5 lines
When adding a new entry to a directory, reset the .implicit attribute to
None so that the directory will get "re-scanned" for implicit dependencies
(the entries within the directory) if it gets re-visited later in the
DAG walk.
................
r2426 | stevenknight | 2007-09-12 12:38:05 -0500 (Wed, 12 Sep 2007) | 2 lines
String method fix for Python 1.5.2.
................
r2427 | stevenknight | 2007-09-12 12:38:29 -0500 (Wed, 12 Sep 2007) | 2 lines
Remove left-over commented-out Trace() call.
................
r2428 | stevenknight | 2007-09-13 01:36:34 -0500 (Thu, 13 Sep 2007) | 2 lines
Handle Java inner class names with $ in them. (Tzvetan Mikov)
................
r2429 | stevenknight | 2007-09-13 14:29:09 -0500 (Thu, 13 Sep 2007) | 2 lines
Add a check that should (possibly) avoid import errors on Solaris.
................
r2430 | stevenknight | 2007-09-13 14:30:36 -0500 (Thu, 13 Sep 2007) | 5 lines
Windows portability fixes for Progress() tests, restoring the original
universal_newlines setting on the test infrastructure, changing file
names to make the order in which Nodes are visited case-insensitive, and
fixing line endings on output we expect to read from files.
................
r2431 | stevenknight | 2007-09-14 07:58:44 -0500 (Fri, 14 Sep 2007) | 2 lines
Make sure all function examples are introduced by "Example:" or "Examples:".
................
r2432 | stevenknight | 2007-09-14 11:35:03 -0500 (Fri, 14 Sep 2007) | 3 lines
Clean up Options files initialization so we don't have to check on usage
for whether it's been set.
................
r2433 | stevenknight | 2007-09-14 23:12:32 -0500 (Fri, 14 Sep 2007) | 4 lines
Fix use of {Source,Target}Signatures() with construction variable
overrides by making the default decider_*() things unbound functions,
instead of bound methods.
................
r2434 | stevenknight | 2007-09-14 23:14:25 -0500 (Fri, 14 Sep 2007) | 2 lines
Don't use "True" in the test infrastructure, use 1 instead.
................
r2435 | stevenknight | 2007-09-15 00:52:56 -0500 (Sat, 15 Sep 2007) | 2 lines
Avoid use of False in two tests.
................
r2436 | pscholl | 2007-09-17 08:14:35 -0500 (Mon, 17 Sep 2007) | 21 lines
* additional testcase for packaging:
* building multiple packages from one scons call
* supplying explicit target names for msi and rpm
* test the Find*Files() functions
* modify the package-type option test to not only catch
the PACKAGETYPE, but also the --package-type alias
command line argument
* move FindInstalledFiles to Environment.py and remove
Find*Files() functions from the packaging module.
* capitalize the VENDOR tag for msi
* remove the superfluous packager.py files
* Add documentation for FindInstalledFiles() and
FindSourceFiles()
................
r2437 | stevenknight | 2007-09-17 11:06:28 -0500 (Mon, 17 Sep 2007) | 2 lines
Provide compatibility for the variables defined in SCons.Sig.
................
r2438 | stevenknight | 2007-09-17 23:13:27 -0500 (Mon, 17 Sep 2007) | 5 lines
Handle duplicate fies in a target list: only decrement the reference
count once for the target, not once for each time it shows up in the
list, so dependencies don't "disappear" from the DAG walk because the
reference count gets decremented past zero.
................
r2439 | stevenknight | 2007-09-17 23:15:45 -0500 (Mon, 17 Sep 2007) | 2 lines
Fix syntax errors from failure to quote the package type.
................
r2440 | stevenknight | 2007-09-17 23:21:20 -0500 (Mon, 17 Sep 2007) | 2 lines
Use AddOption() to support the --package-type option.
................
r2441 | stevenknight | 2007-09-17 23:31:40 -0500 (Mon, 17 Sep 2007) | 2 lines
Skip the test if tar isn't available.
................
r2442 | stevenknight | 2007-09-18 06:35:04 -0500 (Tue, 18 Sep 2007) | 2 lines
Remove Tool/packaging/packager.py from the manifest.
................
r2443 | garyo | 2007-09-18 08:13:15 -0500 (Tue, 18 Sep 2007) | 1 line
SGI IRIX: use proper C++ compiler for SHCXX, don't hardcode CC.
................
r2444 | garyo | 2007-09-18 08:17:03 -0500 (Tue, 18 Sep 2007) | 1 line
Avoid running MinGW tests on IRIX; they're irrelevant there and they fail anyway (they seem to pass on Linux, so left them in there.)
................
r2445 | garyo | 2007-09-18 08:20:52 -0500 (Tue, 18 Sep 2007) | 1 line
Fixed wrong return type in test code; caused warnings on IRIX which made test fail.
................
r2446 | stevenknight | 2007-09-18 11:21:53 -0500 (Tue, 18 Sep 2007) | 2 lines
Record Gary's change to the sgic++ tool.
................
r2451 | stevenknight | 2007-09-19 00:01:46 -0500 (Wed, 19 Sep 2007) | 2 lines
Windows portability fix (when checking for path names in output).
................
r2452 | stevenknight | 2007-09-19 00:08:23 -0500 (Wed, 19 Sep 2007) | 2 lines
Handle the ImportError if there's no threading module on Windows.
................
Diffstat (limited to 'src/engine')
70 files changed, 2984 insertions, 2276 deletions
diff --git a/src/engine/MANIFEST.in b/src/engine/MANIFEST.in index 9eccbdb..52323d2 100644 --- a/src/engine/MANIFEST.in +++ b/src/engine/MANIFEST.in @@ -2,6 +2,7 @@ SCons/__init__.py SCons/Action.py SCons/Builder.py SCons/compat/__init__.py +SCons/compat/_scons_hashlib.py SCons/compat/_scons_optparse.py SCons/compat/_scons_sets.py SCons/compat/_scons_sets15.py @@ -56,9 +57,7 @@ SCons/Script/__init__.py SCons/Script/Main.py SCons/Script/SConscript.py SCons/Script/SConsOptions.py -SCons/Sig/__init__.py -SCons/Sig/MD5.py -SCons/Sig/TimeStamp.py +SCons/Sig.py SCons/Subst.py SCons/Taskmaster.py SCons/Tool/__init__.py @@ -125,7 +124,6 @@ SCons/Tool/nasm.py SCons/Tool/packaging/__init__.py SCons/Tool/packaging/ipk.py SCons/Tool/packaging/msi.py -SCons/Tool/packaging/packager.py SCons/Tool/packaging/rpm.py SCons/Tool/packaging/src_tarbz2.py SCons/Tool/packaging/src_targz.py diff --git a/src/engine/SCons/Action.py b/src/engine/SCons/Action.py index 04e68a3..bdedc99 100644 --- a/src/engine/SCons/Action.py +++ b/src/engine/SCons/Action.py @@ -31,8 +31,8 @@ other modules: get_contents() Fetches the "contents" of an Action for signature calculation. - This is what the Sig/*.py subsystem uses to decide if a target - needs to be rebuilt because its action changed. + This is what gets MD5 checksumm'ed to decide if a target needs + to be rebuilt because its action changed. genstring() Returns a string representation of the Action *without* @@ -495,6 +495,22 @@ class CommandAction(_ActionAction): cmd = str(cmd) return env.subst_target_source(cmd, SUBST_SIG, target, source) + def get_implicit_deps(self, target, source, env): + icd = env.get('IMPLICIT_COMMAND_DEPENDENCIES', True) + if SCons.Util.is_String(icd) and icd[:1] == '$': + icd = env.subst(icd) + if not icd or icd in ('0', 'None'): + return [] + from SCons.Subst import SUBST_SIG + cmd_list = env.subst_list(self.cmd_list, SUBST_SIG, target, source) + res = [] + for cmd_line in cmd_list: + if cmd_line: + d = env.WhereIs(str(cmd_line[0])) + if d: + res.append(env.fs.File(d)) + return res + class CommandGeneratorAction(ActionBase): """Class for command-generator actions.""" def __init__(self, generator, *args, **kw): @@ -542,6 +558,9 @@ class CommandGeneratorAction(ActionBase): """ return self._generate(target, source, env, 1).get_contents(target, source, env) + def get_implicit_deps(self, target, source, env): + return self._generate(target, source, env, 1).get_implicit_deps(target, source, env) + # A LazyAction is a kind of hybrid generator and command action for @@ -717,6 +736,9 @@ class FunctionAction(_ActionAction): return contents + env.subst(string.join(map(lambda v: '${'+v+'}', self.varlist))) + def get_implicit_deps(self, target, source, env): + return [] + class ListAction(ActionBase): """Class for lists of other actions.""" def __init__(self, list): @@ -760,6 +782,12 @@ class ListAction(ActionBase): return stat return 0 + def get_implicit_deps(self, target, source, env): + result = [] + for act in self.list: + result.extend(act.get_implicit_deps(target, source, env)) + return result + class ActionCaller: """A class for delaying calling an Action function with specific (positional and keyword) arguments until the Action is actually diff --git a/src/engine/SCons/Action.xml b/src/engine/SCons/Action.xml index da2f3e3..a82b038 100644 --- a/src/engine/SCons/Action.xml +++ b/src/engine/SCons/Action.xml @@ -4,6 +4,41 @@ __COPYRIGHT__ This file is processed by the bin/SConsDoc.py module. See its __doc__ string for a discussion of the format. --> +<cvar name="IMPLICIT_COMMAND_DEPENDENCIES"> +<summary> +Controls whether or not SCons will +add implicit dependencies for the commands +executed to build targets. + +By default, SCons will add +to each target +an implicit dependency on the command +represented by the first argument on any +command line it executes. +The specific file for the dependency is +found by searching the +<varname>PATH</varname> +variable in the +<varname>ENV</varname> +environment used to execute the command. + +If the construction variable +&cv-IMPLICIT_COMMAND_DEPENDENCIES; +is set to a false value +(<literal>None</literal>, +<literal>False</literal>, +<literal>0</literal>, +etc.), +then the implicit dependency will +not be added to the targets +built with that construction environment. + +<example> +env = Environment(IMPLICIT_COMMAND_DEPENDENCIES = 0) +</example> +</summary> +</cvar> + <cvar name="PRINT_CMD_LINE_FUNC"> <summary> A Python function used to print the command lines as they are executed diff --git a/src/engine/SCons/Builder.py b/src/engine/SCons/Builder.py index ff31c5e..6164a55 100644 --- a/src/engine/SCons/Builder.py +++ b/src/engine/SCons/Builder.py @@ -39,9 +39,6 @@ used by other modules: variable. This also takes care of warning about possible mistakes in keyword arguments. - targets() - Returns the list of targets for a specific builder instance. - add_emitter() Adds an emitter for a specific file suffix, used by some Tool modules to specify that (for example) a yacc invocation on a .y @@ -367,6 +364,7 @@ class BuilderBase: chdir = _null, is_explicit = 1, src_builder = [], + ensure_suffix = False, **overrides): if __debug__: logInstanceCreation(self, 'Builder.BuilderBase') self._memo = {} @@ -394,6 +392,7 @@ class BuilderBase: self.set_suffix(suffix) self.set_src_suffix(src_suffix) + self.ensure_suffix = ensure_suffix self.target_factory = target_factory self.source_factory = source_factory @@ -467,7 +466,7 @@ class BuilderBase: executor.add_sources(slist) return executor - def _adjustixes(self, files, pre, suf): + def _adjustixes(self, files, pre, suf, ensure_suffix=False): if not files: return [] result = [] @@ -476,7 +475,7 @@ class BuilderBase: for f in files: if SCons.Util.is_String(f): - f = SCons.Util.adjustixes(f, pre, suf) + f = SCons.Util.adjustixes(f, pre, suf, ensure_suffix) result.append(f) return result @@ -505,7 +504,7 @@ class BuilderBase: splitext = lambda S,self=self,env=env: self.splitext(S,env) tlist = [ t_from_s(pre, suf, splitext) ] else: - target = self._adjustixes(target, pre, suf) + target = self._adjustixes(target, pre, suf, self.ensure_suffix) tlist = env.arg2nodes(target, target_factory) if self.emitter: @@ -652,13 +651,6 @@ class BuilderBase: return '' return ret[0] - def targets(self, node): - """Return the list of targets for this builder instance. - - For most normal builders, this is just the supplied node. - """ - return [ node ] - def add_emitter(self, suffix, emitter): """Add a suffix-emitter mapping to this Builder. @@ -669,16 +661,6 @@ class BuilderBase: """ self.emitter[suffix] = emitter - def push_emitter(self, emitter): - """Add a emitter to the beginning of the emitter list of this Builder. - - This creates an empty list if the emitter is None. - """ - if not self.emitter: - self.emitter = ListEmitter( [emitter] ) - else: - self.emitter.insert(0, emitter) - def add_src_builder(self, builder): """ Add a new Builder to the list of src_builders. diff --git a/src/engine/SCons/CacheDir.py b/src/engine/SCons/CacheDir.py index 6a1cc04..e3730a4 100644 --- a/src/engine/SCons/CacheDir.py +++ b/src/engine/SCons/CacheDir.py @@ -89,10 +89,19 @@ def CachePushFunc(target, source, env): cd.CacheDebug('CachePush(%s): pushing to %s\n', t, cachefile) + tempfile = cachefile+'.tmp'+str(os.getpid()) + errfmt = "Unable to copy %s to cache. Cache file is %s" + if not fs.isdir(cachedir): - fs.makedirs(cachedir) + try: + fs.makedirs(cachedir) + except EnvironmentError: + # We may have received an exception because another process + # has beaten us creating the directory. + if not fs.isdir(cachedir): + msg = errfmt % (str(target), cachefile) + raise SCons.Errors.EnvironmentError, msg - tempfile = cachefile+'.tmp' try: if fs.islink(t.path): fs.symlink(fs.readlink(t.path), tempfile) @@ -101,15 +110,14 @@ def CachePushFunc(target, source, env): fs.rename(tempfile, cachefile) st = fs.stat(t.path) fs.chmod(cachefile, stat.S_IMODE(st[stat.ST_MODE]) | stat.S_IWRITE) - except (IOError, OSError): + except EnvironmentError: # It's possible someone else tried writing the file at the # same time we did, or else that there was some problem like # the CacheDir being on a separate file system that's full. # In any case, inability to push a file to cache doesn't affect # the correctness of the build, so just print a warning. - SCons.Warnings.warn(SCons.Warnings.CacheWriteErrorWarning, - "Unable to copy %s to cache. Cache file is %s" - % (str(target), cachefile)) + msg = errfmt % (str(target), cachefile) + SCons.Warnings.warn(SCons.Warnings.CacheWriteErrorWarning, msg) CachePush = SCons.Action.Action(CachePushFunc, None) @@ -117,9 +125,9 @@ class CacheDir: def __init__(self, path): try: - import SCons.Sig.MD5 + import hashlib except ImportError: - msg = "No MD5 module available, CacheDir() not supported" + msg = "No hashlib or MD5 module available, CacheDir() not supported" SCons.Warnings.warn(SCons.Warnings.NoMD5ModuleWarning, msg) else: self.path = path @@ -203,5 +211,7 @@ class CacheDir: class Null(SCons.Util.Null): def repr(self): return 'CacheDir.Null()' + def cachepath(self, node): + return None, None def retrieve(self, node): return False diff --git a/src/engine/SCons/CacheDirTests.py b/src/engine/SCons/CacheDirTests.py index 564b762..e31f67b 100644 --- a/src/engine/SCons/CacheDirTests.py +++ b/src/engine/SCons/CacheDirTests.py @@ -35,17 +35,23 @@ import SCons.CacheDir built_it = None class Action: - def __call__(self, targets, sources, env, errfunc, **kw): + def __call__(self, targets, sources, env, **kw): global built_it if kw.get('execute', 1): built_it = 1 return 0 + def genstring(self, target, source, env): + return str(self) + def get_contents(self, target, source, env): + return '' class Builder: def __init__(self, environment, action): self.env = environment self.action = action self.overrides = {} + self.source_scanner = None + self.target_scanner = None class Environment: def __init__(self, cachedir): @@ -71,8 +77,9 @@ class BaseTestCase(unittest.TestCase): node = self.fs.File(name) node.builder_set(Builder(Environment(self._CacheDir), action)) if bsig: - node.binfo = node.BuildInfo(node) - node.binfo.ninfo.bsig = bsig + node.cachesig = bsig + #node.binfo = node.BuildInfo(node) + #node.binfo.ninfo.bsig = bsig return node class CacheDirTestCase(BaseTestCase): @@ -86,8 +93,8 @@ class CacheDirTestCase(BaseTestCase): # of the file in cache. def my_collect(list): return list[0] - save_collect = SCons.Sig.MD5.collect - SCons.Sig.MD5.collect = my_collect + save_collect = SCons.Util.MD5collect + SCons.Util.MD5collect = my_collect try: f5 = self.File("cd.f5", 'a_fake_bsig') @@ -96,7 +103,7 @@ class CacheDirTestCase(BaseTestCase): filename = os.path.join(dirname, 'a_fake_bsig') assert result == (dirname, filename), result finally: - SCons.Sig.MD5.collect = save_collect + SCons.Util.MD5collect = save_collect class FileTestCase(BaseTestCase): """ @@ -210,18 +217,6 @@ class FileTestCase(BaseTestCase): finally: SCons.CacheDir.CachePush = save_CachePush - def test_no_bsig(self): - """Test that no bsig raises an InternalError""" - - f6 = self.File("cd.f6") - f6.binfo = f6.BuildInfo(f6) - exc_caught = 0 - try: - cp = self._CacheDir.cachepath(f6) - except SCons.Errors.InternalError: - exc_caught = 1 - assert exc_caught - def test_warning(self): """Test raising a warning if we can't copy a file to cache.""" diff --git a/src/engine/SCons/Debug.py b/src/engine/SCons/Debug.py index 47d2134..c54eae2 100644 --- a/src/engine/SCons/Debug.py +++ b/src/engine/SCons/Debug.py @@ -197,3 +197,4 @@ def Trace(msg, file=None, mode='w'): # Assume we were passed an open file pointer. fp = file fp.write(msg) + fp.flush() diff --git a/src/engine/SCons/Defaults.py b/src/engine/SCons/Defaults.py index 29868ec..9308051 100644 --- a/src/engine/SCons/Defaults.py +++ b/src/engine/SCons/Defaults.py @@ -49,7 +49,6 @@ import SCons.Builder import SCons.CacheDir import SCons.Environment import SCons.PathList -import SCons.Sig import SCons.Subst import SCons.Tool @@ -61,12 +60,40 @@ _default_env = None # Lazily instantiate the default environment so the overhead of creating # it doesn't apply when it's not needed. +def _fetch_DefaultEnvironment(*args, **kw): + """ + Returns the already-created default construction environment. + """ + global _default_env + return _default_env + def DefaultEnvironment(*args, **kw): + """ + Initial public entry point for creating the default construction + Environment. + + After creating the environment, we overwrite our name + (DefaultEnvironment) with the _fetch_DefaultEnvironment() function, + which more efficiently returns the initialized default construction + environment without checking for its existence. + + (This function still exists with its _default_check because someone + else (*cough* Script/__init__.py *cough*) may keep a reference + to this function. So we can't use the fully functional idiom of + having the name originally be a something that *only* creates the + construction environment and then overwrites the name.) + """ global _default_env if not _default_env: + import SCons.Util _default_env = apply(SCons.Environment.Environment, args, kw) - _default_env._build_signature = 1 - _default_env._calc_module = SCons.Sig.default_module + _default_env.TargetSignatures('source') + if SCons.Util.md5: + _default_env.SourceSignatures('MD5') + else: + _default_env.SourceSignatures('timestamp') + global DefaultEnvironment + DefaultEnvironment = _fetch_DefaultEnvironment _default_env._CacheDir = SCons.CacheDir.Null() return _default_env @@ -106,9 +133,9 @@ LaTeXScan = SCons.Tool.LaTeXScanner ObjSourceScan = SCons.Tool.SourceFileScanner ProgScan = SCons.Tool.ProgramScanner -# This isn't really a tool scanner, so it doesn't quite belong with -# the rest of those in Tool/__init__.py, but I'm not sure where else it -# should go. Leave it here for now. +# These aren't really tool scanners, so they don't quite belong with +# the rest of those in Tool/__init__.py, but I'm not sure where else +# they should go. Leave them here for now. import SCons.Scanner.Dir DirScanner = SCons.Scanner.Dir.DirScanner() DirEntryScanner = SCons.Scanner.Dir.DirEntryScanner() diff --git a/src/engine/SCons/Environment.py b/src/engine/SCons/Environment.py index 012c36c..2f4c34e 100644 --- a/src/engine/SCons/Environment.py +++ b/src/engine/SCons/Environment.py @@ -38,6 +38,7 @@ __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" import copy import os import os.path +import shlex import string from UserDict import UserDict @@ -53,7 +54,6 @@ import SCons.Node.FS import SCons.Node.Python import SCons.Platform import SCons.SConsign -import SCons.Sig import SCons.Subst import SCons.Tool import SCons.Util @@ -125,8 +125,9 @@ def _set_BUILDERS(env, key, value): for k in bd.keys(): del bd[k] except KeyError: - env._dict[key] = BuilderDict(kwbd, env) - env._dict[key].update(value) + bd = BuilderDict(kwbd, env) + env._dict[key] = bd + bd.update(value) def _del_SCANNERS(env, key): del env._dict[key] @@ -136,13 +137,72 @@ def _set_SCANNERS(env, key, value): env._dict[key] = value env.scanner_map_delete() -class BuilderWrapper: - """Wrapper class that associates an environment with a Builder at - instantiation.""" - def __init__(self, env, builder): - self.env = env - self.builder = builder + +# The following is partly based on code in a comment added by Peter +# Shannon at the following page (there called the "transplant" class): +# +# ASPN : Python Cookbook : Dynamically added methods to a class +# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/81732 +# +# We had independently been using the idiom as BuilderWrapper, but +# factoring out the common parts into this base class, and making +# BuilderWrapper a subclass that overrides __call__() to enforce specific +# Builder calling conventions, simplified some of our higher-layer code. + +class MethodWrapper: + """ + A generic Wrapper class that associates a method (which can + actually be any callable) with an object. As part of creating this + MethodWrapper object an attribute with the specified (by default, + the name of the supplied method) is added to the underlying object. + When that new "method" is called, our __call__() method adds the + object as the first argument, simulating the Python behavior of + supplying "self" on method calls. + + We hang on to the name by which the method was added to the underlying + base class so that we can provide a method to "clone" ourselves onto + a new underlying object being copied (without which we wouldn't need + to save that info). + """ + def __init__(self, object, method, name=None): + if name is None: + name = method.__name__ + self.object = object + self.method = method + self.name = name + setattr(self.object, name, self) + + def __call__(self, *args, **kwargs): + nargs = (self.object,) + args + return apply(self.method, nargs, kwargs) + + def clone(self, new_object): + """ + Returns an object that re-binds the underlying "method" to + the specified new object. + """ + return self.__class__(new_object, self.method, self.name) + +class BuilderWrapper(MethodWrapper): + """ + A MethodWrapper subclass that that associates an environment with + a Builder. + + This mainly exists to wrap the __call__() function so that all calls + to Builders can have their argument lists massaged in the same way + (treat a lone argument as the source, treat two arguments as target + then source, make sure both target and source are lists) without + having to have cut-and-paste code to do it. + + As a bit of obsessive backwards compatibility, we also intercept + attempts to get or set the "env" or "builder" attributes, which were + the names we used before we put the common functionality into the + MethodWrapper base class. We'll keep this around for a while in case + people shipped Tool modules that reached into the wrapper (like the + Tool/qt.py module does, or did). There shouldn't be a lot attribute + fetching or setting on these, so a little extra work shouldn't hurt. + """ def __call__(self, target=None, source=_null, *args, **kw): if source is _null: source = target @@ -151,7 +211,26 @@ class BuilderWrapper: target = [target] if not source is None and not SCons.Util.is_List(source): source = [source] - return apply(self.builder, (self.env, target, source) + args, kw) + return apply(MethodWrapper.__call__, (self, target, source) + args, kw) + + def __repr__(self): + return '<BuilderWrapper %s>' % repr(self.name) + + def __getattr__(self, name): + if name == 'env': + return self.object + elif name == 'builder': + return self.method + else: + return self.__dict__[name] + + def __setattr__(self, name, value): + if name == 'env': + self.object = value + elif name == 'builder': + self.method = value + else: + self.__dict__[name] = value # This allows a Builder to be executed directly # through the Environment to which it's attached. @@ -160,9 +239,9 @@ class BuilderWrapper: # But we do have a unit test for this, and can't # yet rule out that it would be useful in the # future, so leave it for now. - def execute(self, **kw): - kw['env'] = self.env - apply(self.builder.execute, (), kw) + #def execute(self, **kw): + # kw['env'] = self.env + # apply(self.builder.execute, (), kw) class BuilderDict(UserDict): """This is a dictionary-like class used by an Environment to hold @@ -181,26 +260,7 @@ class BuilderDict(UserDict): def __setitem__(self, item, val): UserDict.__setitem__(self, item, val) - try: - self.setenvattr(item, val) - except AttributeError: - # Have to catch this because sometimes __setitem__ gets - # called out of __init__, when we don't have an env - # attribute yet, nor do we want one! - pass - - def setenvattr(self, item, val): - """Set the corresponding environment attribute for this Builder. - - If the value is already a BuilderWrapper, we pull the builder - out of it and make another one, so that making a copy of an - existing BuilderDict is guaranteed separate wrappers for each - Builder + Environment pair.""" - try: - builder = val.builder - except AttributeError: - builder = val - setattr(self.env, item, BuilderWrapper(self.env, builder)) + BuilderWrapper(self.env, val, item) def __delitem__(self, item): UserDict.__delitem__(self, item) @@ -248,6 +308,7 @@ class SubstitutionEnvironment: self.lookup_list = SCons.Node.arg2nodes_lookups self._dict = kw.copy() self._init_special() + self.added_methods = [] #self._memo = {} def _init_special(self): @@ -450,7 +511,16 @@ class SubstitutionEnvironment: environment with the specified name. If the name is omitted, the default name is the name of the function itself. """ - SCons.Util.AddMethod(self, function, name) + method = MethodWrapper(self, function, name) + self.added_methods.append(method) + + def RemoveMethod(self, function): + """ + Removes the specified function's MethodWrapper from the + added_methods list, so we don't re-bind it when making a clone. + """ + is_not_func = lambda dm, f=function: not dm.method is f + self.added_methods = filter(is_not_func, self.added_methods) def Override(self, overrides): """ @@ -543,7 +613,7 @@ class SubstitutionEnvironment: # -R dir (deprecated linker rpath) # IBM compilers may also accept -qframeworkdir=foo - params = string.split(arg) + params = shlex.split(arg) append_next_arg_to = None # for multi-word args for arg in params: if append_next_arg_to: @@ -695,6 +765,14 @@ def build_source(ss, result): elif isinstance(s.disambiguate(), SCons.Node.FS.File): result.append(s) +def default_decide_source(dependency, target, prev_ni): + f = SCons.Defaults.DefaultEnvironment().decide_source + return f(dependency, target, prev_ni) + +def default_decide_target(dependency, target, prev_ni): + f = SCons.Defaults.DefaultEnvironment().decide_target + return f(dependency, target, prev_ni) + class Base(SubstitutionEnvironment): """Base class for "real" construction Environments. These are the primary objects used to communicate dependency and construction @@ -747,6 +825,16 @@ class Base(SubstitutionEnvironment): self.lookup_list = SCons.Node.arg2nodes_lookups self._dict = semi_deepcopy(SCons.Defaults.ConstructionEnvironment) self._init_special() + self.added_methods = [] + + # We don't use AddMethod, or define these as methods in this + # class, because we *don't* want these functions to be bound + # methods. They need to operate independently so that the + # settings will work properly regardless of whether a given + # target ends up being built with a Base environment or an + # OverrideEnvironment or what have you. + self.decide_target = default_decide_target + self.decide_source = default_decide_source self._dict['BUILDERS'] = BuilderDict(self._dict['BUILDERS'], self) @@ -777,6 +865,8 @@ class Base(SubstitutionEnvironment): # reserved variable name like TARGETS. pass + SCons.Tool.Initializers(self) + if tools is None: tools = self._dict.get('TOOLS', None) if tools is None: @@ -802,17 +892,6 @@ class Base(SubstitutionEnvironment): except KeyError: return None - def get_calculator(self): - try: - module = self._calc_module - c = apply(SCons.Sig.Calculator, (module,), CalculatorArgs) - except AttributeError: - # Note that we're calling get_calculator() here, so the - # DefaultEnvironment() must have a _calc_module attribute - # to avoid infinite recursion. - c = SCons.Defaults.DefaultEnvironment().get_calculator() - return c - def get_CacheDir(self): try: return self._CacheDir @@ -901,13 +980,21 @@ class Base(SubstitutionEnvironment): """ self._dict.update(dict) - def use_build_signature(self): + def get_src_sig_type(self): + try: + return self.src_sig_type + except AttributeError: + t = SCons.Defaults.DefaultEnvironment().src_sig_type + self.src_sig_type = t + return t + + def get_tgt_sig_type(self): try: - return self._build_signature + return self.tgt_sig_type except AttributeError: - b = SCons.Defaults.DefaultEnvironment()._build_signature - self._build_signature = b - return b + t = SCons.Defaults.DefaultEnvironment().tgt_sig_type + self.tgt_sig_type = t + return t ####################################################################### # Public methods for manipulating an Environment. These begin with @@ -1040,28 +1127,84 @@ class Base(SubstitutionEnvironment): """ clone = copy.copy(self) clone._dict = semi_deepcopy(self._dict) + try: cbd = clone._dict['BUILDERS'] - clone._dict['BUILDERS'] = BuilderDict(cbd, clone) except KeyError: pass + else: + clone._dict['BUILDERS'] = BuilderDict(cbd, clone) - clone._memo = {} + clone.added_methods = [] + for mw in self.added_methods: + mw.clone(clone) - apply_tools(clone, tools, toolpath) + clone._memo = {} - # Apply passed-in variables after the new tools. + # Apply passed-in variables before the tools + # so the tools can use the new variables kw = copy_non_reserved_keywords(kw) new = {} for key, value in kw.items(): new[key] = SCons.Subst.scons_subst_once(value, self, key) apply(clone.Replace, (), new) + + apply_tools(clone, tools, toolpath) + + # apply them again in case the tools overwrote them + apply(clone.Replace, (), new) + if __debug__: logInstanceCreation(self, 'Environment.EnvironmentClone') return clone def Copy(self, *args, **kw): return apply(self.Clone, args, kw) + def _changed_build(self, dependency, target, prev_ni): + if dependency.changed_state(target, prev_ni): + return 1 + return self.decide_source(dependency, target, prev_ni) + + def _changed_content(self, dependency, target, prev_ni): + return dependency.changed_content(target, prev_ni) + + def _changed_source(self, dependency, target, prev_ni): + target_env = dependency.get_build_env() + type = target_env.get_tgt_sig_type() + if type == 'source': + return target_env.decide_source(dependency, target, prev_ni) + else: + return target_env.decide_target(dependency, target, prev_ni) + + def _changed_timestamp_then_content(self, dependency, target, prev_ni): + return dependency.changed_timestamp_then_content(target, prev_ni) + + def _changed_timestamp_newer(self, dependency, target, prev_ni): + return dependency.changed_timestamp_newer(target, prev_ni) + + def _changed_timestamp_match(self, dependency, target, prev_ni): + return dependency.changed_timestamp_match(target, prev_ni) + + def Decider(self, function): + if function in ('MD5', 'content'): + if not SCons.Util.md5: + raise UserError, "MD5 signatures are not available in this version of Python." + function = self._changed_content + elif function == 'MD5-timestamp': + function = self._changed_timestamp_then_content + elif function in ('timestamp-newer', 'make'): + function = self._changed_timestamp_newer + elif function == 'timestamp-match': + function = self._changed_timestamp_match + elif not callable(function): + raise UserError, "Unknown Decider value %s" % repr(function) + + # We don't use AddMethod because we don't want to turn the + # function, which only expects three arguments, into a bound + # method, which would add self as an initial, fourth argument. + self.decide_target = function + self.decide_source = function + def Detect(self, progs): """Return the first available program in progs. """ @@ -1294,11 +1437,13 @@ class Base(SubstitutionEnvironment): with new construction variables and/or values. """ try: - kwbd = semi_deepcopy(kw['BUILDERS']) - del kw['BUILDERS'] - self.__setitem__('BUILDERS', kwbd) + kwbd = kw['BUILDERS'] except KeyError: pass + else: + kwbd = semi_deepcopy(kwbd) + del kw['BUILDERS'] + self.__setitem__('BUILDERS', kwbd) kw = copy_non_reserved_keywords(kw) self._update(semi_deepcopy(kw)) self.scanner_map_delete(kw) @@ -1659,21 +1804,15 @@ class Base(SubstitutionEnvironment): def SourceSignatures(self, type): type = self.subst(type) + self.src_sig_type = type if type == 'MD5': - try: - import SCons.Sig.MD5 - except ImportError: - msg = "No MD5 module available, using time stamps" - SCons.Warnings.warn(SCons.Warnings.NoMD5ModuleWarning, msg) - import SCons.Sig.TimeStamp - self._calc_module = SCons.Sig.TimeStamp - else: - self._calc_module = SCons.Sig.MD5 + if not SCons.Util.md5: + raise UserError, "MD5 signatures are not available in this version of Python." + self.decide_source = self._changed_content elif type == 'timestamp': - import SCons.Sig.TimeStamp - self._calc_module = SCons.Sig.TimeStamp + self.decide_source = self._changed_timestamp_newer else: - raise UserError, "Unknown source signature type '%s'"%type + raise UserError, "Unknown source signature type '%s'" % type def Split(self, arg): """This function converts a string or list into a list of strings @@ -1695,12 +1834,19 @@ class Base(SubstitutionEnvironment): def TargetSignatures(self, type): type = self.subst(type) - if type == 'build': - self._build_signature = 1 - elif type == 'content': - self._build_signature = 0 + self.tgt_sig_type = type + if type in ('MD5', 'content'): + if not SCons.Util.md5: + raise UserError, "MD5 signatures are not available in this version of Python." + self.decide_target = self._changed_content + elif type == 'timestamp': + self.decide_target = self._changed_timestamp_newer + elif type == 'build': + self.decide_target = self._changed_build + elif type == 'source': + self.decide_target = self._changed_source else: - raise SCons.Errors.UserError, "Unknown target signature type '%s'"%type + raise UserError, "Unknown target signature type '%s'"%type def Value(self, value, built_value=None): """ @@ -1740,6 +1886,14 @@ class Base(SubstitutionEnvironment): # remove duplicates return list(set(sources)) + def FindInstalledFiles(self): + """ returns the list of all targets of the Install and InstallAs Builder. + """ + from SCons.Tool import install + if install._UNIQUE_INSTALLED_FILES is None: + install._UNIQUE_INSTALLED_FILES = SCons.Util.uniquer_hashables(install._INSTALLED_FILES) + return install._UNIQUE_INSTALLED_FILES + class OverrideEnvironment(Base): """A proxy that overrides variables in a wrapped construction environment by returning values from an overrides dictionary in diff --git a/src/engine/SCons/EnvironmentTests.py b/src/engine/SCons/EnvironmentTests.py index 82f220a..630f594 100644 --- a/src/engine/SCons/EnvironmentTests.py +++ b/src/engine/SCons/EnvironmentTests.py @@ -729,7 +729,9 @@ sys.exit(1) assert d == empty, d s = "-I/usr/include/fum -I bar -X\n" + \ + '-I"C:\\Program Files\\ASCEND\\include" ' + \ "-L/usr/fax -L foo -lxxx -l yyy " + \ + '-L"C:\\Program Files\\ASCEND" -lascend ' + \ "-Wa,-as -Wl,-link " + \ "-Wl,-rpath=rpath1 " + \ "-Wl,-R,rpath2 " + \ @@ -743,10 +745,44 @@ sys.exit(1) "-pthread " + \ "-mno-cygwin -mwindows " + \ "-arch i386 -isysroot /tmp +DD64 " + \ - "-DFOO -DBAR=value -D BAZ" + "-DFOO -DBAR=value -D BAZ " d = env.ParseFlags(s) + if sys.version[:3] in ('1.5', '1.6', '2.0', '2.1', '2.2'): + # Pre-2.3 Python has no shlex.split() function. + # The compatibility layer does its best can by wrapping + # the old shlex.shlex class, but that class doesn't really + # understand quoting within the body of a token. We're just + # going to live with this; it's the behavior they'd + # have anyway if they use the shlex module... + # + # (Note that we must test the actual Python version numbers + # above, not just test for whether trying to use shlex.split() + # throws an AttributeError, because the compatibility layer + # adds our wrapper function to the module as shlex.split().) + + expect_CPPPATH = ['/usr/include/fum', + 'bar', + '"C:\\Program'] + expect_LIBPATH = ['/usr/fax', + 'foo', + '"C:\\Program'] + expect_LIBS = ['Files\\ASCEND\\include"', + 'xxx', + 'yyy', + 'Files\\ASCEND"', + 'ascend'] + else: + expect_CPPPATH = ['/usr/include/fum', + 'bar', + 'C:\\Program Files\\ASCEND\\include'] + expect_LIBPATH = ['/usr/fax', + 'foo', + 'C:\\Program Files\\ASCEND'] + expect_LIBS = ['xxx', 'yyy', 'ascend'] + + assert d['ASFLAGS'] == ['-as'], d['ASFLAGS'] assert d['CFLAGS'] == ['-std=c99'] assert d['CCFLAGS'] == ['-X', '-Wa,-as', @@ -755,11 +791,12 @@ sys.exit(1) '+DD64'], d['CCFLAGS'] assert d['CPPDEFINES'] == ['FOO', ['BAR', 'value'], 'BAZ'], d['CPPDEFINES'] assert d['CPPFLAGS'] == ['-Wp,-cpp'], d['CPPFLAGS'] - assert d['CPPPATH'] == ['/usr/include/fum', 'bar'], d['CPPPATH'] + assert d['CPPPATH'] == expect_CPPPATH, d['CPPPATH'] assert d['FRAMEWORKPATH'] == ['fwd1', 'fwd2', 'fwd3'], d['FRAMEWORKPATH'] assert d['FRAMEWORKS'] == ['Carbon'], d['FRAMEWORKS'] - assert d['LIBPATH'] == ['/usr/fax', 'foo'], d['LIBPATH'] - assert d['LIBS'] == ['xxx', 'yyy'], d['LIBS'] + assert d['LIBPATH'] == expect_LIBPATH, d['LIBPATH'] + LIBS = map(str, d['LIBS']) + assert LIBS == expect_LIBS, (d['LIBS'], LIBS) assert d['LINKFLAGS'] == ['-Wl,-link', '-pthread', '-mno-cygwin', '-mwindows', ('-arch', 'i386'), @@ -866,7 +903,11 @@ class BaseTestCase(unittest.TestCase,TestEnvironmentFixture): - def test_Builder_execs(self): + # This unit test is currently disabled because we don't think the + # underlying method it tests (Environment.BuilderWrapper.execute()) + # is necessary, but we're leaving the code here for now in case + # that's mistaken. + def _DO_NOT_test_Builder_execs(self): """Test Builder execution through different environments One environment is initialized with a single @@ -890,8 +931,10 @@ class BaseTestCase(unittest.TestCase,TestEnvironmentFixture): assert built_it['out3'] env4 = env3.Clone() - assert env4.builder1.env is env4, "builder1.env (%s) == env3 (%s)?" % (env4.builder1.env, env3) - assert env4.builder2.env is env4, "builder2.env (%s) == env3 (%s)?" % (env4.builder1.env, env3) + assert env4.builder1.env is env4, "builder1.env (%s) == env3 (%s)?" % ( +env4.builder1.env, env3) + assert env4.builder2.env is env4, "builder2.env (%s) == env3 (%s)?" % ( +env4.builder1.env, env3) # Now test BUILDERS as a dictionary. built_it = {} @@ -911,6 +954,8 @@ class BaseTestCase(unittest.TestCase,TestEnvironmentFixture): assert built_it['out1'] assert built_it['out2'] + + def test_Scanners(self): """Test setting SCANNERS in various ways @@ -1568,15 +1613,15 @@ def exists(env): # env1 = self.TestEnvironment(BUILDERS = {'b1' : 1}) assert hasattr(env1, 'b1'), "env1.b1 was not set" - assert env1.b1.env == env1, "b1.env doesn't point to env1" + assert env1.b1.object == env1, "b1.object doesn't point to env1" env2 = env1.Clone(BUILDERS = {'b2' : 2}) assert env2 is env2 assert env2 == env2 assert hasattr(env1, 'b1'), "b1 was mistakenly cleared from env1" - assert env1.b1.env == env1, "b1.env was changed" + assert env1.b1.object == env1, "b1.object was changed" assert not hasattr(env2, 'b1'), "b1 was not cleared from env2" assert hasattr(env2, 'b2'), "env2.b2 was not set" - assert env2.b2.env == env2, "b2.env doesn't point to env2" + assert env2.b2.object == env2, "b2.object doesn't point to env2" # Ensure that specifying new tools in a copied environment # works. @@ -1640,6 +1685,25 @@ def generate(env): env = env.Clone(tools=['yyy']) assert env['YYY'] == 'two', env['YYY'] + + # Test that + real_value = [4] + + def my_tool(env, rv=real_value): + assert env['KEY_THAT_I_WANT'] == rv[0] + env['KEY_THAT_I_WANT'] = rv[0] + 1 + + env = self.TestEnvironment() + + real_value[0] = 5 + env = env.Clone(KEY_THAT_I_WANT=5, tools=[my_tool]) + assert env['KEY_THAT_I_WANT'] == real_value[0], env['KEY_THAT_I_WANT'] + + real_value[0] = 6 + env = env.Clone(KEY_THAT_I_WANT=6, tools=[my_tool]) + assert env['KEY_THAT_I_WANT'] == real_value[0], env['KEY_THAT_I_WANT'] + + def test_Copy(self): """Test copying using the old env.Copy() method""" env1 = self.TestEnvironment(XXX = 'x', YYY = 'y') @@ -2999,6 +3063,8 @@ def generate(env): def test_SourceSignatures(type): """Test the SourceSignatures() method""" + import SCons.Errors + env = type.TestEnvironment(M = 'MD5', T = 'timestamp') exc_caught = None @@ -3007,19 +3073,31 @@ def generate(env): except SCons.Errors.UserError: exc_caught = 1 assert exc_caught, "did not catch expected UserError" - assert not hasattr(env, '_calc_module') env.SourceSignatures('MD5') - m = env._calc_module + assert env.src_sig_type == 'MD5', env.src_sig_type env.SourceSignatures('$M') - assert env._calc_module is m + assert env.src_sig_type == 'MD5', env.src_sig_type env.SourceSignatures('timestamp') - t = env._calc_module + assert env.src_sig_type == 'timestamp', env.src_sig_type env.SourceSignatures('$T') - assert env._calc_module is t + assert env.src_sig_type == 'timestamp', env.src_sig_type + + try: + import SCons.Util + save_md5 = SCons.Util.md5 + SCons.Util.md5 = None + try: + env.SourceSignatures('MD5') + except SCons.Errors.UserError: + pass + else: + self.fail('Did not catch expected UserError') + finally: + SCons.Util.md5 = save_md5 def test_Split(self): """Test the Split() method""" @@ -3039,6 +3117,8 @@ def generate(env): def test_TargetSignatures(type): """Test the TargetSignatures() method""" + import SCons.Errors + env = type.TestEnvironment(B = 'build', C = 'content') exc_caught = None @@ -3050,16 +3130,41 @@ def generate(env): assert not hasattr(env, '_build_signature') env.TargetSignatures('build') - assert env._build_signature == 1, env._build_signature - - env.TargetSignatures('content') - assert env._build_signature == 0, env._build_signature + assert env.tgt_sig_type == 'build', env.tgt_sig_type env.TargetSignatures('$B') - assert env._build_signature == 1, env._build_signature + assert env.tgt_sig_type == 'build', env.tgt_sig_type + + env.TargetSignatures('content') + assert env.tgt_sig_type == 'content', env.tgt_sig_type env.TargetSignatures('$C') - assert env._build_signature == 0, env._build_signature + assert env.tgt_sig_type == 'content', env.tgt_sig_type + + env.TargetSignatures('MD5') + assert env.tgt_sig_type == 'MD5', env.tgt_sig_type + + env.TargetSignatures('timestamp') + assert env.tgt_sig_type == 'timestamp', env.tgt_sig_type + + try: + import SCons.Util + save_md5 = SCons.Util.md5 + SCons.Util.md5 = None + try: + env.TargetSignatures('MD5') + except SCons.Errors.UserError: + pass + else: + self.fail('Did not catch expected UserError') + try: + env.TargetSignatures('content') + except SCons.Errors.UserError: + pass + else: + self.fail('Did not catch expected UserError') + finally: + SCons.Util.md5 = save_md5 def test_Value(self): """Test creating a Value() object diff --git a/src/engine/SCons/Executor.py b/src/engine/SCons/Executor.py index 7d8df68..88a46cc 100644 --- a/src/engine/SCons/Executor.py +++ b/src/engine/SCons/Executor.py @@ -74,10 +74,17 @@ class Executor: def get_action_list(self): return self.pre_actions + self.action_list + self.post_actions + memoizer_counters.append(SCons.Memoize.CountValue('get_build_env')) + def get_build_env(self): """Fetch or create the appropriate build Environment for this Executor. """ + try: + return self._memo['get_build_env'] + except KeyError: + pass + # Create the build environment instance with appropriate # overrides. These get evaluated against the current # environment's construction variables so that users can @@ -91,11 +98,12 @@ class Executor: env = self.env or SCons.Defaults.DefaultEnvironment() build_env = env.Override(overrides) + self._memo['get_build_env'] = build_env + return build_env def get_build_scanner_path(self, scanner): - """Fetch the scanner path for this executor's targets - and sources. + """Fetch the scanner path for this executor's targets and sources. """ env = self.get_build_env() try: @@ -109,24 +117,26 @@ class Executor: result.update(kw) return result - def do_nothing(self, target, exitstatfunc, kw): - pass + def do_nothing(self, target, kw): + return 0 - def do_execute(self, target, exitstatfunc, kw): + def do_execute(self, target, kw): """Actually execute the action list.""" env = self.get_build_env() kw = self.get_kw(kw) + status = 0 for act in self.get_action_list(): - apply(act, - (self.targets, self.sources, env, exitstatfunc), - kw) + status = apply(act, (self.targets, self.sources, env), kw) + if status: + break + return status # use extra indirection because with new-style objects (Python 2.2 # and above) we can't override special methods, and nullify() needs # to be able to do this. - def __call__(self, target, exitstatfunc, **kw): - self.do_execute(target, exitstatfunc, kw) + def __call__(self, target, **kw): + return self.do_execute(target, kw) def cleanup(self): self._memo = {} @@ -218,10 +228,13 @@ class Executor: scanner_list = map(select_specific_scanner, scanner_list) scanner_list = filter(remove_null_scanners, scanner_list) scanner_path_list = map(add_scanner_path, scanner_list) + deps = [] for node, scanner, path in scanner_path_list: deps.extend(node.get_implicit_deps(env, scanner, path)) + deps.extend(self.get_implicit_deps()) + for tgt in self.targets: tgt.add_to_implicit(deps) @@ -280,6 +293,15 @@ class Executor: return result + def get_implicit_deps(self): + """Return the executor's implicit dependencies, i.e. the nodes of + the commands to be executed.""" + result = [] + build_env = self.get_build_env() + for act in self.get_action_list(): + result.extend(act.get_implicit_deps(self.targets, self.sources, build_env)) + return result + _Executor = Executor @@ -297,7 +319,15 @@ class Null(_Executor): apply(_Executor.__init__, (self,), kw) def get_build_env(self): import SCons.Util - return SCons.Util.Null() + class NullEnvironment(SCons.Util.Null): + #def get_scanner(self, key): + # return None + #def changed_since_last_build(self, dependency, target, prev_ni): + # return dependency.changed_since_last_buld(target, prev_ni) + def get_CacheDir(self): + import SCons.CacheDir + return SCons.CacheDir.Null() + return NullEnvironment() def get_build_scanner_path(self): return None def cleanup(self): diff --git a/src/engine/SCons/ExecutorTests.py b/src/engine/SCons/ExecutorTests.py index 8aedb76..59deca5 100644 --- a/src/engine/SCons/ExecutorTests.py +++ b/src/engine/SCons/ExecutorTests.py @@ -46,13 +46,15 @@ class MyEnvironment: class MyAction: def __init__(self, actions=['action1', 'action2']): self.actions = actions - def __call__(self, target, source, env, errfunc, **kw): + def __call__(self, target, source, env, **kw): for action in self.actions: - apply(action, (target, source, env, errfunc), kw) + apply(action, (target, source, env), kw) def genstring(self, target, source, env): return string.join(['GENSTRING'] + map(str, self.actions) + target + source) def get_contents(self, target, source, env): return string.join(self.actions + target + source) + def get_implicit_deps(self, target, source, env): + return [] class MyBuilder: def __init__(self, env, overrides): @@ -69,7 +71,7 @@ class MyNode: self.missing_val = None def __str__(self): return self.name - def build(self, errfunc=None): + def build(self): executor = SCons.Executor.Executor(MyAction(self.pre_actions + [self.builder.action] + self.post_actions), @@ -77,7 +79,7 @@ class MyNode: [], [self], ['s1', 's2']) - apply(executor, (self, errfunc), {}) + apply(executor, (self), {}) def get_env_scanner(self, env, kw): return MyScanner('dep-') def get_implicit_deps(self, env, scanner, path): @@ -207,13 +209,13 @@ class ExecutorTestCase(unittest.TestCase): def test__call__(self): """Test calling an Executor""" result = [] - def pre(target, source, env, errfunc, result=result, **kw): + def pre(target, source, env, result=result, **kw): result.append('pre') - def action1(target, source, env, errfunc, result=result, **kw): + def action1(target, source, env, result=result, **kw): result.append('action1') - def action2(target, source, env, errfunc, result=result, **kw): + def action2(target, source, env, result=result, **kw): result.append('action2') - def post(target, source, env, errfunc, result=result, **kw): + def post(target, source, env, result=result, **kw): result.append('post') env = MyEnvironment() @@ -223,32 +225,18 @@ class ExecutorTestCase(unittest.TestCase): x = SCons.Executor.Executor(a, env, [], t, ['s1', 's2']) x.add_pre_action(pre) x.add_post_action(post) - x(t, lambda x: x) + x(t) assert result == ['pre', 'action1', 'action2', 'post'], result del result[:] - def pre_err(target, source, env, errfunc, result=result, **kw): + def pre_err(target, source, env, result=result, **kw): result.append('pre_err') - if errfunc: - errfunc(1) return 1 x = SCons.Executor.Executor(a, env, [], t, ['s1', 's2']) x.add_pre_action(pre_err) x.add_post_action(post) - x(t, lambda x: x) - assert result == ['pre_err', 'action1', 'action2', 'post'], result - del result[:] - - def errfunc(stat): - raise "errfunc %s" % stat - - try: - x(t, errfunc) - except: - assert sys.exc_type == "errfunc 1", sys.exc_type - else: - assert 0, "did not catch expected exception" + x(t) assert result == ['pre_err'], result del result[:] @@ -319,14 +307,14 @@ class ExecutorTestCase(unittest.TestCase): env = MyEnvironment(S='string') result = [] - def action1(target, source, env, errfunc, result=result, **kw): + def action1(target, source, env, result=result, **kw): result.append('action1') env = MyEnvironment() a = MyAction([action1]) x = SCons.Executor.Executor(a, env, [], ['t1', 't2'], ['s1', 's2']) - x(MyNode('', [], []), None) + x(MyNode('', [], [])) assert result == ['action1'], result s = str(x) assert s[:10] == 'GENSTRING ', s @@ -335,7 +323,7 @@ class ExecutorTestCase(unittest.TestCase): x.nullify() assert result == [], result - x(MyNode('', [], []), None) + x(MyNode('', [], [])) assert result == [], result s = str(x) assert s == '', s @@ -367,7 +355,7 @@ class ExecutorTestCase(unittest.TestCase): t1 = MyNode('t1') t2 = MyNode('t2') sources = [MyNode('s1'), MyNode('s2')] - x = SCons.Executor.Executor('b', env, [{}], [t1, t2], sources) + x = SCons.Executor.Executor(MyAction(), env, [{}], [t1, t2], sources) deps = x.scan_targets(None) assert t1.implicit == ['dep-t1', 'dep-t2'], t1.implicit @@ -386,7 +374,7 @@ class ExecutorTestCase(unittest.TestCase): t1 = MyNode('t1') t2 = MyNode('t2') sources = [MyNode('s1'), MyNode('s2')] - x = SCons.Executor.Executor('b', env, [{}], [t1, t2], sources) + x = SCons.Executor.Executor(MyAction(), env, [{}], [t1, t2], sources) deps = x.scan_sources(None) assert t1.implicit == ['dep-s1', 'dep-s2'], t1.implicit diff --git a/src/engine/SCons/JobTests.py b/src/engine/SCons/JobTests.py index 5a47da7..5f056e8 100644 --- a/src/engine/SCons/JobTests.py +++ b/src/engine/SCons/JobTests.py @@ -359,14 +359,21 @@ import SCons.Taskmaster import SCons.Node import time +class DummyNodeInfo: + def update(self, obj): + pass class testnode (SCons.Node.Node): def __init__(self): SCons.Node.Node.__init__(self) self.expect_to_be = SCons.Node.executed + self.ninfo = DummyNodeInfo() class goodnode (testnode): - pass + def __init__(self): + SCons.Node.Node.__init__(self) + self.expect_to_be = SCons.Node.up_to_date + self.ninfo = DummyNodeInfo() class slowgoodnode (goodnode): def prepare(self): @@ -469,8 +476,9 @@ class _SConsTaskTest(unittest.TestCase): # mislabelling of results). for N in testnodes: - self.failUnless(N.get_state() in [SCons.Node.no_state, N.expect_to_be], - "node ran but got unexpected result") + state = N.get_state() + self.failUnless(state in [SCons.Node.no_state, N.expect_to_be], + "Node %s got unexpected result: %s" % (N, state)) self.failUnless(filter(lambda N: N.get_state(), testnodes), "no nodes ran at all.") diff --git a/src/engine/SCons/Node/Alias.py b/src/engine/SCons/Node/Alias.py index 6d4440b..15de664 100644 --- a/src/engine/SCons/Node/Alias.py +++ b/src/engine/SCons/Node/Alias.py @@ -57,10 +57,13 @@ class AliasNameSpace(UserDict.UserDict): return None class AliasNodeInfo(SCons.Node.NodeInfoBase): - pass + current_version_id = 1 + field_list = ['csig'] + def str_to_node(self, s): + return default_ans.Alias(s) class AliasBuildInfo(SCons.Node.BuildInfoBase): - pass + current_version_id = 1 class Alias(SCons.Node.Node): @@ -74,8 +77,11 @@ class Alias(SCons.Node.Node): def __str__(self): return self.name + def make_ready(self): + self.get_csig() + really_build = SCons.Node.Node.build - current = SCons.Node.Node.children_are_up_to_date + is_up_to_date = SCons.Node.Node.children_are_up_to_date def is_under(self, dir): # Make Alias nodes get built regardless of @@ -97,6 +103,13 @@ class Alias(SCons.Node.Node): # # + def changed_since_last_build(self, target, prev_ni): + cur_csig = self.get_csig() + try: + return cur_csig != prev_ni.csig + except AttributeError: + return 1 + def build(self): """A "builder" for aliases.""" pass @@ -107,6 +120,25 @@ class Alias(SCons.Node.Node): self.reset_executor() self.build = self.really_build + def get_csig(self): + """ + Generate a node's content signature, the digested signature + of its content. + + node - the node + cache - alternate node to use for the signature cache + returns - the content signature + """ + try: + return self.ninfo.csig + except AttributeError: + pass + + contents = self.get_contents() + csig = SCons.Util.MD5signature(contents) + self.get_ninfo().csig = csig + return csig + default_ans = AliasNameSpace() SCons.Node.arg2nodes_lookups.append(default_ans.lookup) diff --git a/src/engine/SCons/Node/FS.py b/src/engine/SCons/Node/FS.py index 74dc655..964af62 100644 --- a/src/engine/SCons/Node/FS.py +++ b/src/engine/SCons/Node/FS.py @@ -49,10 +49,13 @@ from SCons.Debug import logInstanceCreation import SCons.Errors import SCons.Memoize import SCons.Node +import SCons.Node.Alias import SCons.Subst import SCons.Util import SCons.Warnings +from SCons.Debug import Trace + # The max_drift value: by default, use a cached signature value for # any file that's been untouched for more than two days. default_max_drift = 2*24*60*60 @@ -487,6 +490,7 @@ class Base(SCons.Node.Node): assert directory, "A directory must be provided" self.abspath = directory.entry_abspath(name) + self.labspath = directory.entry_labspath(name) if directory.path == '.': self.path = name else: @@ -841,11 +845,17 @@ class Entry(Base): directory.""" return self.disambiguate().exists() - def rel_path(self, other): - d = self.disambiguate() - if d.__class__ == Entry: - raise "rel_path() could not disambiguate File/Dir" - return d.rel_path(other) +# def rel_path(self, other): +# d = self.disambiguate() +# if d.__class__ == Entry: +# raise "rel_path() could not disambiguate File/Dir" +# return d.rel_path(other) + + def new_ninfo(self): + return self.disambiguate().new_ninfo() + + def changed_since_last_build(self, target, prev_ni): + return self.disambiguate().changed_since_last_build(target, prev_ni) # This is for later so we can differentiate between Entry the class and Entry # the method of the FS class. @@ -960,11 +970,14 @@ class FS(LocalFS): self.pathTop = path self.defaultDrive = _my_normcase(os.path.splitdrive(self.pathTop)[0]) - self.Top = self._doLookup(Dir, os.path.normpath(self.pathTop)) + self.Top = self.Dir(self.pathTop) self.Top.path = '.' self.Top.tpath = '.' self._cwd = self.Top + DirNodeInfo.top = self.Top + FileNodeInfo.top = self.Top + def set_SConstruct_dir(self, dir): self.SConstruct_dir = dir @@ -977,161 +990,6 @@ class FS(LocalFS): def getcwd(self): return self._cwd - def _doLookup_key(self, fsclass, name, directory = None, create = 1): - return (fsclass, name, directory) - - memoizer_counters.append(SCons.Memoize.CountDict('_doLookup', _doLookup_key)) - - def _doLookup(self, fsclass, name, directory = None, create = 1): - """This method differs from the File and Dir factory methods in - one important way: the meaning of the directory parameter. - In this method, if directory is None or not supplied, the supplied - name is expected to be an absolute path. If you try to look up a - relative path with directory=None, then an AssertionError will be - raised. - """ - memo_key = (fsclass, name, directory) - try: - memo_dict = self._memo['_doLookup'] - except KeyError: - memo_dict = {} - self._memo['_doLookup'] = memo_dict - else: - try: - return memo_dict[memo_key] - except KeyError: - pass - - if not name: - # This is a stupid hack to compensate for the fact that the - # POSIX and Windows versions of os.path.normpath() behave - # differently in older versions of Python. In particular, - # in POSIX: - # os.path.normpath('./') == '.' - # in Windows: - # os.path.normpath('./') == '' - # os.path.normpath('.\\') == '' - # - # This is a definite bug in the Python library, but we have - # to live with it. - name = '.' - path_orig = string.split(name, os.sep) - path_norm = string.split(_my_normcase(name), os.sep) - - first_orig = path_orig.pop(0) # strip first element - unused = path_norm.pop(0) # strip first element - - drive, path_first = os.path.splitdrive(first_orig) - if path_first: - path_orig = [ path_first, ] + path_orig - path_norm = [ _my_normcase(path_first), ] + path_norm - else: - # Absolute path - drive = _my_normcase(drive) - try: - directory = self.Root[drive] - except KeyError: - if not create: - raise SCons.Errors.UserError - directory = RootDir(drive, self) - self.Root[drive] = directory - if not drive: - self.Root[self.defaultDrive] = directory - elif drive == self.defaultDrive: - self.Root[''] = directory - - if not path_orig: - memo_dict[memo_key] = directory - return directory - - last_orig = path_orig.pop() # strip last element - last_norm = path_norm.pop() # strip last element - - # Lookup the directory - for orig, norm in map(None, path_orig, path_norm): - try: - entries = directory.entries - except AttributeError: - # We tried to look up the entry in either an Entry or - # a File. Give whatever it is a chance to do what's - # appropriate: morph into a Dir or raise an exception. - directory.must_be_same(Dir) - entries = directory.entries - try: - directory = entries[norm] - except KeyError: - if not create: - raise SCons.Errors.UserError - - d = Dir(orig, directory, self) - - # Check the file system (or not, as configured) to make - # sure there isn't already a file there. - d.diskcheck_match() - - directory.entries[norm] = d - directory.add_wkid(d) - directory = d - - directory.must_be_same(Dir) - - try: - e = directory.entries[last_norm] - except KeyError: - if not create: - raise SCons.Errors.UserError - - result = fsclass(last_orig, directory, self) - - # Check the file system (or not, as configured) to make - # sure there isn't already a directory at the path on - # disk where we just created a File node, and vice versa. - result.diskcheck_match() - - directory.entries[last_norm] = result - directory.add_wkid(result) - else: - e.must_be_same(fsclass) - result = e - - memo_dict[memo_key] = result - - return result - - def _transformPath(self, name, directory): - """Take care of setting up the correct top-level directory, - usually in preparation for a call to doLookup(). - - If the path name is prepended with a '#', then it is unconditionally - interpreted as relative to the top-level directory of this FS. - - If directory is None, and name is a relative path, - then the same applies. - """ - try: - # Decide if this is a top-relative look up. The normal case - # (by far) is handed a non-zero-length string to look up, - # so just (try to) check for the initial '#'. - top_relative = (name[0] == '#') - except (AttributeError, IndexError): - # The exceptions we may encounter in unusual cases: - # AttributeError: a proxy without a __getitem__() method. - # IndexError: a null string. - top_relative = False - name = str(name) - if top_relative: - directory = self.Top - name = name[1:] - if name and (name[0] == os.sep or name[0] == '/'): - # Correct such that '#/foo' is equivalent - # to '#foo'. - name = name[1:] - name = os.path.normpath(os.path.join('.', name)) - return (name, directory) - elif not directory: - directory = self._cwd - return (os.path.normpath(name), directory) - def chdir(self, dir, change_os_dir=0): """Change the current working directory for lookups. If change_os_dir is true, we will also change the "real" cwd @@ -1147,25 +1005,86 @@ class FS(LocalFS): self._cwd = curr raise - def Entry(self, name, directory = None, create = 1, klass=None): + def get_root(self, drive): + """ + Returns the root directory for the specified drive, creating + it if necessary. + """ + drive = _my_normcase(drive) + try: + return self.Root[drive] + except KeyError: + root = RootDir(drive, self) + self.Root[drive] = root + if not drive: + self.Root[self.defaultDrive] = root + elif drive == self.defaultDrive: + self.Root[''] = root + return root + + def _lookup(self, p, directory, fsclass, create=1): + """ + The generic entry point for Node lookup with user-supplied data. + + This translates arbitrary input into a canonical Node.FS object + of the specified fsclass. The general approach for strings is + to turn it into a normalized absolute path and then call the + root directory's lookup_abs() method for the heavy lifting. + + If the path name begins with '#', it is unconditionally + interpreted relative to the top-level directory of this FS. + + If the path name is relative, then the path is looked up relative + to the specified directory, or the current directory (self._cwd, + typically the SConscript directory) if the specified directory + is None. + """ + if isinstance(p, Base): + # It's already a Node.FS object. Make sure it's the right + # class and return. + p.must_be_same(fsclass) + return p + # str(p) in case it's something like a proxy object + p = str(p) + drive, p = os.path.splitdrive(p) + if drive and not p: + # A drive letter without a path... + p = os.sep + root = self.get_root(drive) + elif os.path.isabs(p): + # An absolute path... + p = os.path.normpath(p) + root = self.get_root(drive) + else: + if p[0:1] == '#': + # A top-relative path... + directory = self.Top + offset = 1 + if p[1:2] in(os.sep, '/'): + offset = 2 + p = p[offset:] + else: + # A relative path... + if not directory: + # ...to the current (SConscript) directory. + directory = self._cwd + elif not isinstance(directory, Dir): + # ...to the specified directory. + directory = self.Dir(directory) + p = os.path.normpath(directory.labspath + '/' + p) + root = directory.root + if os.sep != '/': + p = string.replace(p, os.sep, '/') + return root._lookup_abs(p, fsclass, create) + + def Entry(self, name, directory = None, create = 1): """Lookup or create a generic Entry node with the specified name. If the name is a relative path (begins with ./, ../, or a file name), then it is looked up relative to the supplied directory node, or to the top level directory of the FS (supplied at construction time) if no directory is supplied. """ - - if not klass: - klass = Entry - - if isinstance(name, Base): - name.must_be_same(klass) - return name - else: - if directory and not isinstance(directory, Dir): - directory = self.Dir(directory) - name, directory = self._transformPath(name, directory) - return self._doLookup(klass, name, directory, create) + return self._lookup(name, directory, Entry, create) def File(self, name, directory = None, create = 1): """Lookup or create a File node with the specified name. If @@ -1177,7 +1096,7 @@ class FS(LocalFS): This method will raise TypeError if a directory is found at the specified path. """ - return self.Entry(name, directory, create, File) + return self._lookup(name, directory, File, create) def Dir(self, name, directory = None, create = 1): """Lookup or create a Dir node with the specified name. If @@ -1189,7 +1108,7 @@ class FS(LocalFS): This method will raise TypeError if a normal file is found at the specified path. """ - return self.Entry(name, directory, create, Dir) + return self._lookup(name, directory, Dir, create) def BuildDir(self, build_dir, src_dir, duplicate=1): """Link the supplied build directory to the source directory @@ -1242,10 +1161,22 @@ class FS(LocalFS): return targets, message class DirNodeInfo(SCons.Node.NodeInfoBase): - pass + # This should get reset by the FS initialization. + current_version_id = 1 + + top = None + + def str_to_node(self, s): + top = self.top + if os.path.isabs(s): + n = top.fs._lookup(s, top, Entry) + else: + s = top.labspath + '/' + s + n = top.root._lookup_abs(s, Entry) + return n class DirBuildInfo(SCons.Node.BuildInfoBase): - pass + current_version_id = 1 class Dir(Base): """A class for directories in a file system. @@ -1280,6 +1211,7 @@ class Dir(Base): self.searched = 0 self._sconsign = None self.build_dirs = [] + self.root = self.dir.root # Don't just reset the executor, replace its action list, # because it might have some pre-or post-actions that need to @@ -1314,18 +1246,44 @@ class Dir(Base): node.duplicate = node.get_dir().duplicate def Entry(self, name): - """Create an entry node named 'name' relative to this directory.""" + """ + Looks up or creates an entry node named 'name' relative to + this directory. + """ return self.fs.Entry(name, self) def Dir(self, name): - """Create a directory node named 'name' relative to this directory.""" + """ + Looks up or creates a directory node named 'name' relative to + this directory. + """ dir = self.fs.Dir(name, self) return dir def File(self, name): - """Create a file node named 'name' relative to this directory.""" + """ + Looks up or creates a file node named 'name' relative to + this directory. + """ return self.fs.File(name, self) + def _lookup_rel(self, name, klass, create=1): + """ + Looks up a *normalized* relative path name, relative to this + directory. + + This method is intended for use by internal lookups with + already-normalized path data. For general-purpose lookups, + use the Entry(), Dir() and File() methods above. + + This method does *no* input checking and will die or give + incorrect results if it's passed a non-normalized path name (e.g., + a path containing '..'), an absolute path name, a top-relative + ('#foo') path name, or any kind of object. + """ + name = self.labspath + '/' + name + return self.root._lookup_abs(name, klass, create) + def link(self, srcdir, duplicate): """Set this directory as the build directory for the supplied source directory.""" @@ -1371,57 +1329,66 @@ class Dir(Base): def up(self): return self.entries['..'] - def _rel_path_key(self, other): - return str(other) - - memoizer_counters.append(SCons.Memoize.CountDict('rel_path', _rel_path_key)) - - def rel_path(self, other): - """Return a path to "other" relative to this directory. - """ - try: - memo_dict = self._memo['rel_path'] - except KeyError: - memo_dict = {} - self._memo['rel_path'] = memo_dict - else: - try: - return memo_dict[other] - except KeyError: - pass - - if self is other: - - result = '.' - - elif not other in self.path_elements: - - try: - other_dir = other.get_dir() - except AttributeError: - result = str(other) - else: - if other_dir is None: - result = other.name - else: - dir_rel_path = self.rel_path(other_dir) - if dir_rel_path == '.': - result = other.name - else: - result = dir_rel_path + os.sep + other.name - - else: - - i = self.path_elements.index(other) + 1 - - path_elems = ['..'] * (len(self.path_elements) - i) \ - + map(lambda n: n.name, other.path_elements[i:]) - - result = string.join(path_elems, os.sep) - - memo_dict[other] = result - - return result +# This complicated method, which constructs relative paths between +# arbitrary Node.FS objects, is no longer used. It was introduced to +# store dependency paths in .sconsign files relative to the target, but +# that ended up being significantly inefficient. We're leaving the code +# here, commented out, because it would be too easy for someone to decide +# to re-invent this wheel in the future (if it becomes necessary) because +# they didn't know this code was buried in some source-code change from +# the distant past... +# +# def _rel_path_key(self, other): +# return str(other) +# +# memoizer_counters.append(SCons.Memoize.CountDict('rel_path', _rel_path_key)) +# +# def rel_path(self, other): +# """Return a path to "other" relative to this directory. +# """ +# try: +# memo_dict = self._memo['rel_path'] +# except KeyError: +# memo_dict = {} +# self._memo['rel_path'] = memo_dict +# else: +# try: +# return memo_dict[other] +# except KeyError: +# pass +# +# if self is other: +# +# result = '.' +# +# elif not other in self.path_elements: +# +# try: +# other_dir = other.get_dir() +# except AttributeError: +# result = str(other) +# else: +# if other_dir is None: +# result = other.name +# else: +# dir_rel_path = self.rel_path(other_dir) +# if dir_rel_path == '.': +# result = other.name +# else: +# result = dir_rel_path + os.sep + other.name +# +# else: +# +# i = self.path_elements.index(other) + 1 +# +# path_elems = ['..'] * (len(self.path_elements) - i) \ +# + map(lambda n: n.name, other.path_elements[i:]) +# +# result = string.join(path_elems, os.sep) +# +# memo_dict[other] = result +# +# return result def get_env_scanner(self, env, kw={}): import SCons.Defaults @@ -1452,12 +1419,23 @@ class Dir(Base): self.clear() return scanner(self, env, path) + # + # Taskmaster interface subsystem + # + + def prepare(self): + pass + def build(self, **kw): """A null "builder" for directories.""" global MkdirBuilder if not self.builder is MkdirBuilder: apply(SCons.Node.Node.build, [self,], kw) + # + # + # + def _create(self): """Create this directory, silently and without worrying about whether the builder is the default or not.""" @@ -1506,13 +1484,12 @@ class Dir(Base): contents = map(lambda n: n.get_contents(), self.children()) return string.join(contents, '') - def prepare(self): - pass - def do_duplicate(self, src): pass - def current(self, calc=None): + changed_since_last_build = SCons.Node.Node.state_has_changed + + def is_up_to_date(self): """If any child is not up-to-date, then this directory isn't, either.""" if not self.builder is MkdirBuilder and not self.exists(): @@ -1560,6 +1537,9 @@ class Dir(Base): def entry_abspath(self, name): return self.abspath + os.sep + name + def entry_labspath(self, name): + return self.labspath + '/' + name + def entry_path(self, name): return self.path + os.sep + name @@ -1640,7 +1620,7 @@ class Dir(Base): def func(node): if (isinstance(node, File) or isinstance(node, Entry)) and \ - (node.is_derived() or node.is_pseudo_derived() or node.exists()): + (node.is_derived() or node.exists()): return node return None @@ -1727,30 +1707,89 @@ class RootDir(Dir): # attribute) so we have to set up some values so Base.__init__() # won't gag won't it calls some of our methods. self.abspath = '' + self.labspath = '' self.path = '' self.tpath = '' self.path_elements = [] self.duplicate = 0 + self.root = self Base.__init__(self, name, self, fs) # Now set our paths to what we really want them to be: the - # initial drive letter (the name) plus the directory separator. + # initial drive letter (the name) plus the directory separator, + # except for the "lookup abspath," which does not have the + # drive letter. self.abspath = name + os.sep + self.labspath = '/' self.path = name + os.sep self.tpath = name + os.sep self._morph() + self._lookupDict = {} + + # The // and os.sep + os.sep entries are necessary because + # os.path.normpath() seems to preserve double slashes at the + # beginning of a path (presumably for UNC path names), but + # collapses triple slashes to a single slash. + self._lookupDict['/'] = self + self._lookupDict['//'] = self + self._lookupDict[os.sep] = self + self._lookupDict[os.sep + os.sep] = self + def must_be_same(self, klass): if klass is Dir: return Base.must_be_same(self, klass) + def _lookup_abs(self, p, klass, create=1): + """ + Fast (?) lookup of a *normalized* absolute path. + + This method is intended for use by internal lookups with + already-normalized path data. For general-purpose lookups, + use the FS.Entry(), FS.Dir() or FS.File() methods. + + The caller is responsible for making sure we're passed a + normalized absolute path; we merely let Python's dictionary look + up and return the One True Node.FS object for the path. + + If no Node for the specified "p" doesn't already exist, and + "create" is specified, the Node may be created after recursive + invocation to find or create the parent directory or directories. + """ + k = _my_normcase(p) + try: + result = self._lookupDict[k] + except KeyError: + if not create: + raise SCons.Errors.UserError + # There is no Node for this path name, and we're allowed + # to create it. + dir_name, file_name = os.path.split(p) + dir_node = self._lookup_abs(dir_name, Dir) + result = klass(file_name, dir_node, self.fs) + self._lookupDict[k] = result + dir_node.entries[_my_normcase(file_name)] = result + dir_node.implicit = None + + # Double-check on disk (as configured) that the Node we + # created matches whatever is out there in the real world. + result.diskcheck_match() + else: + # There is already a Node for this path name. Allow it to + # complain if we were looking for an inappropriate type. + result.must_be_same(klass) + return result + def __str__(self): return self.abspath def entry_abspath(self, name): return self.abspath + name + def entry_labspath(self, name): + return self.labspath + name + def entry_path(self, name): return self.path + name @@ -1773,90 +1812,96 @@ class RootDir(Dir): return _null class FileNodeInfo(SCons.Node.NodeInfoBase): - def __init__(self, node): - SCons.Node.NodeInfoBase.__init__(self, node) - self.update(node) - def __cmp__(self, other): - try: return cmp(self.bsig, other.bsig) - except AttributeError: return 1 - def update(self, node): - self.timestamp = node.get_timestamp() - self.size = node.getsize() + current_version_id = 1 + + field_list = ['csig', 'timestamp', 'size'] + + # This should get reset by the FS initialization. + top = None + + def str_to_node(self, s): + top = self.top + if os.path.isabs(s): + n = top.fs._lookup(s, top, Entry) + else: + s = top.labspath + '/' + s + n = top.root._lookup_abs(s, Entry) + return n class FileBuildInfo(SCons.Node.BuildInfoBase): - def __init__(self, node): - SCons.Node.BuildInfoBase.__init__(self, node) - self.node = node + current_version_id = 1 + def convert_to_sconsign(self): - """Convert this FileBuildInfo object for writing to a .sconsign file + """ + Converts this FileBuildInfo object for writing to a .sconsign file - We hung onto the node that we refer to so that we can translate - the lists of bsources, bdepends and bimplicit Nodes into strings - relative to the node, but we don't want to write out that Node - itself to the .sconsign file, so we delete the attribute in - preparation. + This replaces each Node in our various dependency lists with its + usual string representation: relative to the top-level SConstruct + directory, or an absolute path if it's outside. """ - rel_path = self.node.rel_path - delattr(self, 'node') + if os.sep == '/': + node_to_str = str + else: + def node_to_str(n): + try: + s = n.path + except AttributeError: + s = str(n) + else: + s = string.replace(s, os.sep, '/') + return s for attr in ['bsources', 'bdepends', 'bimplicit']: try: val = getattr(self, attr) except AttributeError: pass else: - setattr(self, attr, map(rel_path, val)) + setattr(self, attr, map(node_to_str, val)) def convert_from_sconsign(self, dir, name): - """Convert a newly-read FileBuildInfo object for in-SCons use - - An on-disk BuildInfo comes without a reference to the node for - which it's intended, so we have to convert the arguments and add - back a self.node attribute. We don't worry here about converting - the bsources, bdepends and bimplicit lists from strings to Nodes - because they're not used in the normal case of just deciding - whether or not to rebuild things. """ - self.node = dir.Entry(name) + Converts a newly-read FileBuildInfo object for in-SCons use + + For normal up-to-date checking, we don't have any conversion to + perform--but we're leaving this method here to make that clear. + """ + pass def prepare_dependencies(self): - """Prepare a FileBuildInfo object for explaining what changed + """ + Prepares a FileBuildInfo object for explaining what changed - The bsources, bdepends and bimplicit lists have all been stored - on disk as paths relative to the Node for which they're stored - as dependency info. Convert the strings to actual Nodes (for - use by the --debug=explain code and --implicit-cache). + The bsources, bdepends and bimplicit lists have all been + stored on disk as paths relative to the top-level SConstruct + directory. Convert the strings to actual Nodes (for use by the + --debug=explain code and --implicit-cache). """ - def str_to_node(s, entry=self.node.dir.Entry): - # This is a little bogus; we're going to mimic the lookup - # order of env.arg2nodes() by hard-coding an Alias lookup - # before we assume it's an Entry. This should be able to - # go away once the Big Signature Refactoring pickles the - # actual NodeInfo object, which will let us know precisely - # what type of Node to turn it into. - import SCons.Node.Alias - n = SCons.Node.Alias.default_ans.lookup(s) - if not n: - n = entry(s) - return n - for attr in ['bsources', 'bdepends', 'bimplicit']: + attrs = [ + ('bsources', 'bsourcesigs'), + ('bdepends', 'bdependsigs'), + ('bimplicit', 'bimplicitsigs'), + ] + for (nattr, sattr) in attrs: try: - val = getattr(self, attr) + strings = getattr(self, nattr) + nodeinfos = getattr(self, sattr) except AttributeError: pass else: - setattr(self, attr, map(str_to_node, val)) - def format(self): - result = [ self.ninfo.format() ] + nodes = [] + for s, ni in zip(strings, nodeinfos): + if not isinstance(s, SCons.Node.Node): + s = ni.str_to_node(s) + nodes.append(s) + setattr(self, nattr, nodes) + def format(self, names=0): + result = [] bkids = self.bsources + self.bdepends + self.bimplicit bkidsigs = self.bsourcesigs + self.bdependsigs + self.bimplicitsigs - for i in xrange(len(bkids)): - result.append(str(bkids[i]) + ': ' + bkidsigs[i].format()) + for bkid, bkidsig in zip(bkids, bkidsigs): + result.append(str(bkid) + ': ' + + string.join(bkidsig.format(names=names), ' ')) + result.append('%s [%s]' % (self.bactsig, self.bact)) return string.join(result, '\n') -class NodeInfo(FileNodeInfo): - pass - -class BuildInfo(FileBuildInfo): - pass - class File(Base): """A class for files in a file system. """ @@ -1908,6 +1953,19 @@ class File(Base): if not hasattr(self, '_local'): self._local = 0 + # If there was already a Builder set on this entry, then + # we need to make sure we call the target-decider function, + # not the source-decider. Reaching in and doing this by hand + # is a little bogus. We'd prefer to handle this by adding + # an Entry.builder_set() method that disambiguates like the + # other methods, but that starts running into problems with the + # fragile way we initialize Dir Nodes with their Mkdir builders, + # yet still allow them to be overridden by the user. Since it's + # not clear right now how to fix that, stick with what works + # until it becomes clear... + if self.has_builder(): + self.changed_since_last_build = self.decide_target + def scanner_key(self): return self.get_suffix() @@ -1923,47 +1981,198 @@ class File(Base): raise return r + memoizer_counters.append(SCons.Memoize.CountValue('get_size')) + + def get_size(self): + try: + return self._memo['get_size'] + except KeyError: + pass + + if self.rexists(): + size = self.rfile().getsize() + else: + size = 0 + + self._memo['get_size'] = size + + return size + + memoizer_counters.append(SCons.Memoize.CountValue('get_timestamp')) + def get_timestamp(self): + try: + return self._memo['get_timestamp'] + except KeyError: + pass + if self.rexists(): - return self.rfile().getmtime() + timestamp = self.rfile().getmtime() else: - return 0 + timestamp = 0 + + self._memo['get_timestamp'] = timestamp + + return timestamp - def store_info(self, obj): + def store_info(self): # Merge our build information into the already-stored entry. # This accomodates "chained builds" where a file that's a target # in one build (SConstruct file) is a source in a different build. # See test/chained-build.py for the use case. - entry = self.get_stored_info() - entry.merge(obj) - self.dir.sconsign().set_entry(self.name, entry) + self.dir.sconsign().store_info(self.name, self) + + convert_copy_attrs = [ + 'bsources', + 'bimplicit', + 'bdepends', + 'bact', + 'bactsig', + 'ninfo', + ] + + + convert_sig_attrs = [ + 'bsourcesigs', + 'bimplicitsigs', + 'bdependsigs', + ] + + def convert_old_entry(self, old_entry): + # Convert a .sconsign entry from before the Big Signature + # Refactoring, doing what we can to convert its information + # to the new .sconsign entry format. + # + # The old format looked essentially like this: + # + # BuildInfo + # .ninfo (NodeInfo) + # .bsig + # .csig + # .timestamp + # .size + # .bsources + # .bsourcesigs ("signature" list) + # .bdepends + # .bdependsigs ("signature" list) + # .bimplicit + # .bimplicitsigs ("signature" list) + # .bact + # .bactsig + # + # The new format looks like this: + # + # .ninfo (NodeInfo) + # .bsig + # .csig + # .timestamp + # .size + # .binfo (BuildInfo) + # .bsources + # .bsourcesigs (NodeInfo list) + # .bsig + # .csig + # .timestamp + # .size + # .bdepends + # .bdependsigs (NodeInfo list) + # .bsig + # .csig + # .timestamp + # .size + # .bimplicit + # .bimplicitsigs (NodeInfo list) + # .bsig + # .csig + # .timestamp + # .size + # .bact + # .bactsig + # + # The basic idea of the new structure is that a NodeInfo always + # holds all available information about the state of a given Node + # at a certain point in time. The various .b*sigs lists can just + # be a list of pointers to the .ninfo attributes of the different + # dependent nodes, without any copying of information until it's + # time to pickle it for writing out to a .sconsign file. + # + # The complicating issue is that the *old* format only stored one + # "signature" per dependency, based on however the *last* build + # was configured. We don't know from just looking at it whether + # it was a build signature, a content signature, or a timestamp + # "signature". Since we no longer use build signatures, the + # best we can do is look at the length and if it's thirty two, + # assume that it was (or might have been) a content signature. + # If it was actually a build signature, then it will cause a + # rebuild anyway when it doesn't match the new content signature, + # but that's probably the best we can do. + import SCons.SConsign + new_entry = SCons.SConsign.SConsignEntry() + new_entry.binfo = self.new_binfo() + binfo = new_entry.binfo + for attr in self.convert_copy_attrs: + try: + value = getattr(old_entry, attr) + except AttributeError: + pass + else: + setattr(binfo, attr, value) + delattr(old_entry, attr) + for attr in self.convert_sig_attrs: + try: + sig_list = getattr(old_entry, attr) + except AttributeError: + pass + else: + value = [] + for sig in sig_list: + ninfo = self.new_ninfo() + if len(sig) == 32: + ninfo.csig = sig + else: + ninfo.timestamp = sig + value.append(ninfo) + setattr(binfo, attr, value) + delattr(old_entry, attr) + return new_entry + + memoizer_counters.append(SCons.Memoize.CountValue('get_stored_info')) def get_stored_info(self): try: - stored = self.dir.sconsign().get_entry(self.name) + return self._memo['get_stored_info'] + except KeyError: + pass + + try: + sconsign_entry = self.dir.sconsign().get_entry(self.name) except (KeyError, OSError): - return self.new_binfo() + import SCons.SConsign + sconsign_entry = SCons.SConsign.SConsignEntry() + sconsign_entry.binfo = self.new_binfo() + sconsign_entry.ninfo = self.new_ninfo() else: - if not hasattr(stored, 'ninfo'): - # Transition: The .sconsign file entry has no NodeInfo - # object, which means it's a slightly older BuildInfo. - # Copy over the relevant attributes. - ninfo = stored.ninfo = self.new_ninfo() - for attr in ninfo.__dict__.keys(): - try: - setattr(ninfo, attr, getattr(stored, attr)) - except AttributeError: - pass - return stored + if isinstance(sconsign_entry, FileBuildInfo): + # This is a .sconsign file from before the Big Signature + # Refactoring; convert it as best we can. + sconsign_entry = self.convert_old_entry(sconsign_entry) + try: + delattr(sconsign_entry.ninfo, 'bsig') + except AttributeError: + pass + + self._memo['get_stored_info'] = sconsign_entry + + return sconsign_entry def get_stored_implicit(self): - binfo = self.get_stored_info() + binfo = self.get_stored_info().binfo binfo.prepare_dependencies() try: return binfo.bimplicit except AttributeError: return None - def rel_path(self, other): - return self.dir.rel_path(other) +# def rel_path(self, other): +# return self.dir.rel_path(other) def _get_found_includes_key(self, env, scanner, path): return (id(env), id(scanner), path) @@ -2037,6 +2246,63 @@ class File(Base): if self.exists(): self.get_build_env().get_CacheDir().push_if_forced(self) + ninfo = self.get_ninfo() + old = self.get_stored_info() + + csig = None + mtime = self.get_timestamp() + size = self.get_size() + + max_drift = self.fs.max_drift + if max_drift > 0: + if (time.time() - mtime) > max_drift: + try: + n = old.ninfo + if n.timestamp and n.csig and n.timestamp == mtime: + csig = n.csig + except AttributeError: + pass + elif max_drift == 0: + try: + csig = old.ninfo.csig + except AttributeError: + pass + + if csig: + ninfo.csig = csig + + ninfo.timestamp = mtime + ninfo.size = size + + if not self.has_builder(): + # This is a source file, but it might have been a target file + # in another build that included more of the DAG. Copy + # any build information that's stored in the .sconsign file + # into our binfo object so it doesn't get lost. + self.get_binfo().__dict__.update(old.binfo.__dict__) + + self.store_info() + + def find_src_builder(self): + if self.rexists(): + return None + scb = self.dir.src_builder() + if scb is _null: + if diskcheck_sccs(self.dir, self.name): + scb = get_DefaultSCCSBuilder() + elif diskcheck_rcs(self.dir, self.name): + scb = get_DefaultRCSBuilder() + else: + scb = None + if scb is not None: + try: + b = self.builder + except AttributeError: + b = None + if b is None: + self.builder_set(scb) + return scb + def has_src_builder(self): """Return whether this Node has a source builder or not. @@ -2051,20 +2317,7 @@ class File(Base): try: scb = self.sbuilder except AttributeError: - if self.rexists(): - scb = None - else: - scb = self.dir.src_builder() - if scb is _null: - if diskcheck_sccs(self.dir, self.name): - scb = get_DefaultSCCSBuilder() - elif diskcheck_rcs(self.dir, self.name): - scb = get_DefaultRCSBuilder() - else: - scb = None - if scb is not None: - self.builder_set(scb) - self.sbuilder = scb + scb = self.sbuilder = self.find_src_builder() return not scb is None def alter_targets(self): @@ -2074,13 +2327,18 @@ class File(Base): return [], None return self.fs.build_dir_target_climb(self, self.dir, [self.name]) - def is_pseudo_derived(self): - return self.has_src_builder() - def _rmv_existing(self): self.clear_memoized_values() Unlink(self, [], None) + # + # Taskmaster interface subsystem + # + + def make_ready(self): + self.has_src_builder() + self.get_binfo() + def prepare(self): """Prepare for this file to be created.""" SCons.Node.Node.prepare(self) @@ -2096,6 +2354,10 @@ class File(Base): desc = "No drive `%s' for target `%s'." % (drive, self) raise SCons.Errors.StopError, desc + # + # + # + def remove(self): """Remove this file.""" if self.exists() or self.islink(): @@ -2156,7 +2418,7 @@ class File(Base): # SIGNATURE SUBSYSTEM # - def get_csig(self, calc=None): + def get_csig(self): """ Generate a node's content signature, the digested signature of its content. @@ -2165,74 +2427,101 @@ class File(Base): cache - alternate node to use for the signature cache returns - the content signature """ + ninfo = self.get_ninfo() try: - return self.binfo.ninfo.csig + return ninfo.csig except AttributeError: pass - if calc is None: - calc = self.calculator() + try: + contents = self.get_contents() + except IOError: + # This can happen if there's actually a directory on-disk, + # which can be the case if they've disabled disk checks, + # or if an action with a File target actually happens to + # create a same-named directory by mistake. + csig = None + else: + csig = SCons.Util.MD5signature(contents) - max_drift = self.fs.max_drift - mtime = self.get_timestamp() - use_stored = max_drift >= 0 and (time.time() - mtime) > max_drift + ninfo.csig = csig - csig = None - if use_stored: - old = self.get_stored_info().ninfo + return csig + + # + # DECISION SUBSYSTEM + # + + def builder_set(self, builder): + SCons.Node.Node.builder_set(self, builder) + self.changed_since_last_build = self.decide_target + + def changed_content(self, target, prev_ni): + cur_csig = self.get_csig() + try: + return cur_csig != prev_ni.csig + except AttributeError: + return 1 + + def changed_state(self, target, prev_ni): + return (self.state != SCons.Node.up_to_date) + + def changed_timestamp_then_content(self, target, prev_ni): + if not self.changed_timestamp_match(target, prev_ni): try: - if old.timestamp and old.csig and old.timestamp == mtime: - csig = old.csig + self.get_ninfo().csig = prev_ni.csig except AttributeError: pass - if csig is None: - csig = calc.module.signature(self) + return False + return self.changed_content(target, prev_ni) - binfo = self.get_binfo() - ninfo = binfo.ninfo - ninfo.csig = csig - ninfo.update(self) + def changed_timestamp_newer(self, target, prev_ni): + try: + return self.get_timestamp() > target.get_timestamp() + except AttributeError: + return 1 + + def changed_timestamp_match(self, target, prev_ni): + try: + return self.get_timestamp() != prev_ni.timestamp + except AttributeError: + return 1 - if use_stored: - self.store_info(binfo) + def decide_source(self, target, prev_ni): + return target.get_build_env().decide_source(self, target, prev_ni) - return csig + def decide_target(self, target, prev_ni): + return target.get_build_env().decide_target(self, target, prev_ni) - # - # - # + # Initialize this Node's decider function to decide_source() because + # every file is a source file until it has a Builder attached... + changed_since_last_build = decide_source - def is_up_to_date(self, node=None, bi=None): - """Returns if the node is up-to-date with respect to stored - BuildInfo. The default is to compare it against our own - previously stored BuildInfo, but the stored BuildInfo from another - Node (typically one in a Repository) can be used instead.""" - if bi is None: - if node is None: - node = self - bi = node.get_stored_info() - new = self.get_binfo() - return new == bi - - def current(self, calc=None): - self.binfo = self.gen_binfo(calc) - return self._cur2() - def _cur2(self): + def is_up_to_date(self): + T = 0 + if T: Trace('is_up_to_date(%s):' % self) if not self.exists(): + if T: Trace(' not self.exists():') # The file doesn't exist locally... r = self.rfile() if r != self: # ...but there is one in a Repository... - if self.is_up_to_date(r): + if not self.changed(r): + if T: Trace(' changed(%s):' % r) # ...and it's even up-to-date... if self._local: # ...and they'd like a local copy. LocalCopy(self, r, None) - self.store_info(self.get_binfo()) + self.store_info() + if T: Trace(' 1\n') return 1 + self.changed() + if T: Trace(' None\n') return None else: - return self.is_up_to_date() + r = self.changed() + if T: Trace(' self.exists(): %s\n' % r) + return not r memoizer_counters.append(SCons.Memoize.CountValue('rfile')) @@ -2258,19 +2547,49 @@ class File(Base): def rstr(self): return str(self.rfile()) + def get_cachedir_csig(self): + """ + Fetch a Node's content signature for purposes of computing + another Node's cachesig. + + This is a wrapper around the normal get_csig() method that handles + the somewhat obscure case of using CacheDir with the -n option. + Any files that don't exist would normally be "built" by fetching + them from the cache, but the normal get_csig() method will try + to open up the local file, which doesn't exist because the -n + option meant we didn't actually pull the file from cachedir. + But since the file *does* actually exist in the cachedir, we + can use its contents for the csig. + """ + try: + return self.cachedir_csig + except AttributeError: + pass + + cachedir, cachefile = self.get_build_env().get_CacheDir().cachepath(self) + if not self.exists() and cachefile and os.path.exists(cachefile): + contents = open(cachefile, 'rb').read() + self.cachedir_csig = SCons.Util.MD5signature(contents) + else: + self.cachedir_csig = self.get_csig() + return self.cachedir_csig + def get_cachedir_bsig(self): - import SCons.Sig.MD5 - ninfo = self.get_binfo().ninfo - if not hasattr(ninfo, 'bsig'): - import SCons.Errors - raise SCons.Errors.InternalError, "cachepath(%s) found no bsig" % self.path - elif ninfo.bsig is None: - import SCons.Errors - raise SCons.Errors.InternalError, "cachepath(%s) found a bsig of None" % self.path + try: + return self.cachesig + except AttributeError: + pass + # Add the path to the cache signature, because multiple # targets built by the same action will all have the same # build signature, and we have to differentiate them somehow. - return SCons.Sig.MD5.collect([ninfo.bsig, self.path]) + children = self.children() + sigs = map(lambda n: n.get_cachedir_csig(), children) + executor = self.get_executor() + sigs.append(SCons.Util.MD5signature(executor.get_contents())) + sigs.append(self.path) + self.cachesig = SCons.Util.MD5collect(sigs) + return self.cachesig default_fs = None diff --git a/src/engine/SCons/Node/FSTests.py b/src/engine/SCons/Node/FSTests.py index ed8d6ec..225226d 100644 --- a/src/engine/SCons/Node/FSTests.py +++ b/src/engine/SCons/Node/FSTests.py @@ -35,6 +35,7 @@ import stat import SCons.Errors import SCons.Node.FS +import SCons.Util import SCons.Warnings built_it = None @@ -80,15 +81,21 @@ class Environment: pass class Action: - def __call__(self, targets, sources, env, errfunc, **kw): + def __call__(self, targets, sources, env, **kw): global built_it if kw.get('execute', 1): built_it = 1 return 0 def show(self, string): pass + def get_contents(self, target, source, env): + return "" + def genstring(self, target, source, env): + return "" def strfunction(self, targets, sources, env): return "" + def get_implicit_deps(self, target, source, env): + return [] class Builder: def __init__(self, factory, action=Action()): @@ -296,7 +303,7 @@ class BuildDirTestCase(unittest.TestCase): class MkdirAction(Action): def __init__(self, dir_made): self.dir_made = dir_made - def __call__(self, target, source, env, errfunc): + def __call__(self, target, source, env): self.dir_made.extend(target) save_Link = SCons.Node.FS.Link @@ -697,49 +704,38 @@ class FileNodeInfoTestCase(_tempdirTestCase): """Test FileNodeInfo initialization""" fff = self.fs.File('fff') ni = SCons.Node.FS.FileNodeInfo(fff) - assert hasattr(ni, 'timestamp') - assert hasattr(ni, 'size') - - def test___cmp__(self): - """Test comparing File.NodeInfo objects""" - f1 = self.fs.File('f1') - f2 = self.fs.File('f2') + assert isinstance(ni, SCons.Node.FS.FileNodeInfo) - ni1 = SCons.Node.FS.FileNodeInfo(f1) - ni2 = SCons.Node.FS.FileNodeInfo(f2) + def test_update(self): + """Test updating a File.NodeInfo with on-disk information""" + test = self.test + fff = self.fs.File('fff') - msg = "cmp(%s, %s) returned %s, not %s" + ni = SCons.Node.FS.FileNodeInfo(fff) - c = cmp(ni1, ni2) - assert c == 1, msg % (ni1, ni2, c, 1) + test.write('fff', "fff\n") - ni1.bsig = 777 - c = cmp(ni1, ni2) - assert c == 1, msg % (ni1.bsig, ni2, c, 1) + st = os.stat('fff') - ni2.bsig = 666 - c = cmp(ni1, ni2) - assert c == 1, msg % (ni1.bsig, ni2.bsig, c, 1) + ni.update(fff) - ni2.bsig = 777 - c = cmp(ni1, ni2) - assert c == 0, msg % (ni1.bsig, ni2.bsig, c, 0) + assert hasattr(ni, 'timestamp') + assert hasattr(ni, 'size') - ni2.bsig = 888 - c = cmp(ni1, ni2) - assert c == -1, msg % (ni1.bsig, ni2.bsig, c, -1) + ni.timestamp = 0 + ni.size = 0 - def test_update(self): - """Test updating a File.NodeInfo with on-disk information""" - test = self.test - fff = self.fs.File('fff') + ni.update(fff) - ni = SCons.Node.FS.FileNodeInfo(fff) + mtime = st[stat.ST_MTIME] + assert ni.timestamp == mtime, (ni.timestamp, mtime) + size = st[stat.ST_SIZE] + assert ni.size == size, (ni.size, size) import time time.sleep(2) - test.write('fff', "fff\n") + test.write('fff', "fff longer size, different time stamp\n") st = os.stat('fff') @@ -748,22 +744,22 @@ class FileNodeInfoTestCase(_tempdirTestCase): size = st[stat.ST_SIZE] assert ni.size != size, (ni.size, size) - fff.clear() - ni.update(fff) + #fff.clear() + #ni.update(fff) - st = os.stat('fff') + #st = os.stat('fff') - mtime = st[stat.ST_MTIME] - assert ni.timestamp == mtime, (ni.timestamp, mtime) - size = st[stat.ST_SIZE] - assert ni.size == size, (ni.size, size) + #mtime = st[stat.ST_MTIME] + #assert ni.timestamp == mtime, (ni.timestamp, mtime) + #size = st[stat.ST_SIZE] + #assert ni.size == size, (ni.size, size) class FileBuildInfoTestCase(_tempdirTestCase): def test___init__(self): """Test File.BuildInfo initialization""" fff = self.fs.File('fff') - bi = SCons.Node.FS.BuildInfo(fff) - assert bi.node is fff, bi.node + bi = SCons.Node.FS.FileBuildInfo(fff) + assert bi, bi def test_convert_to_sconsign(self): """Test converting to .sconsign file format""" @@ -786,14 +782,14 @@ class FileBuildInfoTestCase(_tempdirTestCase): def test_format(self): """Test the format() method""" f1 = self.fs.File('f1') - bi1 = SCons.Node.FS.BuildInfo(f1) + bi1 = SCons.Node.FS.FileBuildInfo(f1) s1sig = SCons.Node.FS.FileNodeInfo(self.fs.File('n1')) - s1sig.a = 1 + s1sig.csig = 1 d1sig = SCons.Node.FS.FileNodeInfo(self.fs.File('n2')) - d1sig.a = 2 + d1sig.timestamp = 2 i1sig = SCons.Node.FS.FileNodeInfo(self.fs.File('n3')) - i1sig.a = 3 + i1sig.size = 3 bi1.bsources = [self.fs.File('s1')] bi1.bdepends = [self.fs.File('d1')] @@ -801,17 +797,19 @@ class FileBuildInfoTestCase(_tempdirTestCase): bi1.bsourcesigs = [s1sig] bi1.bdependsigs = [d1sig] bi1.bimplicitsigs = [i1sig] + bi1.bact = 'action' + bi1.bactsig = 'actionsig' expect_lines = [ - 'None 0', - 's1: 1 None 0', - 'd1: 2 None 0', - 'i1: 3 None 0', + 's1: 1 None None', + 'd1: None 2 None', + 'i1: None None 3', + 'actionsig [action]', ] expect = string.join(expect_lines, '\n') format = bi1.format() - assert format == expect, (repr(format), repr(expect)) + assert format == expect, (repr(expect), repr(format)) class FSTestCase(_tempdirTestCase): def test_runTest(self): @@ -890,7 +888,7 @@ class FSTestCase(_tempdirTestCase): except TypeError: pass else: - assert 0 + raise Exception, "did not catch expected TypeError" assert x1.Entry(x4) == x4 try: @@ -898,7 +896,7 @@ class FSTestCase(_tempdirTestCase): except TypeError: pass else: - assert 0 + raise Exception, "did not catch expected TypeError" x6 = x1.File(x6) assert isinstance(x6, SCons.Node.FS.File) @@ -989,16 +987,26 @@ class FSTestCase(_tempdirTestCase): except: raise - # Test that just specifying the drive works to identify - # its root directory. - p = os.path.abspath(test.workpath('root_file')) + # Test that just specifying the drive works to identify + # its root directory. + p = os.path.abspath(test.workpath('root_file')) + drive, path = os.path.splitdrive(p) + if drive: + # The assert below probably isn't correct for the general + # case, but it works for Windows, which covers a lot + # of ground... + dir = fs.Dir(drive) + assert str(dir) == drive + os.sep, str(dir) + + # Make sure that lookups with and without the drive are + # equivalent. + p = os.path.abspath(test.workpath('some/file')) drive, path = os.path.splitdrive(p) - if drive: - # The assert below probably isn't correct for the general - # case, but it works for Windows, which covers a lot - # of ground... - dir = fs.Dir(drive) - assert str(dir) == drive + os.sep, str(dir) + + e1 = fs.Entry(p) + e2 = fs.Entry(path) + assert e1 is e2, (e1, e2) + assert str(e1) is str(e2), (str(e1), str(e2)) # Test for a bug in 0.04 that did not like looking up # dirs with a trailing slash on Windows. @@ -1179,7 +1187,7 @@ class FSTestCase(_tempdirTestCase): except SCons.Errors.UserError: pass else: - raise TestFailed, "did not catch expected UserError" + raise Exception, "did not catch expected UserError" nonexistent(fs.Entry, 'nonexistent') nonexistent(fs.Entry, 'nonexistent/foo') @@ -1213,15 +1221,15 @@ class FSTestCase(_tempdirTestCase): f = fs.File('f_local') assert f._local == 0 - #XXX test current() for directories + #XXX test_is_up_to_date() for directories - #XXX test sconsign() for directories + #XXX test_sconsign() for directories - #XXX test set_signature() for directories + #XXX test_set_signature() for directories - #XXX test build() for directories + #XXX test_build() for directories - #XXX test root() + #XXX test_root() # test Entry.get_contents() e = fs.Entry('does_not_exist') @@ -1318,9 +1326,7 @@ class FSTestCase(_tempdirTestCase): exc_caught = 1 assert exc_caught, "Should have caught a TypeError" - # XXX test calc_signature() - - # XXX test current() + # XXX test_is_up_to_date() d = fs.Dir('dir') r = d.remove() @@ -1407,8 +1413,7 @@ class FSTestCase(_tempdirTestCase): subdir = fs.Dir('subdir') fs.chdir(subdir, change_os_dir=1) - path, dir = fs._transformPath('#build/file', subdir) - self.fs._doLookup(SCons.Node.FS.File, path, dir) + self.fs._lookup('#build/file', subdir, SCons.Node.FS.File) def test_above_root(self): """Testing looking up a path above the root directory""" @@ -1421,7 +1426,13 @@ class FSTestCase(_tempdirTestCase): above_path = apply(os.path.join, ['..']*len(dirs) + ['above']) above = d2.Dir(above_path) - def test_rel_path(self): + # Note that the rel_path() method is not used right now, but we're + # leaving it commented out and disabling the unit here because + # it would be a shame to have to recreate the logic (or remember + # that it's buried in a long-past code checkin) if we ever need to + # resurrect it. + + def DO_NOT_test_rel_path(self): """Test the rel_path() method""" test = self.test fs = self.fs @@ -1570,6 +1581,27 @@ class DirTestCase(_tempdirTestCase): os.path.join('ddd', 'f2'), os.path.join('ddd', 'f3')], kids + def test_implicit_re_scans(self): + """Test that adding entries causes a directory to be re-scanned + """ + + fs = self.fs + + dir = fs.Dir('ddd') + + fs.File(os.path.join('ddd', 'f1')) + dir.scan() + kids = map(lambda x: x.path, dir.children()) + kids.sort() + assert kids == [os.path.join('ddd', 'f1')], kids + + fs.File(os.path.join('ddd', 'f2')) + dir.scan() + kids = map(lambda x: x.path, dir.children()) + kids.sort() + assert kids == [os.path.join('ddd', 'f1'), + os.path.join('ddd', 'f2')], kids + def test_entry_exists_on_disk(self): """Test the Dir.entry_exists_on_disk() method """ @@ -1696,15 +1728,11 @@ class DirTestCase(_tempdirTestCase): derived_f = src0.File('derived-f') derived_f.is_derived = return_true - pseudo_f = src0.File('pseudo-f') - pseudo_f.is_pseudo_derived = return_true exists_f = src0.File('exists-f') exists_f.exists = return_true derived_e = src0.Entry('derived-e') derived_e.is_derived = return_true - pseudo_e = src0.Entry('pseudo-e') - pseudo_e.is_pseudo_derived = return_true exists_e = src0.Entry('exists-e') exists_e.exists = return_true @@ -1719,8 +1747,6 @@ class DirTestCase(_tempdirTestCase): n = src0.srcdir_find_file('derived-f') check(n, ['src0/derived-f', 'src0']) - n = src0.srcdir_find_file('pseudo-f') - check(n, ['src0/pseudo-f', 'src0']) n = src0.srcdir_find_file('exists-f') check(n, ['src0/exists-f', 'src0']) n = src0.srcdir_find_file('on-disk-f1') @@ -1728,8 +1754,6 @@ class DirTestCase(_tempdirTestCase): n = src0.srcdir_find_file('derived-e') check(n, ['src0/derived-e', 'src0']) - n = src0.srcdir_find_file('pseudo-e') - check(n, ['src0/pseudo-e', 'src0']) n = src0.srcdir_find_file('exists-e') check(n, ['src0/exists-e', 'src0']) n = src0.srcdir_find_file('on-disk-e1') @@ -1741,8 +1765,6 @@ class DirTestCase(_tempdirTestCase): n = bld0.srcdir_find_file('derived-f') check(n, ['src0/derived-f', 'bld0']) - n = bld0.srcdir_find_file('pseudo-f') - check(n, ['src0/pseudo-f', 'bld0']) n = bld0.srcdir_find_file('exists-f') check(n, ['src0/exists-f', 'bld0']) n = bld0.srcdir_find_file('on-disk-f2') @@ -1750,8 +1772,6 @@ class DirTestCase(_tempdirTestCase): n = bld0.srcdir_find_file('derived-e') check(n, ['src0/derived-e', 'bld0']) - n = bld0.srcdir_find_file('pseudo-e') - check(n, ['src0/pseudo-e', 'bld0']) n = bld0.srcdir_find_file('exists-e') check(n, ['src0/exists-e', 'bld0']) n = bld0.srcdir_find_file('on-disk-e2') @@ -1769,15 +1789,11 @@ class DirTestCase(_tempdirTestCase): derived_f = src1.File('derived-f') derived_f.is_derived = return_true - pseudo_f = src1.File('pseudo-f') - pseudo_f.is_pseudo_derived = return_true exists_f = src1.File('exists-f') exists_f.exists = return_true derived_e = src1.Entry('derived-e') derived_e.is_derived = return_true - pseudo_e = src1.Entry('pseudo-e') - pseudo_e.is_pseudo_derived = return_true exists_e = src1.Entry('exists-e') exists_e.exists = return_true @@ -1787,8 +1803,6 @@ class DirTestCase(_tempdirTestCase): n = src1.srcdir_find_file('derived-f') check(n, ['src1/derived-f', 'src1']) - n = src1.srcdir_find_file('pseudo-f') - check(n, ['src1/pseudo-f', 'src1']) n = src1.srcdir_find_file('exists-f') check(n, ['src1/exists-f', 'src1']) n = src1.srcdir_find_file('on-disk-f1') @@ -1796,8 +1810,6 @@ class DirTestCase(_tempdirTestCase): n = src1.srcdir_find_file('derived-e') check(n, ['src1/derived-e', 'src1']) - n = src1.srcdir_find_file('pseudo-e') - check(n, ['src1/pseudo-e', 'src1']) n = src1.srcdir_find_file('exists-e') check(n, ['src1/exists-e', 'src1']) n = src1.srcdir_find_file('on-disk-e1') @@ -1809,8 +1821,6 @@ class DirTestCase(_tempdirTestCase): n = bld1.srcdir_find_file('derived-f') check(n, ['bld1/derived-f', 'src1']) - n = bld1.srcdir_find_file('pseudo-f') - check(n, ['bld1/pseudo-f', 'src1']) n = bld1.srcdir_find_file('exists-f') check(n, ['bld1/exists-f', 'src1']) n = bld1.srcdir_find_file('on-disk-f2') @@ -1818,8 +1828,6 @@ class DirTestCase(_tempdirTestCase): n = bld1.srcdir_find_file('derived-e') check(n, ['bld1/derived-e', 'src1']) - n = bld1.srcdir_find_file('pseudo-e') - check(n, ['bld1/pseudo-e', 'src1']) n = bld1.srcdir_find_file('exists-e') check(n, ['bld1/exists-e', 'src1']) n = bld1.srcdir_find_file('on-disk-e2') @@ -1917,19 +1925,6 @@ class EntryTestCase(_tempdirTestCase): test.subdir('e5d') test.write('e5f', "e5f\n") - e5f = fs.Entry('e5f') - sig = e5f.calc_signature(MyCalc(666)) - assert e5f.__class__ is SCons.Node.FS.File, e5f.__class__ - # This node has no builder, so it just calculates the - # signature once: the source content signature. - assert sig == 888, sig - - e5n = fs.Entry('e5n') - sig = e5n.calc_signature(MyCalc(777)) - assert e5n.__class__ is SCons.Node.FS.File, e5n.__class__ - # Doesn't exist, no sources, and no builder: no sig - assert sig is None, sig - def test_Entry_Entry_lookup(self): """Test looking up an Entry within another Entry""" self.fs.Entry('#topdir') @@ -2299,9 +2294,7 @@ class RepositoryTestCase(_tempdirTestCase): finally: test.unlink(["rep3", "contents"]) - #def test calc_signature(self): - - #def test current(self): + #def test_is_up_to_date(self): @@ -2402,8 +2395,7 @@ class stored_infoTestCase(unittest.TestCase): d = fs.Dir('sub') f = fs.File('file1', d) bi = f.get_stored_info() - assert bi.ninfo.timestamp == 0, bi.ninfo.timestamp - assert bi.ninfo.size == None, bi.ninfo.size + assert hasattr(bi, 'ninfo') class MySConsign: class Null: @@ -2509,7 +2501,7 @@ class prepareTestCase(unittest.TestCase): class MkdirAction(Action): def __init__(self, dir_made): self.dir_made = dir_made - def __call__(self, target, source, env, errfunc): + def __call__(self, target, source, env): self.dir_made.extend(target) dir_made = [] @@ -2529,6 +2521,8 @@ class prepareTestCase(unittest.TestCase): dir = fs.Dir("dir") dir.prepare() + + class SConstruct_dirTestCase(unittest.TestCase): def runTest(self): """Test setting the SConstruct directory""" @@ -2537,6 +2531,19 @@ class SConstruct_dirTestCase(unittest.TestCase): fs.set_SConstruct_dir(fs.Dir('xxx')) assert fs.SConstruct_dir.path == 'xxx' + + +class CacheDirTestCase(unittest.TestCase): + + def test_get_cachedir_csig(self): + fs = SCons.Node.FS.FS() + + f9 = fs.File('f9') + r = f9.get_cachedir_csig() + assert r == 'd41d8cd98f00b204e9800998ecf8427e', r + + + class clearTestCase(unittest.TestCase): def runTest(self): """Test clearing FS nodes of cached data.""" @@ -2584,6 +2591,8 @@ class clearTestCase(unittest.TestCase): assert not f.rexists() assert str(f) == test.workpath('f'), str(f) + + class disambiguateTestCase(unittest.TestCase): def runTest(self): """Test calling the disambiguate() method.""" @@ -2659,6 +2668,8 @@ class postprocessTestCase(unittest.TestCase): f = fs.File('f') f.postprocess() + + class SpecialAttrTestCase(unittest.TestCase): def runTest(self): """Test special attributes of file nodes.""" @@ -2744,7 +2755,7 @@ class SpecialAttrTestCase(unittest.TestCase): assert s == os.path.normpath('baz/bar/baz.blat'), s assert f.srcpath.is_literal(), f.srcpath g = f.srcpath.get() - assert isinstance(g, SCons.Node.FS.Entry), g.__class__ + assert isinstance(g, SCons.Node.FS.File), g.__class__ s = str(f.srcdir) assert s == os.path.normpath('baz/bar'), s @@ -2814,6 +2825,8 @@ class SpecialAttrTestCase(unittest.TestCase): caught = 1 assert caught, "did not catch expected AttributeError" + + class SaveStringsTestCase(unittest.TestCase): def runTest(self): """Test caching string values of nodes.""" @@ -2873,6 +2886,8 @@ class SaveStringsTestCase(unittest.TestCase): expect = map(os.path.normpath, ['src/f', 'd1/f', 'd0/b', 'd1/b']) assert s == expect, 'node str() not cached: %s'%s + + if __name__ == "__main__": suite = unittest.TestSuite() suite.addTest(BuildDirTestCase()) @@ -2889,6 +2904,7 @@ if __name__ == "__main__": suite.addTest(SaveStringsTestCase()) tclasses = [ BaseTestCase, + CacheDirTestCase, DirTestCase, DirBuildInfoTestCase, DirNodeInfoTestCase, diff --git a/src/engine/SCons/Node/NodeTests.py b/src/engine/SCons/Node/NodeTests.py index 09ab5c7..fe42035 100644 --- a/src/engine/SCons/Node/NodeTests.py +++ b/src/engine/SCons/Node/NodeTests.py @@ -67,7 +67,7 @@ class MyAction(MyActionBase): def __init__(self): self.order = 0 - def __call__(self, target, source, env, errfunc): + def __call__(self, target, source, env): global built_it, built_target, built_source, built_args, built_order built_it = 1 built_target = target @@ -77,6 +77,9 @@ class MyAction(MyActionBase): self.order = built_order return 0 + def get_implicit_deps(self, target, source, env): + return [] + class MyExecutor: def __init__(self, env=None, targets=[], sources=[]): self.env = env @@ -104,9 +107,9 @@ class MyExecutor: class MyListAction(MyActionBase): def __init__(self, list): self.list = list - def __call__(self, target, source, env, errfunc): + def __call__(self, target, source, env): for A in self.list: - A(target, source, env, errfunc) + A(target, source, env) class Environment: def __init__(self, **kw): @@ -122,8 +125,6 @@ class Environment: return apply(Environment, (), d) def _update(self, dict): self._dict.update(dict) - def get_calculator(self): - return SCons.Sig.default_calc def get_factory(self, factory): return factory or MyNode def get_scanner(self, scanner_key): @@ -217,27 +218,6 @@ class Calculator: class NodeInfoBaseTestCase(unittest.TestCase): - def test_prepare_dependencies(self): - """Test that we have a prepare_dependencies() method""" - ni = SCons.Node.NodeInfoBase(SCons.Node.Node()) - ni.prepare_dependencies() - - def test___cmp__(self): - """Test comparing NodeInfoBase objects""" - ni1 = SCons.Node.NodeInfoBase(SCons.Node.Node()) - ni2 = SCons.Node.NodeInfoBase(SCons.Node.Node()) - - assert ni1 == ni2, "%s != %s" % (ni1.__dict__, ni2.__dict__) - - ni1.foo = 777 - assert ni1 != ni2, "%s == %s" % (ni1.__dict__, ni2.__dict__) - - ni2.foo = 888 - assert ni1 != ni2, "%s == %s" % (ni1.__dict__, ni2.__dict__) - - ni1.foo = 888 - assert ni1 == ni2, "%s != %s" % (ni1.__dict__, ni2.__dict__) - def test_merge(self): """Test merging NodeInfoBase attributes""" ni1 = SCons.Node.NodeInfoBase(SCons.Node.Node()) @@ -250,7 +230,8 @@ class NodeInfoBaseTestCase(unittest.TestCase): ni2.a3 = 333 ni1.merge(ni2) - assert ni1.__dict__ == {'a1':1, 'a2':222, 'a3':333}, ni1.__dict__ + expect = {'a1':1, 'a2':222, 'a3':333, '_version_id':1} + assert ni1.__dict__ == expect, ni1.__dict__ def test_update(self): """Test the update() method""" @@ -265,12 +246,12 @@ class NodeInfoBaseTestCase(unittest.TestCase): ni1.zzz = 'z' f = ni1.format() - assert f == 'x y z', f + assert f == ['1', 'x', 'y', 'z'], f ni1.field_list = ['xxx', 'zzz', 'aaa'] f = ni1.format() - assert f == 'x z None', f + assert f == ['x', 'z', 'None'], f @@ -278,38 +259,16 @@ class BuildInfoBaseTestCase(unittest.TestCase): def test___init__(self): """Test BuildInfoBase initialization""" - bi = SCons.Node.BuildInfoBase(SCons.Node.Node()) - assert hasattr(bi, 'ninfo') - - class MyNode(SCons.Node.Node): - def NodeInfo(self, node): - return 'ninfo initialization' - bi = SCons.Node.BuildInfoBase(MyNode()) - assert bi.ninfo == 'ninfo initialization', bi.ninfo - - def test___cmp__(self): - """Test comparing BuildInfoBase objects""" - bi1 = SCons.Node.BuildInfoBase(SCons.Node.Node()) - bi2 = SCons.Node.BuildInfoBase(SCons.Node.Node()) - - assert bi1 == bi2, "%s != %s" % (bi1.__dict__, bi2.__dict__) - - bi1.ninfo.foo = 777 - assert bi1 != bi2, "%s == %s" % (bi1.__dict__, bi2.__dict__) - - bi2.ninfo.foo = 888 - assert bi1 != bi2, "%s == %s" % (bi1.__dict__, bi2.__dict__) - - bi1.ninfo.foo = 888 - assert bi1 == bi2, "%s != %s" % (bi1.__dict__, bi2.__dict__) - - bi1.foo = 999 - assert bi1 == bi2, "%s != %s" % (bi1.__dict__, bi2.__dict__) + n = SCons.Node.Node() + bi = SCons.Node.BuildInfoBase(n) + assert bi def test_merge(self): """Test merging BuildInfoBase attributes""" - bi1 = SCons.Node.BuildInfoBase(SCons.Node.Node()) - bi2 = SCons.Node.BuildInfoBase(SCons.Node.Node()) + n1 = SCons.Node.Node() + bi1 = SCons.Node.BuildInfoBase(n1) + n2 = SCons.Node.Node() + bi2 = SCons.Node.BuildInfoBase(n2) bi1.a1 = 1 bi1.a2 = 2 @@ -317,19 +276,10 @@ class BuildInfoBaseTestCase(unittest.TestCase): bi2.a2 = 222 bi2.a3 = 333 - bi1.ninfo.a4 = 4 - bi1.ninfo.a5 = 5 - bi2.ninfo.a5 = 555 - bi2.ninfo.a6 = 666 - bi1.merge(bi2) assert bi1.a1 == 1, bi1.a1 assert bi1.a2 == 222, bi1.a2 assert bi1.a3 == 333, bi1.a3 - assert bi1.ninfo.a4 == 4, bi1.ninfo.a4 - assert bi1.ninfo.a5 == 555, bi1.ninfo.a5 - assert bi1.ninfo.a6 == 666, bi1.ninfo.a6 - class NodeTestCase(unittest.TestCase): @@ -469,13 +419,18 @@ class NodeTestCase(unittest.TestCase): def test_built(self): """Test the built() method""" + class SubNodeInfo(SCons.Node.NodeInfoBase): + def update(self, node): + self.updated = 1 class SubNode(SCons.Node.Node): def clear(self): self.cleared = 1 n = SubNode() + n.ninfo = SubNodeInfo(n) n.built() assert n.cleared, n.cleared + assert n.ninfo.updated, n.ninfo.cleared def test_retrieve_from_cache(self): """Test the base retrieve_from_cache() method""" @@ -560,11 +515,11 @@ class NodeTestCase(unittest.TestCase): assert t == [], t assert m == None, m - def test_current(self): - """Test the default current() method + def test_is_up_to_date(self): + """Test the default is_up_to_date() method """ node = SCons.Node.Node() - assert node.current() is None + assert node.is_up_to_date() is None def test_children_are_up_to_date(self): """Test the children_are_up_to_date() method used by subclasses @@ -572,16 +527,14 @@ class NodeTestCase(unittest.TestCase): n1 = SCons.Node.Node() n2 = SCons.Node.Node() - calc = Calculator(111) - - n1.add_source(n2) - assert n1.children_are_up_to_date(calc), "expected up to date" + n1.add_source([n2]) + assert n1.children_are_up_to_date(), "expected up to date" n2.set_state(SCons.Node.executed) - assert not n1.children_are_up_to_date(calc), "expected not up to date" + assert not n1.children_are_up_to_date(), "expected not up to date" n2.set_state(SCons.Node.up_to_date) - assert n1.children_are_up_to_date(calc), "expected up to date" + assert n1.children_are_up_to_date(), "expected up to date" n1.always_build = 1 - assert not n1.children_are_up_to_date(calc), "expected not up to date" + assert not n1.children_are_up_to_date(), "expected not up to date" def test_env_set(self): """Test setting a Node's Environment @@ -599,23 +552,21 @@ class NodeTestCase(unittest.TestCase): a = node.builder.get_actions() assert isinstance(a[0], MyAction), a[0] - def test_get_bsig(self): - """Test generic build signature calculation + def test_get_csig(self): + """Test generic content signature calculation """ node = SCons.Node.Node() - result = node.get_bsig(Calculator(222)) - assert result == 222, result - result = node.get_bsig(Calculator(333)) - assert result == 222, result + node.get_contents = lambda: 444 + result = node.get_csig() + assert result == '550a141f12de6341fba65b0ad0433500', result - def test_get_csig(self): - """Test generic content signature calculation + def test_get_cachedir_csig(self): + """Test content signature calculation for CacheDir """ node = SCons.Node.Node() - result = node.get_csig(Calculator(444)) - assert result == 444, result - result = node.get_csig(Calculator(555)) - assert result == 444, result + node.get_contents = lambda: 555 + result = node.get_cachedir_csig() + assert result == '15de21c670ae7c3f6f3f1f37029303c9', result def test_get_binfo(self): """Test fetching/creating a build information structure @@ -625,20 +576,15 @@ class NodeTestCase(unittest.TestCase): binfo = node.get_binfo() assert isinstance(binfo, SCons.Node.BuildInfoBase), binfo - node.binfo = 777 - binfo = node.get_binfo() - assert binfo == 777, binfo - - def test_gen_binfo(self): - """Test generating a build information structure - """ node = SCons.Node.Node() d = SCons.Node.Node() + d.get_ninfo().csig = 777 i = SCons.Node.Node() + i.get_ninfo().csig = 888 node.depends = [d] node.implicit = [i] - node.gen_binfo(Calculator(666)) - binfo = node.binfo + + binfo = node.get_binfo() assert isinstance(binfo, SCons.Node.BuildInfoBase), binfo assert hasattr(binfo, 'bsources') assert hasattr(binfo, 'bsourcesigs') @@ -646,7 +592,6 @@ class NodeTestCase(unittest.TestCase): assert hasattr(binfo, 'bdependsigs') assert binfo.bimplicit == [i] assert hasattr(binfo, 'bimplicitsigs') - assert binfo.ninfo.bsig == 1998, binfo.ninfo.bsig def test_explain(self): """Test explaining why a Node must be rebuilt @@ -662,15 +607,21 @@ class NodeTestCase(unittest.TestCase): class testNode2(SCons.Node.Node): def __str__(self): return 'null_binfo' + class FS: + pass node = testNode2() + node.fs = FS() + node.fs.Top = SCons.Node.Node() result = node.explain() assert result == None, result def get_null_info(): - class Null_BInfo: - def prepare_dependencies(self): - pass - return Null_BInfo() + class Null_SConsignEntry: + class Null_BuildInfo: + def prepare_dependencies(self): + pass + binfo = Null_BuildInfo() + return Null_SConsignEntry() node.get_stored_info = get_null_info #see above: node.__str__ = lambda: 'null_binfo' @@ -690,10 +641,8 @@ class NodeTestCase(unittest.TestCase): def test_store_info(self): """Test calling the method to store build information """ - class Entry: - pass node = SCons.Node.Node() - node.store_info(Entry()) + node.store_info() def test_get_stored_info(self): """Test calling the method to fetch stored build information @@ -814,7 +763,7 @@ class NodeTestCase(unittest.TestCase): five = SCons.Node.Node() six = SCons.Node.Node() - node.add_dependency(zero) + node.add_dependency([zero]) assert node.depends == [zero] node.add_dependency([one]) assert node.depends == [zero, one] @@ -846,7 +795,7 @@ class NodeTestCase(unittest.TestCase): five = SCons.Node.Node() six = SCons.Node.Node() - node.add_source(zero) + node.add_source([zero]) assert node.sources == [zero] node.add_source([one]) assert node.sources == [zero, one] @@ -877,7 +826,7 @@ class NodeTestCase(unittest.TestCase): five = SCons.Node.Node() six = SCons.Node.Node() - node.add_ignore(zero) + node.add_ignore([zero]) assert node.ignore == [zero] node.add_ignore([one]) assert node.ignore == [zero, one] @@ -1037,21 +986,11 @@ class NodeTestCase(unittest.TestCase): # if the stored dependencies need recalculation. class StoredNode(MyNode): def get_stored_implicit(self): - return ['implicit1', 'implicit2'] - - class NotCurrent: - def current(self, node, sig): - return None - def bsig(self, node): - return 0 - - import SCons.Sig + return [MyNode('implicit1'), MyNode('implicit2')] - save_default_calc = SCons.Sig.default_calc save_implicit_cache = SCons.Node.implicit_cache save_implicit_deps_changed = SCons.Node.implicit_deps_changed save_implicit_deps_unchanged = SCons.Node.implicit_deps_unchanged - SCons.Sig.default_calc = NotCurrent() SCons.Node.implicit_cache = 1 SCons.Node.implicit_deps_changed = None SCons.Node.implicit_deps_unchanged = None @@ -1066,7 +1005,6 @@ class NodeTestCase(unittest.TestCase): assert sn.children() == [], sn.children() finally: - SCons.Sig.default_calc = save_default_calc SCons.Node.implicit_cache = save_implicit_cache SCons.Node.implicit_deps_changed = save_implicit_deps_changed SCons.Node.implicit_deps_unchanged = save_implicit_deps_unchanged @@ -1278,10 +1216,8 @@ class NodeTestCase(unittest.TestCase): n.clear() - assert not hasattr(n, 'binfo'), n.bsig assert n.includes is None, n.includes assert n.found_includes == {}, n.found_includes - assert n.implicit is None, n.implicit assert x.cleaned_up def test_get_subst_proxy(self): diff --git a/src/engine/SCons/Node/Python.py b/src/engine/SCons/Node/Python.py index a639aee..7cdea14 100644 --- a/src/engine/SCons/Node/Python.py +++ b/src/engine/SCons/Node/Python.py @@ -32,10 +32,15 @@ __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" import SCons.Node class ValueNodeInfo(SCons.Node.NodeInfoBase): - pass + current_version_id = 1 + + field_list = ['csig'] + + def str_to_node(self, s): + return Value(s) class ValueBuildInfo(SCons.Node.BuildInfoBase): - pass + current_version_id = 1 class Value(SCons.Node.Node): """A class for Python variables, typically passed on the command line @@ -54,11 +59,14 @@ class Value(SCons.Node.Node): def __str__(self): return repr(self.value) + def make_ready(self): + self.get_csig() + def build(self, **kw): if not hasattr(self, 'built_value'): apply (SCons.Node.Node.build, (self,), kw) - current = SCons.Node.Node.children_are_up_to_date + is_up_to_date = SCons.Node.Node.children_are_up_to_date def is_under(self, dir): # Make Value nodes get built regardless of @@ -88,17 +96,21 @@ class Value(SCons.Node.Node): contents = contents + kid.get_contents() return contents + def changed_since_last_build(self, target, prev_ni): + cur_csig = self.get_csig() + try: + return cur_csig != prev_ni.csig + except AttributeError: + return 1 + def get_csig(self, calc=None): """Because we're a Python value node and don't have a real timestamp, we get to ignore the calculator and just use the value contents.""" try: - binfo = self.binfo - except AttributeError: - binfo = self.binfo = self.new_binfo() - try: - return binfo.ninfo.csig + return self.ninfo.csig except AttributeError: - binfo.ninfo.csig = self.get_contents() - self.store_info(binfo) - return binfo.ninfo.csig + pass + contents = self.get_contents() + self.get_ninfo().csig = contents + return contents diff --git a/src/engine/SCons/Node/PythonTests.py b/src/engine/SCons/Node/PythonTests.py index 62bcf8b..1827a30 100644 --- a/src/engine/SCons/Node/PythonTests.py +++ b/src/engine/SCons/Node/PythonTests.py @@ -52,7 +52,7 @@ class ValueTestCase(unittest.TestCase): """Test "building" a Value Node """ class fake_executor: - def __call__(self, node, exitstatfunc): + def __call__(self, node): node.write('faked') v1 = SCons.Node.Python.Value('b', 'built') diff --git a/src/engine/SCons/Node/__init__.py b/src/engine/SCons/Node/__init__.py index f550b5b..7ddca37 100644 --- a/src/engine/SCons/Node/__init__.py +++ b/src/engine/SCons/Node/__init__.py @@ -53,9 +53,13 @@ import UserList from SCons.Debug import logInstanceCreation import SCons.Executor import SCons.Memoize -import SCons.SConsign import SCons.Util +from SCons.Debug import Trace + +def classname(obj): + return string.split(str(obj.__class__), '.')[-1] + # Node states # # These are in "priority" order, so that the maximum value for any @@ -103,38 +107,53 @@ class NodeInfoBase: Node subclasses should subclass NodeInfoBase to provide their own logic for dealing with their own Node-specific signature information. """ + current_version_id = 1 def __init__(self, node): - """A null initializer so that subclasses have a superclass - initialization method to call for future use. - """ - pass - def __cmp__(self, other): - return cmp(self.__dict__, other.__dict__) + # Create an object attribute from the class attribute so it ends up + # in the pickled data in the .sconsign file. + self._version_id = self.current_version_id def update(self, node): - pass - def merge(self, other): - for key, val in other.__dict__.items(): - self.__dict__[key] = val - def prepare_dependencies(self): - pass - def format(self): try: field_list = self.field_list except AttributeError: - field_list = self.__dict__.keys() - field_list.sort() + return + for f in field_list: + try: + delattr(self, f) + except AttributeError: + pass + try: + func = getattr(node, 'get_' + f) + except AttributeError: + pass + else: + setattr(self, f, func()) + def convert(self, node, val): + pass + def merge(self, other): + self.__dict__.update(other.__dict__) + def format(self, field_list=None, names=0): + if field_list is None: + try: + field_list = self.field_list + except AttributeError: + field_list = self.__dict__.keys() + field_list.sort() fields = [] for field in field_list: try: f = getattr(self, field) except AttributeError: f = None - fields.append(str(f)) - return string.join(fields, " ") + f = str(f) + if names: + f = field + ': ' + f + fields.append(f) + return fields class BuildInfoBase: """ - The generic base clasee for build information for a Node. + The generic base class for build information for a Node. This is what gets stored in a .sconsign file for each target file. It contains a NodeInfo instance for this node (signature information @@ -142,22 +161,17 @@ class BuildInfoBase: generic build stuff we have to track: sources, explicit dependencies, implicit dependencies, and action information. """ + current_version_id = 1 def __init__(self, node): - self.ninfo = node.NodeInfo(node) + # Create an object attribute from the class attribute so it ends up + # in the pickled data in the .sconsign file. + self._version_id = self.current_version_id self.bsourcesigs = [] self.bdependsigs = [] self.bimplicitsigs = [] self.bactsig = None - def __cmp__(self, other): - return cmp(self.ninfo, other.ninfo) def merge(self, other): - for key, val in other.__dict__.items(): - try: - merge = self.__dict__[key].merge - except (AttributeError, KeyError): - self.__dict__[key] = val - else: - merge(val) + self.__dict__.update(other.__dict__) class Node: """The base Node class, for entities that we know how to @@ -225,10 +239,18 @@ class Node: def get_suffix(self): return '' + memoizer_counters.append(SCons.Memoize.CountValue('get_build_env')) + def get_build_env(self): """Fetch the appropriate Environment to build this node. """ - return self.get_executor().get_build_env() + try: + return self._memo['get_build_env'] + except KeyError: + pass + result = self.get_executor().get_build_env() + self._memo['get_build_env'] = result + return result def get_build_scanner_path(self, scanner): """Fetch the appropriate scanner path for this node.""" @@ -286,19 +308,64 @@ class Node: """ return 0 + # + # Taskmaster interface subsystem + # + + def make_ready(self): + """Get a Node ready for evaluation. + + This is called before the Taskmaster decides if the Node is + up-to-date or not. Overriding this method allows for a Node + subclass to be disambiguated if necessary, or for an implicit + source builder to be attached. + """ + pass + + def prepare(self): + """Prepare for this Node to be built. + + This is called after the Taskmaster has decided that the Node + is out-of-date and must be rebuilt, but before actually calling + the method to build the Node. + + This default implemenation checks that all children either exist + or are derived, and initializes the BuildInfo structure that + will hold the information about how this node is, uh, built. + + Overriding this method allows for for a Node subclass to remove + the underlying file from the file system. Note that subclass + methods should call this base class method to get the child + check and the BuildInfo structure. + """ + l = self.depends + if not self.implicit is None: + l = l + self.implicit + missing_sources = self.get_executor().get_missing_sources() \ + + filter(lambda c: c.missing(), l) + if missing_sources: + desc = "Source `%s' not found, needed by target `%s'." % (missing_sources[0], self) + raise SCons.Errors.StopError, desc + + self.binfo = self.get_binfo() + def build(self, **kw): """Actually build the node. + This is called by the Taskmaster after it's decided that the + Node is out-of-date and must be rebuilt, and after the prepare() + method has gotten everything, uh, prepared. + This method is called from multiple threads in a parallel build, - so only do thread safe stuff here. Do thread unsafe stuff in - built(). + so only do thread safe stuff here. Do thread unsafe stuff + in built(). + """ - def exitstatfunc(stat, node=self): - if stat: - msg = "Error %d" % stat - raise SCons.Errors.BuildError(node=node, errstr=msg) executor = self.get_executor() - apply(executor, (self, exitstatfunc), kw) + stat = apply(executor, (self,), kw) + if stat: + msg = "Error %d" % stat + raise SCons.Errors.BuildError(node=self, errstr=msg) def built(self): """Called just after this node is successfully built.""" @@ -309,28 +376,26 @@ class Node: parent.implicit = None parent.del_binfo() + self.clear() + + self.ninfo.update(self) + + def visited(self): + """Called just after this node has been visited (with or + without a build).""" try: - new = self.binfo + binfo = self.binfo except AttributeError: - # Node arrived here without build info; apparently it - # doesn't need it, so don't bother calculating or storing - # it. - new = None - - # Reset this Node's cached state since it was just built and - # various state has changed. - self.clear() + # Apparently this node doesn't need build info, so + # don't bother calculating or storing it. + pass + else: + self.ninfo.update(self) + self.store_info() - if new: - # It had build info, so it should be stored in the signature - # cache. However, if the build info included a content - # signature then it must be recalculated before being stored. - if hasattr(new.ninfo, 'csig'): - self.get_csig() - else: - new.ninfo.update(self) - self.binfo = new - self.store_info(self.binfo) + # + # + # def add_to_waiting_s_e(self, node): self.waiting_s_e[node] = 1 @@ -367,27 +432,33 @@ class Node: can be re-evaluated by interfaces that do continuous integration builds). """ + # Note in case it's important in the future: We also used to clear + # the build information (the lists of dependencies) here like this: + # + # self.del_binfo() + # + # But we now rely on the fact that we're going to look at that + # once before the build, and then store the results in the + # .sconsign file after the build. self.clear_memoized_values() + self.ninfo = self.new_ninfo() self.executor_cleanup() - self.del_binfo() try: delattr(self, '_calculated_sig') except AttributeError: pass self.includes = None self.found_includes = {} - self.implicit = None def clear_memoized_values(self): self._memo = {} - def visited(self): - """Called just after this node has been visited - without requiring a build..""" - pass - def builder_set(self, builder): self.builder = builder + try: + del self.executor + except AttributeError: + pass def has_builder(self): """Return whether this Node has a builder or not. @@ -405,8 +476,7 @@ class Node: except AttributeError: # There was no explicit builder for this Node, so initialize # the self.builder attribute to None now. - self.builder = None - b = self.builder + b = self.builder = None return not b is None def set_explicit(self, is_explicit): @@ -446,14 +516,6 @@ class Node: """ return self.has_builder() or self.side_effect - def is_pseudo_derived(self): - """ - Returns true iff this node is built, but should use a source path - when duplicate=0 and should contribute a content signature (i.e. - source signature) when used as a source for other derived files. - """ - return 0 - def alter_targets(self): """Return a list of alternate targets for this Node. """ @@ -557,28 +619,15 @@ class Node: if implicit_cache and not implicit_deps_changed: implicit = self.get_stored_implicit() if implicit is not None: - factory = build_env.get_factory(self.builder.source_factory) - nodes = [] - for i in implicit: - try: - n = factory(i) - except TypeError: - # The implicit dependency was cached as one type - # of Node last time, but the configuration has - # changed (probably) and it's a different type - # this time. Just ignore the mismatch and go - # with what our current configuration says the - # Node is. - pass - else: - nodes.append(n) - self._add_child(self.implicit, self.implicit_dict, nodes) - calc = build_env.get_calculator() - if implicit_deps_unchanged or self.current(calc): + # We now add the implicit dependencies returned from the + # stored .sconsign entry to have already been converted + # to Nodes for us. (We used to run them through a + # source_factory function here.) + self._add_child(self.implicit, self.implicit_dict, implicit) + if implicit_deps_unchanged or self.is_up_to_date(): return - # one of this node's sources has changed, so - # we need to recalculate the implicit deps, - # and the bsig: + # one of this node's sources has changed, + # so we must recalculate the implicit deps: self.implicit = [] self.implicit_dict = {} self._children_reset() @@ -620,65 +669,24 @@ class Node: NodeInfo = NodeInfoBase BuildInfo = BuildInfoBase - def calculator(self): - import SCons.Defaults - - env = self.env or SCons.Defaults.DefaultEnvironment() - return env.get_calculator() - - memoizer_counters.append(SCons.Memoize.CountValue('calc_signature')) - - def calc_signature(self, calc=None): - """ - Select and calculate the appropriate build signature for a node. - - self - the node - calc - the signature calculation module - returns - the signature - """ - try: - return self._memo['calc_signature'] - except KeyError: - pass - if self.is_derived(): - import SCons.Defaults - - env = self.env or SCons.Defaults.DefaultEnvironment() - if env.use_build_signature(): - result = self.get_bsig(calc) - else: - result = self.get_csig(calc) - elif not self.rexists(): - result = None - else: - result = self.get_csig(calc) - self._memo['calc_signature'] = result - return result - def new_ninfo(self): - return self.NodeInfo(self) - - def new_binfo(self): - return self.BuildInfo(self) + ninfo = self.NodeInfo(self) + return ninfo - def get_binfo(self): + def get_ninfo(self): try: - return self.binfo + return self.ninfo except AttributeError: - self.binfo = self.new_binfo() - return self.binfo + self.ninfo = self.new_ninfo() + return self.ninfo - def del_binfo(self): - """Delete the build info from this node.""" - try: - delattr(self, 'binfo') - except AttributeError: - pass + def new_binfo(self): + binfo = self.BuildInfo(self) + return binfo - def gen_binfo(self, calc=None, scan=1): + def get_binfo(self): """ - Generate a node's build signature, the digested signatures - of its dependency files and build information. + Fetch a node's build information. node - the node whose sources will be collected cache - alternate node to use for the signature cache @@ -689,21 +697,17 @@ class Node: already built and updated by someone else, if that's what's wanted. """ + try: + return self.binfo + except AttributeError: + pass - if calc is None: - calc = self.calculator() - - binfo = self.get_binfo() - - if scan: - self.scan() + binfo = self.new_binfo() + self.binfo = binfo executor = self.get_executor() - def calc_signature(node, calc=calc): - return node.calc_signature(calc) sources = executor.get_unignored_sources(self.ignore) - sourcesigs = executor.process_sources(calc_signature, self.ignore) depends = self.depends implicit = self.implicit or [] @@ -712,15 +716,16 @@ class Node: depends = filter(self.do_not_ignore, depends) implicit = filter(self.do_not_ignore, implicit) - dependsigs = map(calc_signature, depends) - implicitsigs = map(calc_signature, implicit) + def get_ninfo(node): + return node.get_ninfo() - sigs = sourcesigs + dependsigs + implicitsigs + sourcesigs = map(get_ninfo, sources) + dependsigs = map(get_ninfo, depends) + implicitsigs = map(get_ninfo, implicit) if self.has_builder(): binfo.bact = str(executor) - binfo.bactsig = calc.module.signature(executor) - sigs.append(binfo.bactsig) + binfo.bactsig = SCons.Util.MD5signature(executor.get_contents()) binfo.bsources = sources binfo.bdepends = depends @@ -730,33 +735,34 @@ class Node: binfo.bdependsigs = dependsigs binfo.bimplicitsigs = implicitsigs - binfo.ninfo.bsig = calc.module.collect(filter(None, sigs)) - return binfo - def get_bsig(self, calc=None): - binfo = self.get_binfo() + def del_binfo(self): + """Delete the build info from this node.""" try: - return binfo.ninfo.bsig + delattr(self, 'binfo') except AttributeError: - self.binfo = self.gen_binfo(calc) - return self.binfo.ninfo.bsig + pass - def get_csig(self, calc=None): - binfo = self.get_binfo() + def get_csig(self): try: - return binfo.ninfo.csig + return self.ninfo.csig except AttributeError: - if calc is None: - calc = self.calculator() - csig = binfo.ninfo.csig = calc.module.signature(self) - return csig + ninfo = self.get_ninfo() + ninfo.csig = SCons.Util.MD5signature(self.get_contents()) + return self.ninfo.csig + + def get_cachedir_csig(self): + return self.get_csig() - def store_info(self, obj): + def store_info(self): """Make the build signature permanent (that is, store it in the .sconsign file or equivalent).""" pass + def do_not_store_info(self): + pass + def get_stored_info(self): return None @@ -800,23 +806,8 @@ class Node: def missing(self): return not self.is_derived() and \ - not self.is_pseudo_derived() and \ not self.linked and \ not self.rexists() - - def prepare(self): - """Prepare for this Node to be created. - The default implemenation checks that all children either exist - or are derived. - """ - l = self.depends - if not self.implicit is None: - l = l + self.implicit - missing_sources = self.get_executor().get_missing_sources() \ - + filter(lambda c: c.missing(), l) - if missing_sources: - desc = "Source `%s' not found, needed by target `%s'." % (missing_sources[0], self) - raise SCons.Errors.StopError, desc def remove(self): """Remove this Node: no-op by default.""" @@ -861,11 +852,11 @@ class Node: def _add_child(self, collection, dict, child): """Adds 'child' to 'collection', first checking 'dict' to see if it's already present.""" - if type(child) is not type([]): - child = [child] - for c in child: - if not isinstance(c, Node): - raise TypeError, c + #if type(child) is not type([]): + # child = [child] + #for c in child: + # if not isinstance(c, Node): + # raise TypeError, c added = None for c in child: if not dict.has_key(c): @@ -883,7 +874,7 @@ class Node: def _children_reset(self): self.clear_memoized_values() # We need to let the Executor clear out any calculated - # bsig info that it's cached so we can re-calculate it. + # build info that it's cached so we can re-calculate it. self.executor_cleanup() def do_not_ignore(self, node): @@ -944,19 +935,107 @@ class Node: def get_state(self): return self.state - def current(self, calc=None): + def state_has_changed(self, target, prev_ni): + return (self.state != SCons.Node.up_to_date) + + def get_env(self): + env = self.env + if not env: + import SCons.Defaults + env = SCons.Defaults.DefaultEnvironment() + return env + + def changed_since_last_build(self, target, prev_ni): + """ + + Must be overridden in a specific subclass to return True if this + Node (a dependency) has changed since the last time it was used + to build the specified target. prev_ni is this Node's state (for + example, its file timestamp, length, maybe content signature) + as of the last time the target was built. + + Note that this method is called through the dependency, not the + target, because a dependency Node must be able to use its own + logic to decide if it changed. For example, File Nodes need to + obey if we're configured to use timestamps, but Python Value Nodes + never use timestamps and always use the content. If this method + were called through the target, then each Node's implementation + of this method would have to have more complicated logic to + handle all the different Node types on which it might depend. + """ + raise NotImplementedError + + def Decider(self, function): + SCons.Util.AddMethod(self, function, 'changed_since_last_build') + + def changed(self, node=None): + """ + Returns if the node is up-to-date with respect to the BuildInfo + stored last time it was built. The default behavior is to compare + it against our own previously stored BuildInfo, but the stored + BuildInfo from another Node (typically one in a Repository) + can be used instead. + + Note that we now *always* check every dependency. We used to + short-circuit the check by returning as soon as we detected + any difference, but we now rely on checking every dependency + to make sure that any necessary Node information (for example, + the content signature of an #included .h file) is updated. + """ + t = 0 + if t: Trace('changed(%s [%s], %s)' % (self, classname(self), node)) + if node is None: + node = self + + result = False + + bi = node.get_stored_info().binfo + then = bi.bsourcesigs + bi.bdependsigs + bi.bimplicitsigs + children = self.children() + + diff = len(children) - len(then) + if diff: + # The old and new dependency lists are different lengths. + # This always indicates that the Node must be rebuilt. + # We also extend the old dependency list with enough None + # entries to equal the new dependency list, for the benefit + # of the loop below that updates node information. + then.extend([None] * diff) + result = True + + for child, prev_ni in zip(children, then): + if child.changed_since_last_build(self, prev_ni): + if t: Trace(': %s changed' % child) + result = True + + contents = self.get_executor().get_contents() + if self.has_builder(): + import SCons.Util + newsig = SCons.Util.MD5signature(contents) + if bi.bactsig != newsig: + if t: Trace(': bactsig %s != newsig %s' % (bi.bactsig, newsig)) + result = True + + if not result: + if t: Trace(': up to date') + + if t: Trace('\n') + + return result + + def is_up_to_date(self): """Default check for whether the Node is current: unknown Node subtypes are always out of date, so they will always get built.""" return None - def children_are_up_to_date(self, calc=None): + def children_are_up_to_date(self): """Alternate check for whether the Node is current: If all of our children were up-to-date, then this Node was up-to-date, too. The SCons.Node.Alias and SCons.Node.Python.Value subclasses rebind their current() method to this method.""" # Allow the children to calculate their signatures. - self.binfo = self.gen_binfo(calc) + self.binfo = self.get_binfo() if self.always_build: return None state = 0 @@ -1052,6 +1131,8 @@ class Node: old = self.get_stored_info() if old is None: return None + + old = old.binfo old.prepare_dependencies() try: @@ -1087,7 +1168,7 @@ class Node: for k in new_bkids: if not k in old_bkids: lines.append("`%s' is a new dependency\n" % stringify(k)) - elif osig[k] != nsig[k]: + elif k.changed_since_last_build(self, osig[k]): lines.append("`%s' changed\n" % stringify(k)) if len(lines) == 0 and old_bkids != new_bkids: diff --git a/src/engine/SCons/Options/__init__.py b/src/engine/SCons/Options/__init__.py index ebd6052..e2ad80f 100644 --- a/src/engine/SCons/Options/__init__.py +++ b/src/engine/SCons/Options/__init__.py @@ -50,7 +50,7 @@ class Options: Holds all the options, updates the environment with the variables, and renders the help text. """ - def __init__(self, files=None, args={}, is_global=1): + def __init__(self, files=[], args={}, is_global=1): """ files - [optional] List of option configuration files to load (backward compatibility) If a single string is passed it is @@ -58,11 +58,12 @@ class Options: """ self.options = [] self.args = args - self.files = None - if SCons.Util.is_String(files): - self.files = [ files ] - elif files: - self.files = files + if not SCons.Util.is_List(files): + if files: + files = [ files ] + else: + files = [] + self.files = files # create the singleton instance if is_global: @@ -155,10 +156,9 @@ class Options: values[option.key] = option.default # next set the value specified in the options file - if self.files: - for filename in self.files: - if os.path.exists(filename): - execfile(filename, values) + for filename in self.files: + if os.path.exists(filename): + execfile(filename, values) # finally set the values specified on the command line if args is None: diff --git a/src/engine/SCons/SConf.py b/src/engine/SCons/SConf.py index 21305b9..47a552a 100644 --- a/src/engine/SCons/SConf.py +++ b/src/engine/SCons/SConf.py @@ -28,6 +28,8 @@ Autoconf-like configuration support. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" +import SCons.compat + import os import re import string @@ -46,6 +48,8 @@ import SCons.Util import SCons.Warnings import SCons.Conftest +from SCons.Debug import Trace + # Turn off the Conftest error logging SCons.Conftest.LogInputFiles = 0 SCons.Conftest.LogErrorMessages = 0 @@ -158,12 +162,10 @@ class SConfBuildInfo(SCons.Node.FS.FileBuildInfo): """ result = None # -> 0/None -> no error, != 0 error string = None # the stdout / stderr output when building the target - - def __init__(self, node, result, string, sig): - SCons.Node.FS.FileBuildInfo.__init__(self, node) + + def set_build_result(self, result, string): self.result = result self.string = string - self.ninfo.bsig = sig class Streamer: @@ -211,7 +213,7 @@ class SConfBuildTask(SCons.Taskmaster.Task): """ if not isinstance(bi, SConfBuildInfo): SCons.Warnings.warn(SConfWarning, - "The stored build information has an unexpected class.") + "The stored build information has an unexpected class: %s" % bi.__class__) else: self.display("The original builder output was:\n" + string.replace(" |" + str(bi.string), @@ -245,30 +247,39 @@ class SConfBuildTask(SCons.Taskmaster.Task): # cached_error is 1, if the node(s) are up_to_date, but the # build will fail # cachable is 0, if some nodes are not in our cache - is_up_to_date = 1 - cached_error = 0 - cachable = 1 + T = 0 + changed = False + cached_error = False + cachable = True for t in self.targets: - bi = t.get_stored_info() + if T: Trace('%s' % (t)) + bi = t.get_stored_info().binfo if isinstance(bi, SConfBuildInfo): + if T: Trace(': SConfBuildInfo') if cache_mode == CACHE: t.set_state(SCons.Node.up_to_date) + if T: Trace(': set_state(up_to-date)') else: - new_bsig = t.calc_signature(sconf_global.calc) - if t.env.use_build_signature(): - old_bsig = bi.ninfo.bsig - else: - old_bsig = bi.ninfo.csig - is_up_to_date = (is_up_to_date and new_bsig == old_bsig) + if T: Trace(': get_state() %s' % t.get_state()) + if T: Trace(': changed() %s' % t.changed()) + if (t.get_state() != SCons.Node.up_to_date and t.changed()): + changed = True + if T: Trace(': changed %s' % changed) cached_error = cached_error or bi.result else: + if T: Trace(': else') # the node hasn't been built in a SConf context or doesn't # exist - cachable = 0 - is_up_to_date = 0 - return (is_up_to_date, cached_error, cachable) + cachable = False + changed = ( t.get_state() != SCons.Node.up_to_date ) + if T: Trace(': changed %s' % changed) + if T: Trace('\n') + return (not changed, cached_error, cachable) def execute(self): + if not self.targets[0].has_builder(): + return + sconf = sconf_global is_up_to_date, cached_error, cachable = self.collect_node_states() @@ -281,11 +292,13 @@ class SConfBuildTask(SCons.Taskmaster.Task): if cached_error and is_up_to_date: self.display("Building \"%s\" failed in a previous run and all " "its sources are up to date." % str(self.targets[0])) - self.display_cached_string(self.targets[0].get_stored_info()) + binfo = self.targets[0].get_stored_info().binfo + self.display_cached_string(binfo) raise SCons.Errors.BuildError # will be 'caught' in self.failed elif is_up_to_date: self.display("\"%s\" is up to date." % str(self.targets[0])) - self.display_cached_string(self.targets[0].get_stored_info()) + binfo = self.targets[0].get_stored_info().binfo + self.display_cached_string(binfo) elif dryrun: raise ConfigureDryRunError(self.targets[0]) else: @@ -305,19 +318,41 @@ class SConfBuildTask(SCons.Taskmaster.Task): except SystemExit: exc_value = sys.exc_info()[1] raise SCons.Errors.ExplicitExit(self.targets[0],exc_value.code) - except: + except Exception, e: for t in self.targets: - sig = t.calc_signature(sconf.calc) - string = s.getvalue() - binfo = SConfBuildInfo(t,1,string,sig) - t.dir.sconsign().set_entry(t.name, binfo) - raise + binfo = t.get_binfo() + binfo.__class__ = SConfBuildInfo + binfo.set_build_result(1, s.getvalue()) + sconsign_entry = SCons.SConsign.SConsignEntry() + sconsign_entry.binfo = binfo + #sconsign_entry.ninfo = self.get_ninfo() + # We'd like to do this as follows: + # t.store_info(binfo) + # However, we need to store it as an SConfBuildInfo + # object, and store_info() will turn it into a + # regular FileNodeInfo if the target is itself a + # regular File. + sconsign = t.dir.sconsign() + sconsign.set_entry(t.name, sconsign_entry) + sconsign.merge() + raise e else: for t in self.targets: - sig = t.calc_signature(sconf.calc) - string = s.getvalue() - binfo = SConfBuildInfo(t,0,string,sig) - t.dir.sconsign().set_entry(t.name, binfo) + binfo = t.get_binfo() + binfo.__class__ = SConfBuildInfo + binfo.set_build_result(0, s.getvalue()) + sconsign_entry = SCons.SConsign.SConsignEntry() + sconsign_entry.binfo = binfo + #sconsign_entry.ninfo = self.get_ninfo() + # We'd like to do this as follows: + # t.store_info(binfo) + # However, we need to store it as an SConfBuildInfo + # object, and store_info() will turn it into a + # regular FileNodeInfo if the target is itself a + # regular File. + sconsign = t.dir.sconsign() + sconsign.set_entry(t.name, sconsign_entry) + sconsign.merge() class SConf: """This is simply a class to represent a configure context. After @@ -369,7 +404,6 @@ class SConf: self.AddTests(default_tests) self.AddTests(custom_tests) self.confdir = SConfFS.Dir(env.subst(conf_dir)) - self.calc = None if not config_h is None: config_h = SConfFS.File(config_h) self.config_h = config_h @@ -377,7 +411,8 @@ class SConf: def Finish(self): """Call this method after finished with your tests: - env = sconf.Finish()""" + env = sconf.Finish() + """ self._shutdown() return self.env @@ -398,6 +433,13 @@ class SConf: old_os_dir = os.getcwd() SConfFS.chdir(SConfFS.Top, change_os_dir=1) + # Because we take responsibility here for writing out our + # own .sconsign info (see SConfBuildTask.execute(), above), + # we override the store_info() method with a null place-holder + # so we really control how it gets written. + for n in nodes: + n.store_info = n.do_not_store_info + ret = 1 try: diff --git a/src/engine/SCons/SConfTests.py b/src/engine/SCons/SConfTests.py index d4ed346..2c35730 100644 --- a/src/engine/SCons/SConfTests.py +++ b/src/engine/SCons/SConfTests.py @@ -60,7 +60,7 @@ class SConfTestCase(unittest.TestCase): import SCons.SConsign SCons.SConsign.write() # simulate normal scons-finish for n in sys.modules.keys(): - if string.split(n, '.')[0] == 'SCons': + if string.split(n, '.')[0] == 'SCons' and n[:12] != 'SCons.compat': m = sys.modules[n] if type(m) is ModuleType: # if this is really a scons module, clear its namespace @@ -96,9 +96,9 @@ class SConfTestCase(unittest.TestCase): def checks(self, sconf, TryFuncString): TryFunc = self.SConf.SConf.__dict__[TryFuncString] - res1 = TryFunc( sconf, "int main() { return 0; }", ".c" ) + res1 = TryFunc( sconf, "int main() { return 0; }\n", ".c" ) res2 = TryFunc( sconf, - '#include "no_std_header.h"\nint main() {return 0; }', + '#include "no_std_header.h"\nint main() {return 0; }\n', '.c' ) return (res1,res2) @@ -136,8 +136,9 @@ class SConfTestCase(unittest.TestCase): sconf = self.SConf.SConf(self.scons_env, conf_dir=self.test.workpath('config.tests'), log_file=self.test.workpath('config.log')) - test_h = self.test.write( self.test.workpath('config.tests', 'no_std_header.h'), - "/* we are changing a dependency now */" ); + no_std_header_h = self.test.workpath('config.tests', 'no_std_header.h') + test_h = self.test.write( no_std_header_h, + "/* we are changing a dependency now */\n" ); try: res = checks( self, sconf, TryFunc ) log = self.test.read( self.test.workpath('config.log') ) @@ -187,7 +188,7 @@ class SConfTestCase(unittest.TestCase): pass def clear(self): pass - def current(self, calc=None): + def is_up_to_date(self): return None def prepare(self): pass @@ -199,7 +200,7 @@ class SConfTestCase(unittest.TestCase): pass def get_stored_info(self): pass - def calc_signature(self, calc): + def do_not_store_info(self): pass def get_executor(self): class Executor: @@ -236,7 +237,7 @@ int main() { } """ res1 = sconf.TryRun( prog, ".c" ) - res2 = sconf.TryRun( "not a c program", ".c" ) + res2 = sconf.TryRun( "not a c program\n", ".c" ) return (res1, res2) self._resetSConfState() @@ -275,7 +276,7 @@ int main() { """Test SConf.TryAction """ def actionOK(target, source, env): - open(str(target[0]), "w").write( "RUN OK" ) + open(str(target[0]), "w").write( "RUN OK\n" ) return None def actionFAIL(target, source, env): return 1 @@ -285,7 +286,7 @@ int main() { log_file=self.test.workpath('config.log')) try: (ret, output) = sconf.TryAction(action=actionOK) - assert ret and output == "RUN OK", (ret, output) + assert ret and output == "RUN OK" + os.linesep, (ret, output) (ret, output) = sconf.TryAction(action=actionFAIL) assert not ret and output == "", (ret, output) finally: diff --git a/src/engine/SCons/SConsign.py b/src/engine/SCons/SConsign.py index dcd6979..bd42706 100644 --- a/src/engine/SCons/SConsign.py +++ b/src/engine/SCons/SConsign.py @@ -29,12 +29,13 @@ Writing and reading information to the .sconsign file or files. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" +import SCons.compat + import cPickle import os import os.path import SCons.dblite -import SCons.Sig import SCons.Warnings def corrupt_dblite_warning(filename): @@ -107,6 +108,24 @@ def write(): else: syncmethod() +class SConsignEntry: + """ + Wrapper class for the generic entry in a .sconsign file. + The Node subclass populates it with attributes as it pleases. + + XXX As coded below, we do expect a '.binfo' attribute to be added, + but we'll probably generalize this in the next refactorings. + """ + current_version_id = 1 + def __init__(self): + # Create an object attribute from the class attribute so it ends up + # in the pickled data in the .sconsign file. + _version_id = self.current_version_id + def convert_to_sconsign(self): + self.binfo.convert_to_sconsign() + def convert_from_sconsign(self, dir, name): + self.binfo.convert_from_sconsign(dir, name) + class Base: """ This is the controlling class for the signatures for the collection of @@ -116,14 +135,10 @@ class Base: methods for fetching and storing the individual bits of information that make up signature entry. """ - def __init__(self, module=None): - """ - module - the signature module being used - """ - - self.module = module or SCons.Sig.default_calc.module + def __init__(self): self.entries = {} - self.dirty = 0 + self.dirty = False + self.to_be_merged = {} def get_entry(self, filename): """ @@ -136,19 +151,43 @@ class Base: Set the entry. """ self.entries[filename] = obj - self.dirty = 1 + self.dirty = True def do_not_set_entry(self, filename, obj): pass + def store_info(self, filename, node): + entry = node.get_stored_info() + entry.binfo.merge(node.get_binfo()) + self.to_be_merged[filename] = node + self.dirty = True + + def do_not_store_info(self, filename, node): + pass + + def merge(self): + for key, node in self.to_be_merged.items(): + entry = node.get_stored_info() + try: + ninfo = entry.ninfo + except AttributeError: + # This happens with SConf Nodes, because the configuration + # subsystem takes direct control over how the build decision + # is made and its information stored. + pass + else: + ninfo.merge(node.get_ninfo()) + self.entries[key] = entry + self.to_be_merged = {} + class DB(Base): """ A Base subclass that reads and writes signature information from a global .sconsign.db* file--the actual file suffix is - determined by the specified database module. + determined by the database module. """ - def __init__(self, dir, module=None): - Base.__init__(self, module) + def __init__(self, dir): + Base.__init__(self) self.dir = dir @@ -182,6 +221,7 @@ class DB(Base): # a file there. Don't actually set any entry info, so we # won't try to write to that .sconsign.dblite file. self.set_entry = self.do_not_set_entry + self.store_info = self.do_not_store_info global sig_files sig_files.append(self) @@ -190,6 +230,8 @@ class DB(Base): if not self.dirty: return + self.merge() + db, mode = Get_DataBase(self.dir) # Write using the path relative to the top of the SConstruct @@ -211,27 +253,31 @@ class DB(Base): syncmethod() class Dir(Base): - def __init__(self, fp=None, module=None): + def __init__(self, fp=None, dir=None): """ fp - file pointer to read entries from - module - the signature module being used """ - Base.__init__(self, module) + Base.__init__(self) + + if not fp: + return - if fp: - self.entries = cPickle.load(fp) - if type(self.entries) is not type({}): - self.entries = {} - raise TypeError + self.entries = cPickle.load(fp) + if type(self.entries) is not type({}): + self.entries = {} + raise TypeError + + if dir: + for key, entry in self.entries.items(): + entry.convert_from_sconsign(dir, key) class DirFile(Dir): """ Encapsulates reading and writing a per-directory .sconsign file. """ - def __init__(self, dir, module=None): + def __init__(self, dir): """ dir - the directory for the file - module - the signature module being used """ self.dir = dir @@ -243,7 +289,7 @@ class DirFile(Dir): fp = None try: - Dir.__init__(self, fp, module) + Dir.__init__(self, fp, dir) except KeyboardInterrupt: raise except: @@ -253,15 +299,6 @@ class DirFile(Dir): global sig_files sig_files.append(self) - def get_entry(self, filename): - """ - Fetch the specified entry attribute, converting from .sconsign - format to in-memory format. - """ - entry = Dir.get_entry(self, filename) - entry.convert_from_sconsign(self.dir, filename) - return entry - def write(self, sync=1): """ Write the .sconsign file to disk. @@ -275,48 +312,52 @@ class DirFile(Dir): to the .sconsign file. Either way, always try to remove the temporary file at the end. """ - if self.dirty: - temp = os.path.join(self.dir.path, '.scons%d' % os.getpid()) + if not self.dirty: + return + + self.merge() + + temp = os.path.join(self.dir.path, '.scons%d' % os.getpid()) + try: + file = open(temp, 'wb') + fname = temp + except IOError: try: - file = open(temp, 'wb') - fname = temp + file = open(self.sconsign, 'wb') + fname = self.sconsign except IOError: - try: - file = open(self.sconsign, 'wb') - fname = self.sconsign - except IOError: - return - for key, entry in self.entries.items(): - entry.convert_to_sconsign() - cPickle.dump(self.entries, file, 1) - file.close() - if fname != self.sconsign: - try: - mode = os.stat(self.sconsign)[0] - os.chmod(self.sconsign, 0666) - os.unlink(self.sconsign) - except (IOError, OSError): - # Try to carry on in the face of either OSError - # (things like permission issues) or IOError (disk - # or network issues). If there's a really dangerous - # issue, it should get re-raised by the calls below. - pass - try: - os.rename(fname, self.sconsign) - except OSError: - # An OSError failure to rename may indicate something - # like the directory has no write permission, but - # the .sconsign file itself might still be writable, - # so try writing on top of it directly. An IOError - # here, or in any of the following calls, would get - # raised, indicating something like a potentially - # serious disk or network issue. - open(self.sconsign, 'wb').write(open(fname, 'rb').read()) - os.chmod(self.sconsign, mode) + return + for key, entry in self.entries.items(): + entry.convert_to_sconsign() + cPickle.dump(self.entries, file, 1) + file.close() + if fname != self.sconsign: try: - os.unlink(temp) + mode = os.stat(self.sconsign)[0] + os.chmod(self.sconsign, 0666) + os.unlink(self.sconsign) except (IOError, OSError): + # Try to carry on in the face of either OSError + # (things like permission issues) or IOError (disk + # or network issues). If there's a really dangerous + # issue, it should get re-raised by the calls below. pass + try: + os.rename(fname, self.sconsign) + except OSError: + # An OSError failure to rename may indicate something + # like the directory has no write permission, but + # the .sconsign file itself might still be writable, + # so try writing on top of it directly. An IOError + # here, or in any of the following calls, would get + # raised, indicating something like a potentially + # serious disk or network issue. + open(self.sconsign, 'wb').write(open(fname, 'rb').read()) + os.chmod(self.sconsign, mode) + try: + os.unlink(temp) + except (IOError, OSError): + pass ForDirectory = DB diff --git a/src/engine/SCons/SConsignTests.py b/src/engine/SCons/SConsignTests.py index e292aed..2a04c89 100644 --- a/src/engine/SCons/SConsignTests.py +++ b/src/engine/SCons/SConsignTests.py @@ -33,30 +33,33 @@ import SCons.dblite import SCons.SConsign class BuildInfo: + def merge(self, object): + pass + +class DummySConsignEntry: def __init__(self, name): self.name = name + self.binfo = BuildInfo() def convert_to_sconsign(self): self.c_to_s = 1 def convert_from_sconsign(self, dir, name): self.c_from_s = 1 -class DummyModule: - def to_string(self, sig): - return str(sig) - - def from_string(self, sig): - return int(sig) - class FS: def __init__(self, top): self.Top = top self.Top.repositories = [] class DummyNode: - def __init__(self, path='not_a_valid_path'): + def __init__(self, path='not_a_valid_path', binfo=None): self.path = path self.tpath = path self.fs = FS(self) + self.binfo = binfo + def get_stored_info(self): + return self.binfo + def get_binfo(self): + return self.binfo class SConsignTestCase(unittest.TestCase): def setUp(self): @@ -70,17 +73,19 @@ class SConsignTestCase(unittest.TestCase): class BaseTestCase(SConsignTestCase): - def runTest(self): - aaa = BuildInfo('aaa') - bbb = BuildInfo('bbb') + def test_Base(self): + aaa = DummySConsignEntry('aaa') + bbb = DummySConsignEntry('bbb') bbb.arg1 = 'bbb arg1' - ccc = BuildInfo('ccc') + ccc = DummySConsignEntry('ccc') ccc.arg2 = 'ccc arg2' f = SCons.SConsign.Base() f.set_entry('aaa', aaa) f.set_entry('bbb', bbb) + #f.merge() + e = f.get_entry('aaa') assert e == aaa, e assert e.name == 'aaa', e.name @@ -92,17 +97,18 @@ class BaseTestCase(SConsignTestCase): assert not hasattr(e, 'arg2'), e f.set_entry('bbb', ccc) + e = f.get_entry('bbb') assert e.name == 'ccc', e.name assert not hasattr(e, 'arg1'), e assert e.arg2 == 'ccc arg2', e.arg1 - ddd = BuildInfo('ddd') - eee = BuildInfo('eee') - fff = BuildInfo('fff') + ddd = DummySConsignEntry('ddd') + eee = DummySConsignEntry('eee') + fff = DummySConsignEntry('fff') fff.arg = 'fff arg' - f = SCons.SConsign.Base(DummyModule()) + f = SCons.SConsign.Base() f.set_entry('ddd', ddd) f.set_entry('eee', eee) @@ -116,57 +122,149 @@ class BaseTestCase(SConsignTestCase): assert not hasattr(e, 'arg'), e f.set_entry('eee', fff) + + e = f.get_entry('eee') + assert e.name == 'fff', e.name + assert e.arg == 'fff arg', e.arg + + def test_store_info(self): + aaa = DummySConsignEntry('aaa') + bbb = DummySConsignEntry('bbb') + bbb.arg1 = 'bbb arg1' + ccc = DummySConsignEntry('ccc') + ccc.arg2 = 'ccc arg2' + + f = SCons.SConsign.Base() + f.store_info('aaa', DummyNode('aaa', aaa)) + f.store_info('bbb', DummyNode('bbb', bbb)) + + try: + e = f.get_entry('aaa') + except KeyError: + pass + else: + raise "unexpected entry %s" % e + + try: + e = f.get_entry('bbb') + except KeyError: + pass + else: + raise "unexpected entry %s" % e + + f.merge() + + e = f.get_entry('aaa') + assert e == aaa, "aaa = %s, e = %s" % (aaa, e) + assert e.name == 'aaa', e.name + + e = f.get_entry('bbb') + assert e == bbb, "bbb = %s, e = %s" % (bbb, e) + assert e.name == 'bbb', e.name + assert e.arg1 == 'bbb arg1', e.arg1 + assert not hasattr(e, 'arg2'), e + + f.store_info('bbb', DummyNode('bbb', ccc)) + + e = f.get_entry('bbb') + assert e == bbb, e + assert e.name == 'bbb', e.name + assert e.arg1 == 'bbb arg1', e.arg1 + assert not hasattr(e, 'arg2'), e + + f.merge() + + e = f.get_entry('bbb') + assert e.name == 'ccc', e.name + assert not hasattr(e, 'arg1'), e + assert e.arg2 == 'ccc arg2', e.arg1 + + ddd = DummySConsignEntry('ddd') + eee = DummySConsignEntry('eee') + fff = DummySConsignEntry('fff') + fff.arg = 'fff arg' + + f = SCons.SConsign.Base() + f.store_info('ddd', DummyNode('ddd', ddd)) + f.store_info('eee', DummyNode('eee', eee)) + + f.merge() + + e = f.get_entry('ddd') + assert e == ddd, e + assert e.name == 'ddd', e.name + + e = f.get_entry('eee') + assert e == eee, e + assert e.name == 'eee', e.name + assert not hasattr(e, 'arg'), e + + f.store_info('eee', DummyNode('eee', fff)) + + e = f.get_entry('eee') + assert e == eee, e + assert e.name == 'eee', e.name + assert not hasattr(e, 'arg'), e + + f.merge() + e = f.get_entry('eee') assert e.name == 'fff', e.name assert e.arg == 'fff arg', e.arg class SConsignDBTestCase(SConsignTestCase): - def runTest(self): + def test_SConsignDB(self): save_DataBase = SCons.SConsign.DataBase SCons.SConsign.DataBase = {} try: d1 = SCons.SConsign.DB(DummyNode('dir1')) - d1.set_entry('aaa', BuildInfo('aaa name')) - d1.set_entry('bbb', BuildInfo('bbb name')) + d1.set_entry('aaa', DummySConsignEntry('aaa name')) + d1.set_entry('bbb', DummySConsignEntry('bbb name')) + aaa = d1.get_entry('aaa') assert aaa.name == 'aaa name' bbb = d1.get_entry('bbb') assert bbb.name == 'bbb name' d2 = SCons.SConsign.DB(DummyNode('dir2')) - d2.set_entry('ccc', BuildInfo('ccc name')) - d2.set_entry('ddd', BuildInfo('ddd name')) + d2.set_entry('ccc', DummySConsignEntry('ccc name')) + d2.set_entry('ddd', DummySConsignEntry('ddd name')) + ccc = d2.get_entry('ccc') assert ccc.name == 'ccc name' ddd = d2.get_entry('ddd') assert ddd.name == 'ddd name' d31 = SCons.SConsign.DB(DummyNode('dir3/sub1')) - d31.set_entry('eee', BuildInfo('eee name')) - d31.set_entry('fff', BuildInfo('fff name')) + d31.set_entry('eee', DummySConsignEntry('eee name')) + d31.set_entry('fff', DummySConsignEntry('fff name')) + eee = d31.get_entry('eee') assert eee.name == 'eee name' fff = d31.get_entry('fff') assert fff.name == 'fff name' d32 = SCons.SConsign.DB(DummyNode('dir3%ssub2' % os.sep)) - d32.set_entry('ggg', BuildInfo('ggg name')) - d32.set_entry('hhh', BuildInfo('hhh name')) + d32.set_entry('ggg', DummySConsignEntry('ggg name')) + d32.set_entry('hhh', DummySConsignEntry('hhh name')) + ggg = d32.get_entry('ggg') assert ggg.name == 'ggg name' hhh = d32.get_entry('hhh') assert hhh.name == 'hhh name' + finally: + SCons.SConsign.DataBase = save_DataBase class SConsignDirFileTestCase(SConsignTestCase): - def runTest(self): - bi_foo = BuildInfo('foo') - bi_bar = BuildInfo('bar') + def test_SConsignDirFile(self): + bi_foo = DummySConsignEntry('foo') + bi_bar = DummySConsignEntry('bar') - f = SCons.SConsign.DirFile(DummyNode(), DummyModule()) + f = SCons.SConsign.DirFile(DummyNode()) f.set_entry('foo', bi_foo) f.set_entry('bar', bi_bar) @@ -174,18 +272,16 @@ class SConsignDirFileTestCase(SConsignTestCase): assert e == bi_foo, e assert e.name == 'foo', e.name - assert bi_foo.c_from_s, bi_foo.c_from_s - e = f.get_entry('bar') assert e == bi_bar, e assert e.name == 'bar', e.name assert not hasattr(e, 'arg'), e - assert bi_bar.c_from_s, bi_bar.c_from_s - - bbb = BuildInfo('bbb') + bbb = DummySConsignEntry('bbb') bbb.arg = 'bbb arg' + f.set_entry('bar', bbb) + e = f.get_entry('bar') assert e.name == 'bbb', e.name assert e.arg == 'bbb arg', e.arg @@ -193,7 +289,7 @@ class SConsignDirFileTestCase(SConsignTestCase): class SConsignFileTestCase(SConsignTestCase): - def runTest(self): + def test_SConsignFile(self): test = self.test file = test.workpath('sconsign_file') @@ -242,7 +338,7 @@ class SConsignFileTestCase(SConsignTestCase): class writeTestCase(SConsignTestCase): - def runTest(self): + def test_write(self): test = self.test file = test.workpath('sconsign_file') @@ -263,10 +359,10 @@ class writeTestCase(SConsignTestCase): SCons.SConsign.DataBase = {} SCons.SConsign.File(file, fake_dbm) - f = SCons.SConsign.DB(DummyNode(), DummyModule()) + f = SCons.SConsign.DB(DummyNode()) - bi_foo = BuildInfo('foo') - bi_bar = BuildInfo('bar') + bi_foo = DummySConsignEntry('foo') + bi_bar = DummySConsignEntry('bar') f.set_entry('foo', bi_foo) f.set_entry('bar', bi_bar) @@ -278,18 +374,18 @@ class writeTestCase(SConsignTestCase): assert fake_dbm.sync_count == 1, fake_dbm.sync_count -def suite(): - suite = unittest.TestSuite() - suite.addTest(BaseTestCase()) - suite.addTest(SConsignDBTestCase()) - suite.addTest(SConsignDirFileTestCase()) - suite.addTest(SConsignFileTestCase()) - suite.addTest(writeTestCase()) - return suite if __name__ == "__main__": - runner = unittest.TextTestRunner() - result = runner.run(suite()) - if not result.wasSuccessful(): + suite = unittest.TestSuite() + tclasses = [ + BaseTestCase, + SConsignDBTestCase, + SConsignDirFileTestCase, + SConsignFileTestCase, + writeTestCase, + ] + for tclass in tclasses: + names = unittest.getTestCaseNames(tclass, 'test_') + suite.addTests(map(tclass, names)) + if not unittest.TextTestRunner().run(suite).wasSuccessful(): sys.exit(1) - diff --git a/src/engine/SCons/Scanner/ScannerTests.py b/src/engine/SCons/Scanner/ScannerTests.py index ae77908..64d6d77 100644 --- a/src/engine/SCons/Scanner/ScannerTests.py +++ b/src/engine/SCons/Scanner/ScannerTests.py @@ -26,7 +26,6 @@ __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" import sys import unittest import UserDict -import SCons.Sig import SCons.Scanner @@ -51,8 +50,6 @@ class DummyEnvironment(UserDict.UserDict): if type(path) != type([]): path = [path] return map(self.subst, path) - def get_calculator(self): - return SCons.Sig.default_calc def get_factory(self, factory): return factory or self.fs.File @@ -399,7 +396,7 @@ class CurrentTestCase(unittest.TestCase): class MyNode: def __init__(self): self.called_has_builder = None - self.called_current = None + self.called_is_up_to_date = None self.func_called = None def rexists(self): return 1 @@ -411,15 +408,15 @@ class CurrentTestCase(unittest.TestCase): def has_builder(self): self.called_has_builder = 1 return 1 - def current(self, sig): - self.called_current = 1 + def is_up_to_date(self): + self.called_is_up_to_date = 1 return None class IsCurrent(MyNode): def has_builder(self): self.called_has_builder = 1 return 1 - def current(self, sig): - self.called_current = 1 + def is_up_to_date(self): + self.called_is_up_to_date = 1 return 1 def func(node, env, path): node.func_called = 1 @@ -430,17 +427,17 @@ class CurrentTestCase(unittest.TestCase): hnb = HasNoBuilder() s(hnb, env, path) self.failUnless(hnb.called_has_builder, "did not call has_builder()") - self.failUnless(not hnb.called_current, "did call current()") + self.failUnless(not hnb.called_is_up_to_date, "did call is_up_to_date()") self.failUnless(hnb.func_called, "did not call func()") inc = IsNotCurrent() s(inc, env, path) self.failUnless(inc.called_has_builder, "did not call has_builder()") - self.failUnless(inc.called_current, "did not call current()") + self.failUnless(inc.called_is_up_to_date, "did not call is_up_to_date()") self.failUnless(not inc.func_called, "did call func()") ic = IsCurrent() s(ic, env, path) self.failUnless(ic.called_has_builder, "did not call has_builder()") - self.failUnless(ic.called_current, "did not call current()") + self.failUnless(ic.called_is_up_to_date, "did not call is_up_to_date()") self.failUnless(ic.func_called, "did not call func()") class ClassicTestCase(unittest.TestCase): diff --git a/src/engine/SCons/Scanner/__init__.py b/src/engine/SCons/Scanner/__init__.py index db93f61..c8ab155 100644 --- a/src/engine/SCons/Scanner/__init__.py +++ b/src/engine/SCons/Scanner/__init__.py @@ -33,7 +33,6 @@ import re import string import SCons.Node.FS -import SCons.Sig import SCons.Util @@ -306,8 +305,7 @@ class Current(Base): def __init__(self, *args, **kw): def current_check(node, env): - c = not node.has_builder() or node.current(env.get_calculator()) - return c + return not node.has_builder() or node.is_up_to_date() kw['scan_check'] = current_check apply(Base.__init__, (self,) + args, kw) diff --git a/src/engine/SCons/Script/Main.py b/src/engine/SCons/Script/Main.py index a76165f..36dc21e 100644 --- a/src/engine/SCons/Script/Main.py +++ b/src/engine/SCons/Script/Main.py @@ -64,7 +64,6 @@ import SCons.Node import SCons.Node.FS import SCons.SConf import SCons.Script -import SCons.Sig import SCons.Taskmaster import SCons.Util import SCons.Warnings @@ -80,13 +79,77 @@ progress_display = SCons.Util.DisplayEngine() first_command_start = None last_command_end = None +class Progressor: + prev = '' + count = 0 + target_string = '$TARGET' + + def __init__(self, obj, interval=1, file=None, overwrite=False): + if file is None: + file = sys.stdout + + self.obj = obj + self.file = file + self.interval = interval + self.overwrite = overwrite + + if callable(obj): + self.func = obj + elif SCons.Util.is_List(obj): + self.func = self.spinner + elif string.find(obj, self.target_string) != -1: + self.func = self.replace_string + else: + self.func = self.string + + def write(self, s): + self.file.write(s) + self.file.flush() + self.prev = s + + def erase_previous(self): + if self.prev: + length = len(self.prev) + if self.prev[-1] in ('\n', '\r'): + length = length - 1 + self.write(' ' * length + '\r') + self.prev = '' + + def spinner(self, node): + self.write(self.obj[self.count % len(self.obj)]) + + def string(self, node): + self.write(self.obj) + + def replace_string(self, node): + self.write(string.replace(self.obj, self.target_string, str(node))) + + def __call__(self, node): + self.count = self.count + 1 + if (self.count % self.interval) == 0: + if self.overwrite: + self.erase_previous() + self.func(node) + +ProgressObject = SCons.Util.Null() + +def Progress(*args, **kw): + global ProgressObject + ProgressObject = apply(Progressor, args, kw) + # Task control. # class BuildTask(SCons.Taskmaster.Task): """An SCons build task.""" + progress = ProgressObject + def display(self, message): display('scons: ' + message) + def prepare(self): + self.progress(self.targets[0]) + return SCons.Taskmaster.Task.prepare(self) + def execute(self): for target in self.targets: if target.get_state() == SCons.Node.up_to_date: @@ -276,6 +339,12 @@ class CleanTask(SCons.Taskmaster.Task): execute = remove + # We want the Taskmaster to update the Node states (and therefore + # handle reference counts, etc.), but we don't want to call + # back to the Node's post-build methods, which would do things + # we don't want, like store .sconsign information. + executed = SCons.Taskmaster.Task.executed_without_callbacks + # Have the taskmaster arrange to "execute" all of the targets, because # we'll figure out ourselves (in remove() or show() above) whether # anything really needs to be done. @@ -290,7 +359,8 @@ class QuestionTask(SCons.Taskmaster.Task): pass def execute(self): - if self.targets[0].get_state() != SCons.Node.up_to_date: + if self.targets[0].get_state() != SCons.Node.up_to_date or \ + (self.top and not self.targets[0].exists()): global exit_status exit_status = 1 self.tm.stop() @@ -766,6 +836,12 @@ def _main(parser): CleanTask.execute = CleanTask.show if options.question: SCons.SConf.dryrun = 1 + if options.clean or options.help: + # If they're cleaning targets or have asked for help, replace + # the whole SCons.SConf module with a Null object so that the + # Configure() calls when reading the SConscript files don't + # actually do anything. + SCons.SConf.SConf = SCons.Util.Null SCons.SConf.SetCacheMode(options.config) SCons.SConf.SetProgressDisplay(progress_display) @@ -838,7 +914,8 @@ def _main(parser): memory_stats.append('after reading SConscript files:') count_stats.append(('post-', 'read')) - SCons.SConf.CreateConfigHBuilder(SCons.Defaults.DefaultEnvironment()) + if not options.help: + SCons.SConf.CreateConfigHBuilder(SCons.Defaults.DefaultEnvironment()) # Now re-parse the command-line options (any to the left of a '--' # argument, that is) with any user-defined command-line options that @@ -972,6 +1049,8 @@ def _main(parser): except AttributeError: pass + task_class.progress = ProgressObject + if options.random: def order(dependencies): """Randomize the dependencies.""" diff --git a/src/engine/SCons/Script/SConscript.py b/src/engine/SCons/Script/SConscript.py index a1856da..5278d40 100644 --- a/src/engine/SCons/Script/SConscript.py +++ b/src/engine/SCons/Script/SConscript.py @@ -65,6 +65,9 @@ import UserList #CommandLineTargets = [] #DefaultTargets = [] +class SConscriptReturn(Exception): + pass + launch_dir = os.path.abspath(os.curdir) GlobalDict = None @@ -125,6 +128,8 @@ class Frame: # make sure the sconscript attr is a Node. if isinstance(sconscript, SCons.Node.Node): self.sconscript = sconscript + elif sconscript == '-': + self.sconscript = None else: self.sconscript = fs.File(str(sconscript)) @@ -133,7 +138,7 @@ call_stack = [] # For documentation on the methods in this file, see the scons man-page -def Return(*vars): +def Return(*vars, **kw): retval = [] try: for var in vars: @@ -147,6 +152,11 @@ def Return(*vars): else: call_stack[-1].retval = tuple(retval) + stop = kw.get('stop', True) + + if stop: + raise SConscriptReturn + stack_bottom = '% Stack boTTom %' # hard to define a variable w/this name :) @@ -242,7 +252,10 @@ def _SConscript(fs, *files, **kw): except KeyError: pass try: - exec _file_ in call_stack[-1].globals + try: + exec _file_ in call_stack[-1].globals + except SConscriptReturn: + pass finally: if old_file is not None: call_stack[-1].globals.update({__file__:old_file}) diff --git a/src/engine/SCons/Script/__init__.py b/src/engine/SCons/Script/__init__.py index 4010d80..d1a115a 100644 --- a/src/engine/SCons/Script/__init__.py +++ b/src/engine/SCons/Script/__init__.py @@ -110,6 +110,7 @@ QuestionTask = Main.QuestionTask AddOption = Main.AddOption GetOption = Main.GetOption SetOption = Main.SetOption +Progress = Main.Progress #keep_going_on_error = Main.keep_going_on_error #print_dtree = Main.print_dtree @@ -260,7 +261,7 @@ def HelpFunction(text): sconscript_reading = 0 # -def Options(files=None, args=ARGUMENTS): +def Options(files=[], args=ARGUMENTS): return SCons.Options.Options(files, args) # The list of global functions to add to the SConscript name space @@ -288,6 +289,7 @@ GlobalDefaultEnvironmentFunctions = [ 'CacheDir', 'Clean', #The Command() method is handled separately, below. + 'Decider', 'Depends', 'Dir', 'NoClean', diff --git a/src/engine/SCons/Sig/__init__.py b/src/engine/SCons/Sig.py index cfad3e8..80bb810 100644 --- a/src/engine/SCons/Sig/__init__.py +++ b/src/engine/SCons/Sig.py @@ -1,9 +1,3 @@ -"""SCons.Sig - -The Signature package for the scons software construction utility. - -""" - # # __COPYRIGHT__ # @@ -29,25 +23,35 @@ The Signature package for the scons software construction utility. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -try: - import MD5 - default_module = MD5 -except ImportError: - import TimeStamp - default_module = TimeStamp +__doc__ = """Place-holder for the old SCons.Sig module hierarchy + +This is no longer used, but code out there (such as the NSIS module on +the SCons wiki) may try to import SCons.Sig. If so, we generate a warning +that points them to the line that caused the import, and don't die. + +If someone actually tried to use the sub-modules or functions within +the package (for example, SCons.Sig.MD5.signature()), then they'll still +get an AttributeError, but at least they'll know where to start looking. +""" + +import SCons.Util +import SCons.Warnings + +msg = 'The SCons.Sig module no longer exists.\n' \ + ' Remove the following "import SCons.Sig" line to eliminate this warning:' + +SCons.Warnings.warn(SCons.Warnings.DeprecatedWarning, msg) -class Calculator: - """ - Encapsulates signature calculations and .sconsign file generating - for the build engine. - """ +default_calc = None +default_module = None - def __init__(self, module=default_module): - """ - Initialize the calculator. +class MD5Null(SCons.Util.Null): + def __repr__(self): + return "MD5Null()" - module - the signature module to use for signature calculations - """ - self.module = module +class TimeStampNull(SCons.Util.Null): + def __repr__(self): + return "TimeStampNull()" -default_calc = Calculator() +MD5 = MD5Null() +TimeStamp = TimeStampNull() diff --git a/src/engine/SCons/Sig/.aeignore b/src/engine/SCons/Sig/.aeignore deleted file mode 100644 index 22ebd62..0000000 --- a/src/engine/SCons/Sig/.aeignore +++ /dev/null @@ -1,5 +0,0 @@ -*,D -*.pyc -.*.swp -.consign -.sconsign diff --git a/src/engine/SCons/Sig/.cvsignore b/src/engine/SCons/Sig/.cvsignore deleted file mode 100644 index 0d20b64..0000000 --- a/src/engine/SCons/Sig/.cvsignore +++ /dev/null @@ -1 +0,0 @@ -*.pyc diff --git a/src/engine/SCons/Sig/MD5.py b/src/engine/SCons/Sig/MD5.py deleted file mode 100644 index f8f349f..0000000 --- a/src/engine/SCons/Sig/MD5.py +++ /dev/null @@ -1,104 +0,0 @@ -"""SCons.Sig.MD5 - -The MD5 signature package for the SCons software construction -utility. - -""" - -# -# __COPYRIGHT__ -# -# Permission is hereby granted, free of charge, to any person obtaining -# a copy of this software and associated documentation files (the -# "Software"), to deal in the Software without restriction, including -# without limitation the rights to use, copy, modify, merge, publish, -# distribute, sublicense, and/or sell copies of the Software, and to -# permit persons to whom the Software is furnished to do so, subject to -# the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY -# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE -# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" - -import imp -import string - -# Force Python to load the builtin "md5" module. If we do this with a -# normal import statement, then case-insensitive systems (Windows) get -# confused and thinks there's a case mismatch with *this* MD5.py module. -file, name, desc = imp.find_module('md5') -try: - md5 = imp.load_module('md5', file, name, desc) -finally: - if file: - file.close() - -def current(new, old): - """Return whether a new signature is up-to-date with - respect to an old signature. - """ - return new == old - -try: - md5.new('').hexdigest -except AttributeError: - # The md5 objects created by the module have no native hexdigest() - # method (*cough* 1.5.2 *cough*) so provide an equivalent. - class new_md5: - def __init__(self, s): - self.m = md5.new(str(s)) - #def copy(self): - # return self.m.copy() - def digest(self): - return self.m.digest() - def hexdigest(self): - h = string.hexdigits - r = '' - for c in self.m.digest(): - i = ord(c) - r = r + h[(i >> 4) & 0xF] + h[i & 0xF] - return r - def update(self, s): - return self.m.update(s) - -else: - new_md5 = lambda s: md5.new(str(s)) - -def collect(signatures): - """ - Collect a list of signatures into an aggregate signature. - - signatures - a list of signatures - returns - the aggregate signature - """ - if len(signatures) == 1: - return signatures[0] - else: - return new_md5(string.join(signatures, ', ')).hexdigest() - -def signature(obj): - """Generate a signature for an object - """ - try: - gc = obj.get_contents - except AttributeError: - raise AttributeError, "unable to fetch contents of '%s'" % str(obj) - return new_md5(gc()).hexdigest() - -def to_string(signature): - """Convert a signature to a string""" - return signature - -def from_string(string): - """Convert a string to a signature""" - return string diff --git a/src/engine/SCons/Sig/MD5Tests.py b/src/engine/SCons/Sig/MD5Tests.py deleted file mode 100644 index ba18264..0000000 --- a/src/engine/SCons/Sig/MD5Tests.py +++ /dev/null @@ -1,113 +0,0 @@ -# -# __COPYRIGHT__ -# -# Permission is hereby granted, free of charge, to any person obtaining -# a copy of this software and associated documentation files (the -# "Software"), to deal in the Software without restriction, including -# without limitation the rights to use, copy, modify, merge, publish, -# distribute, sublicense, and/or sell copies of the Software, and to -# permit persons to whom the Software is furnished to do so, subject to -# the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY -# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE -# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" - -import sys -import unittest -import string - -from SCons.Sig.MD5 import current, collect, signature, to_string, from_string - - - - -class my_obj: - """A dummy object class that satisfies the interface - requirements of the MD5 class. - """ - - def __init__(self, value = ""): - self.value = value - - def get_signature(self): - if not hasattr(self, "sig"): - self.sig = signature(self) - return self.sig - - def get_contents(self): - return self.value - - - -class MD5TestCase(unittest.TestCase): - - def test_current(self): - """Test deciding if an object is up-to-date - - Simple comparison of different "signature" values. - """ - obj = my_obj('111') - assert not current(obj.get_signature(), signature(my_obj('110'))) - assert current(obj.get_signature(), signature(my_obj('111'))) - assert not current(obj.get_signature(), signature(my_obj('112'))) - - def test_collect(self): - """Test collecting a list of signatures into a new signature value - """ - s = map(signature, map(my_obj, ('111', '222', '333'))) - - assert '698d51a19d8a121ce581499d7b701668' == collect(s[0:1]) - assert '8980c988edc2c78cc43ccb718c06efd5' == collect(s[0:2]) - assert '53fd88c84ff8a285eb6e0a687e55b8c7' == collect(s) - - def test_signature(self): - """Test generating a signature""" - o1 = my_obj(value = '111') - s = signature(o1) - assert '698d51a19d8a121ce581499d7b701668' == s, s - - o2 = my_obj(value = 222) - s = signature(o2) - assert 'bcbe3365e6ac95ea2c0343a2395834dd' == s, s - - try: - signature('string') - except AttributeError, e: - assert string.find(str(e), "unable to fetch contents") == 0, str(e) - else: - raise AttributeError, "unexpected get_contents() attribute" - - # Make sure we don't eat AttributeErrors raised internally - # by the get_contents() method (or anything it calls). - caught = None - try: - class xxx: - def get_contents(self): - raise AttributeError, "internal AttributeError" - signature(xxx()) - except AttributeError, e: - assert str(e) == "internal AttributeError", e - caught = 1 - assert caught, "did not catch expected AttributeError" - - def test_to_string(self): - assert '698d51a19d8a121ce581499d7b701668' == to_string('698d51a19d8a121ce581499d7b701668') - - def test_from_string(self): - assert '698d51a19d8a121ce581499d7b701668' == from_string('698d51a19d8a121ce581499d7b701668') - -if __name__ == "__main__": - suite = unittest.makeSuite(MD5TestCase, 'test_') - if not unittest.TextTestRunner().run(suite).wasSuccessful(): - sys.exit(1) diff --git a/src/engine/SCons/Sig/SigTests.py b/src/engine/SCons/Sig/SigTests.py deleted file mode 100644 index 6b7c83f..0000000 --- a/src/engine/SCons/Sig/SigTests.py +++ /dev/null @@ -1,49 +0,0 @@ -# -# __COPYRIGHT__ -# -# Permission is hereby granted, free of charge, to any person obtaining -# a copy of this software and associated documentation files (the -# "Software"), to deal in the Software without restriction, including -# without limitation the rights to use, copy, modify, merge, publish, -# distribute, sublicense, and/or sell copies of the Software, and to -# permit persons to whom the Software is furnished to do so, subject to -# the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY -# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE -# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" - -import unittest -import SCons.Sig -import sys - -class CalculatorTestCase(unittest.TestCase): - - def runTest(self): - class MySigModule: - pass - calc = SCons.Sig.Calculator(MySigModule) - assert calc.module == MySigModule - - -def suite(): - suite = unittest.TestSuite() - suite.addTest(CalculatorTestCase()) - return suite - -if __name__ == "__main__": - runner = unittest.TextTestRunner() - result = runner.run(suite()) - if not result.wasSuccessful(): - sys.exit(1) - diff --git a/src/engine/SCons/Sig/TimeStamp.py b/src/engine/SCons/Sig/TimeStamp.py deleted file mode 100644 index 8cf23e8..0000000 --- a/src/engine/SCons/Sig/TimeStamp.py +++ /dev/null @@ -1,75 +0,0 @@ -"""SCons.Sig.TimeStamp - -The TimeStamp signature package for the SCons software construction -utility. - -""" - -# -# __COPYRIGHT__ -# -# Permission is hereby granted, free of charge, to any person obtaining -# a copy of this software and associated documentation files (the -# "Software"), to deal in the Software without restriction, including -# without limitation the rights to use, copy, modify, merge, publish, -# distribute, sublicense, and/or sell copies of the Software, and to -# permit persons to whom the Software is furnished to do so, subject to -# the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY -# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE -# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" - -def current(new, old): - """Return whether a new timestamp is up-to-date with - respect to an old timestamp. - """ - return not old is None and new <= old - -def collect(signatures): - """ - Collect a list of timestamps, returning - the most-recent timestamp from the list - - signatures - a list of timestamps - returns - the most recent timestamp - """ - - if len(signatures) == 0: - return 0 - elif len(signatures) == 1: - return signatures[0] - else: - return max(signatures) - -def signature(obj): - """Generate a timestamp. - """ - return obj.get_timestamp() - -def to_string(signature): - """Convert a timestamp to a string""" - return str(signature) - -def from_string(string): - """Convert a string to a timestamp""" - try: - return int(string) - except ValueError: - # if the signature isn't an int, then - # the user probably just switched from - # MD5 signatures to timestamp signatures, - # so ignore the error: - return None - - diff --git a/src/engine/SCons/Sig/TimeStampTests.py b/src/engine/SCons/Sig/TimeStampTests.py deleted file mode 100644 index 49c5cfb..0000000 --- a/src/engine/SCons/Sig/TimeStampTests.py +++ /dev/null @@ -1,84 +0,0 @@ -# -# __COPYRIGHT__ -# -# Permission is hereby granted, free of charge, to any person obtaining -# a copy of this software and associated documentation files (the -# "Software"), to deal in the Software without restriction, including -# without limitation the rights to use, copy, modify, merge, publish, -# distribute, sublicense, and/or sell copies of the Software, and to -# permit persons to whom the Software is furnished to do so, subject to -# the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY -# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE -# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" - -import sys -import unittest - -from SCons.Sig.TimeStamp import current, collect, signature, to_string, from_string - - - -class my_obj: - """A dummy object class that satisfies the interface - requirements of the TimeStamp class. - """ - - def __init__(self, value = 0): - self.value = value - - def get_signature(self): - return self.value - - def get_timestamp(self): - return self.value - - -class TimeStampTestCase(unittest.TestCase): - - def test_current(self): - """Test deciding if an object is up-to-date - - Simple comparison of different timestamp values. - """ - o1 = my_obj(value = 111) - assert not current(o1.get_signature(), 110) - assert current(o1.get_signature(), 111) - assert current(o1.get_signature(), 112) - - def test_collect(self): - """Test collecting a list of signatures into a new signature value - into a new timestamp value. - """ - - assert 111 == collect((111,)) - assert 222 == collect((111, 222)) - assert 333 == collect((333, 222, 111)) - - def test_signature(self): - """Test generating a signature""" - o1 = my_obj(value = 111) - assert 111 == signature(o1) - - def test_to_string(self): - assert '111' == to_string(111) - - def test_from_string(self): - assert 111 == from_string('111') - - -if __name__ == "__main__": - suite = unittest.makeSuite(TimeStampTestCase, 'test_') - if not unittest.TextTestRunner().run(suite).wasSuccessful(): - sys.exit(1) diff --git a/src/engine/SCons/Taskmaster.py b/src/engine/SCons/Taskmaster.py index acb8c9b..598566e 100644 --- a/src/engine/SCons/Taskmaster.py +++ b/src/engine/SCons/Taskmaster.py @@ -205,20 +205,40 @@ class Task: raise SCons.Errors.TaskmasterException(self.targets[0], sys.exc_info()) - def executed(self): + def executed_without_callbacks(self): """ - Called when the task has been successfully executed. + Called when the task has been successfully executed + and the Taskmaster instance doesn't want to call + the Node's callback methods. + """ + for t in self.targets: + if t.get_state() == SCons.Node.executing: + for side_effect in t.side_effects: + side_effect.set_state(SCons.Node.no_state) + t.set_state(SCons.Node.executed) + + def executed_with_callbacks(self): + """ + Called when the task has been successfully executed and + the Taskmaster instance wants to call the Node's callback + methods. This may have been a do-nothing operation (to preserve build - order), so we have to check the node's state before deciding - whether it was "built" or just "visited." + order), so we must check the node's state before deciding whether + it was "built", in which case we call the appropriate Node method. + In any event, we always call "visited()", which will handle any + post-visit actions that must take place regardless of whether + or not the target was an actual built target or a source Node. """ for t in self.targets: if t.get_state() == SCons.Node.executing: + for side_effect in t.side_effects: + side_effect.set_state(SCons.Node.no_state) t.set_state(SCons.Node.executed) t.built() - else: - t.visited() + t.visited() + + executed = executed_with_callbacks def failed(self): """ @@ -275,9 +295,10 @@ class Task: """ self.out_of_date = [] for t in self.targets: - t.disambiguate() try: - is_up_to_date = not t.always_build and t.current() + t.disambiguate().make_ready() + is_up_to_date = not t.has_builder() or \ + (not t.always_build and t.is_up_to_date()) except EnvironmentError, e: raise SCons.Errors.BuildError(node=t, errstr=e.strerror, filename=e.filename) if is_up_to_date: @@ -308,12 +329,14 @@ class Task: # back on the candidates list if the Node is also a waiting # parent. + targets = set(self.targets) + parents = {} - for t in self.targets: + for t in targets: for p in t.waiting_parents.keys(): parents[p] = parents.get(p, 0) + 1 - for t in self.targets: + for t in targets: for s in t.side_effects: if s.get_state() == SCons.Node.executing: s.set_state(SCons.Node.no_state) @@ -329,7 +352,7 @@ class Task: if p.ref_count == 0: self.tm.candidates.append(p) - for t in self.targets: + for t in targets: t.postprocess() # Exception handling subsystem. @@ -403,8 +426,9 @@ class Taskmaster: """ def __init__(self, targets=[], tasker=Task, order=None, trace=None): - self.top_targets = targets[:] - self.top_targets.reverse() + self.original_top = targets + self.top_targets_left = targets[:] + self.top_targets_left.reverse() self.candidates = [] self.tasker = tasker if not order: @@ -438,7 +462,7 @@ class Taskmaster: except IndexError: pass try: - node = self.top_targets.pop() + node = self.top_targets_left.pop() except IndexError: return None self.current_top = node @@ -536,16 +560,14 @@ class Taskmaster: c.sort() T.write(' children:\n %s\n ' % c) - childinfo = map(lambda N: (N.get_state(), - N.is_derived() or N.is_pseudo_derived(), - N), children) + childstate = map(lambda N: (N, N.get_state()), children) # Skip this node if any of its children have failed. This # catches the case where we're descending a top-level target # and one of our children failed while trying to be built # by a *previous* descent of an earlier top-level target. - failed_children = filter(lambda I: I[0] == SCons.Node.failed, - childinfo) + failed_children = filter(lambda I: I[1] == SCons.Node.failed, + childstate) if failed_children: node.set_state(SCons.Node.failed) if S: S.child_failed = S.child_failed + 1 @@ -556,76 +578,48 @@ class Taskmaster: continue # Detect dependency cycles: - pending_nodes = filter(lambda I: I[0] == SCons.Node.pending, childinfo) + pending_nodes = filter(lambda I: I[1] == SCons.Node.pending, childstate) if pending_nodes: for p in pending_nodes: - cycle = find_cycle([p[2], node]) + cycle = find_cycle([p[0], node]) if cycle: desc = "Dependency cycle: " + string.join(map(str, cycle), " -> ") if T: T.write(' dependency cycle\n') raise SCons.Errors.UserError, desc - # Select all of the dependencies that are derived targets - # (that is, children who have builders or are side effects). - derived_children = filter(lambda I: I[1], childinfo) - - not_started = filter(lambda I: not I[0], derived_children) - if not_started: - not_started = map(lambda I: I[2], not_started) - - # We're waiting on one more derived targets that have - # not yet started building. Add this node to the - # waiting_parents lists of those derived files so that - # when they've finished building, our implicit dependency - # list will get cleared and we'll re-scan the newly-built - # file(s) for updated implicit dependencies. - added = map(lambda n, P=node: n.add_to_waiting_parents(P), not_started) - node.ref_count = node.ref_count + reduce(operator.add, added, 0) - - # Now we add these derived targets to the candidates - # list so they can be examined and built. We have to - # add ourselves back to the list first, though, so we get - # a chance to re-scan and build after the dependencies. - # - # We reverse the order in which the children are added - # to the candidates stack so the order in which they're - # popped matches the order in which they show up in our - # children's list. This is more logical / efficient for - # builders with multiple targets, since the "primary" - # target will be examined first. - self.candidates.append(node) - not_started.reverse() - self.candidates.extend(self.order(not_started)) - - if S: S.not_started = S.not_started + 1 - if T: - c = map(str, not_started) - c.sort() - T.write(' waiting on unstarted children:\n %s\n' % c) - continue - - not_built = filter(lambda I: I[0] <= SCons.Node.executing, derived_children) + not_built = filter(lambda I: I[1] <= SCons.Node.executing, childstate) if not_built: - not_built = map(lambda I: I[2], not_built) - # We're waiting on one or more derived targets that have - # started building but not yet finished. Add this node - # to the waiting parents lists of those derived files - # so that when they've finished building, our implicit - # dependency list will get cleared and we'll re-scan the - # newly-built file(s) for updated implicit dependencies. - added = map(lambda n, P=node: n.add_to_waiting_parents(P), not_built) - node.ref_count = node.ref_count + reduce(operator.add, added, 0) + # not yet finished building. + + not_visited = filter(lambda I: not I[1], not_built) + if not_visited: + # Some of them haven't even been visited yet. + # Add them to the list so that on some next pass + # we can take a stab at evaluating them (or + # their children). + not_visited = map(lambda I: I[0], not_visited) + not_visited.reverse() + self.candidates.extend(self.order(not_visited)) + + n_b_nodes = map(lambda I: I[0], not_built) + + # Add this node to the waiting parents lists of anything + # we're waiting on, with a reference count so we can be + # put back on the list for re-evaluation when they've + # all finished. + map(lambda n, P=node: n.add_to_waiting_parents(P), n_b_nodes) + node.ref_count = len(set(n_b_nodes)) if S: S.not_built = S.not_built + 1 if T: - c = map(str, not_built) + c = map(str, n_b_nodes) c.sort() T.write(' waiting on unfinished children:\n %s\n' % c) continue - # Skip this node if it has side-effects that are currently being - # built themselves or waiting for something else being built. + # Skip this node if it has side-effects that are + # currently being built: side_effects = filter(lambda N: N.get_state() == SCons.Node.executing, node.side_effects) @@ -660,7 +654,7 @@ class Taskmaster: tlist = node.get_executor().targets - task = self.tasker(self, tlist, node is self.current_top, node) + task = self.tasker(self, tlist, node in self.original_top, node) try: task.make_ready() except KeyboardInterrupt: diff --git a/src/engine/SCons/TaskmasterTests.py b/src/engine/SCons/TaskmasterTests.py index 718851d..c79fb93 100644 --- a/src/engine/SCons/TaskmasterTests.py +++ b/src/engine/SCons/TaskmasterTests.py @@ -74,10 +74,20 @@ class Node: cache_text.append(self.name + " retrieved") return self.cached + def make_ready(self): + pass + + def prepare(self): + self.prepared = 1 + def build(self): global built_text built_text = self.name + " built" + def built(self): + global built_text + built_text = built_text + " really" + def has_builder(self): return not self.builder is None @@ -87,17 +97,10 @@ class Node: def alter_targets(self): return self.alttargets, None - def built(self): - global built_text - built_text = built_text + " really" - def visited(self): global visited_nodes visited_nodes.append(self.name) - def prepare(self): - self.prepared = 1 - def children(self): if not self.scanned: self.scan() @@ -145,18 +148,11 @@ class Node: def store_bsig(self): pass - def calculator(self): - class Calc: - def bsig(self, node): - return node._bsig_val - def current(self, node, sig): - return node._current_val - return Calc() - - def current(self, calc=None): - if calc is None: - calc = self.calculator() - return calc.current(self, calc.bsig(self)) + def is_pseudo_derived(self): + pass + + def is_up_to_date(self): + return self._current_val def depends_on(self, nodes): for node in nodes: @@ -713,7 +709,7 @@ class TaskmasterTestCase(unittest.TestCase): s = n1.get_state() assert s == SCons.Node.executed, s assert built_text == "xxx really", built_text - assert visited_nodes == [], visited_nodes + assert visited_nodes == ['n1'], visited_nodes n2 = Node("n2") tm = SCons.Taskmaster.Taskmaster([n2]) @@ -744,7 +740,7 @@ class TaskmasterTestCase(unittest.TestCase): assert s == SCons.Node.up_to_date, s s = n4.get_state() assert s == SCons.Node.executed, s - assert visited_nodes == ['n3'], visited_nodes + assert visited_nodes == ['n3', 'n4'], visited_nodes def test_prepare(self): """Test preparation of multiple Nodes for a task @@ -1017,11 +1013,19 @@ class TaskmasterTestCase(unittest.TestCase): t = tm.next_task() t.prepare() t.execute() + t.postprocess() n1.set_state(SCons.Node.executed) t = tm.next_task() t.prepare() t.execute() + t.postprocess() + n2.set_state(SCons.Node.executed) + t = tm.next_task() + t.prepare() + t.execute() + t.postprocess() t = tm.next_task() + assert t is None value = trace.getvalue() expect = """\ @@ -1029,13 +1033,12 @@ Taskmaster: 'n1': evaluating n1 Taskmaster: 'n1': already handled (executed) Taskmaster: 'n3': children: ['n1', 'n2'] - waiting on unstarted children: + waiting on unfinished children: ['n2'] Taskmaster: 'n2': evaluating n2 Taskmaster: 'n3': children: ['n1', 'n2'] - waiting on unfinished children: - ['n2'] + evaluating n3 """ assert value == expect, value diff --git a/src/engine/SCons/Tool/JavaCommon.py b/src/engine/SCons/Tool/JavaCommon.py index 5097c67..0991c37 100644 --- a/src/engine/SCons/Tool/JavaCommon.py +++ b/src/engine/SCons/Tool/JavaCommon.py @@ -56,7 +56,7 @@ if java_parsing: # semi-colons; # periods. _reToken = re.compile(r'(\n|\\\\|//|\\[\'"]|[\'"\{\}\;\.\(\)]|' + - r'[A-Za-z_][\w\.]*|/\*|\*/|\[\])') + r'[A-Za-z_][\w\$\.]*|/\*|\*/|\[\])') class OuterState: """The initial state for parsing a Java file for classes, diff --git a/src/engine/SCons/Tool/JavaCommonTests.py b/src/engine/SCons/Tool/JavaCommonTests.py index 358f675..1675190 100644 --- a/src/engine/SCons/Tool/JavaCommonTests.py +++ b/src/engine/SCons/Tool/JavaCommonTests.py @@ -65,6 +65,21 @@ public class Foo assert classes == ['Foo'], classes + + def test_dollar_sign(self): + """Test class names with $ in them""" + + input = """\ +public class BadDep { + public void new$rand () {} +} +""" + pkg_dir, classes = SCons.Tool.JavaCommon.parse_java(input) + assert pkg_dir == None, pkg_dir + assert classes == ['BadDep'], classes + + + def test_inner_classes(self): """Test parsing various forms of inner classes""" diff --git a/src/engine/SCons/Tool/__init__.py b/src/engine/SCons/Tool/__init__.py index 97a4e70..d4e3815 100644 --- a/src/engine/SCons/Tool/__init__.py +++ b/src/engine/SCons/Tool/__init__.py @@ -422,7 +422,54 @@ def CreateJavaFileBuilder(env): env['JAVASUFFIX'] = '.java' return java_file - +class ToolInitializer: + """ + A class for delayed initialization of Tools modules. + + This is intended to be added to a construction environment in + place of the method(s) normally called for a Builder (env.Object, + env.StaticObject, etc.). When called, it searches the specified + list of tools, applies the first one that exists to the construction + environment, and calls whatever builder was (presumably) added the + construction environment in our place. + """ + def __init__(self, name, tools): + """ + Note: we store the tool name as __name__ so it can be used by + the class that attaches this to a construction environment. + """ + self.__name__ = name + if not SCons.Util.is_List(tools): + tools = [tools] + self.tools = tools + def __call__(self, env, *args, **kw): + for t in self.tools: + tool = SCons.Tool.Tool(t) + if tool.exists(env): + env.Tool(tool) + break + + builder = getattr(env, self.__name__) + if builder is self: + # There was no Builder added, which means no valid Tool + # for this name was found (or possibly there's a mismatch + # between the name we were called by and the Builder name + # added by the Tool module). + # + # (Eventually this is where we'll put a more informative + # error message about the inability to find that tool + # as cut over more Builders+Tools to using this. + return [], [] + + # Let the construction environment remove the added method + # so we no longer copy and re-bind this method when the + # construction environment gets cloned. + env.RemoveMethod(self) + return apply(builder, args, kw) + +def Initializers(env): + env.AddMethod(ToolInitializer('Install', 'install')) + env.AddMethod(ToolInitializer('InstallAs', 'install')) def FindTool(tools, env): for tool in tools: @@ -533,7 +580,7 @@ def tool_list(platform, env): other_tools = FindAllTools(['BitKeeper', 'CVS', 'dmd', - 'install', 'filesystem', + 'filesystem', 'dvipdf', 'dvips', 'gs', 'jar', 'javac', 'javah', 'latex', 'lex', diff --git a/src/engine/SCons/Tool/install.py b/src/engine/SCons/Tool/install.py index 828bb9e..c7eee61 100644 --- a/src/engine/SCons/Tool/install.py +++ b/src/engine/SCons/Tool/install.py @@ -42,6 +42,7 @@ from SCons.Util import make_path_relative # # We keep track of *all* installed files. _INSTALLED_FILES = [] +_UNIQUE_INSTALLED_FILES = None # # Functions doing the actual work of the Install Builder. @@ -101,12 +102,9 @@ def add_targets_to_INSTALLED_FILES(target, source, env): _INSTALLED_FILES global variable. This way all installed files of one scons call will be collected. """ - global _INSTALLED_FILES - files = _INSTALLED_FILES - #files.extend( [ x for x in target if not x in files ] ) - for x in target: - if not x in files: - files.append(x) + global _INSTALLED_FILES, _UNIQUE_INSTALLED_FILES + _INSTALLED_FILES.extend(target) + _UNIQUE_INSTALLED_FILES = None return (target, source) class DESTDIR_factory: @@ -131,9 +129,42 @@ class DESTDIR_factory: install_action = SCons.Action.Action(installFunc, stringFunc) installas_action = SCons.Action.Action(installFunc, stringFunc) -InstallBuilder, InstallAsBuilder = None, None BaseInstallBuilder = None +def InstallBuilderWrapper(env, target, source, dir=None): + if target and dir: + raise SCons.Errors.UserError, "Both target and dir defined for Install(), only one may be defined." + if not dir: + dir=target + + import SCons.Script + install_sandbox = SCons.Script.GetOption('install_sandbox') + if install_sandbox: + target_factory = DESTDIR_factory(env, install_sandbox) + else: + target_factory = env.fs + + try: + dnodes = env.arg2nodes(dir, target_factory.Dir) + except TypeError: + raise SCons.Errors.UserError, "Target `%s' of Install() is a file, but should be a directory. Perhaps you have the Install() arguments backwards?" % str(dir) + sources = env.arg2nodes(source, env.fs.Entry) + tgt = [] + for dnode in dnodes: + for src in sources: + # Prepend './' so the lookup doesn't interpret an initial + # '#' on the file name portion as meaning the Node should + # be relative to the top-level SConstruct directory. + target = env.fs.Entry('.'+os.sep+src.name, dnode) + tgt.extend(BaseInstallBuilder(env, target, src)) + return tgt + +def InstallAsBuilderWrapper(env, target, source): + result = [] + for src, tgt in map(lambda x, y: (x, y), source, target): + result.extend(BaseInstallBuilder(env, tgt, src)) + return result + added = None def generate(env): @@ -148,63 +179,31 @@ def generate(env): action="store", help='A directory under which all installed files will be placed.') - try: - env['BUILDERS']['Install'] - env['BUILDERS']['InstallAs'] - - except KeyError, e: + global BaseInstallBuilder + if BaseInstallBuilder is None: install_sandbox = GetOption('install_sandbox') if install_sandbox: target_factory = DESTDIR_factory(env, install_sandbox) else: target_factory = env.fs - global BaseInstallBuilder - if BaseInstallBuilder is None: - BaseInstallBuilder = SCons.Builder.Builder( - action = install_action, - target_factory = target_factory.Entry, - source_factory = env.fs.Entry, - multi = 1, - emitter = [ add_targets_to_INSTALLED_FILES, ], - name = 'InstallBuilder') - - global InstallBuilder - if InstallBuilder is None: - def InstallBuilderWrapper(env, target, source, dir=None, target_factory=target_factory): - if target and dir: - raise SCons.Errors.UserError, "Both target and dir defined for Install(), only one may be defined." - if not dir: - dir=target - try: - dnodes = env.arg2nodes(dir, target_factory.Dir) - except TypeError: - raise SCons.Errors.UserError, "Target `%s' of Install() is a file, but should be a directory. Perhaps you have the Install() arguments backwards?" % str(dir) - sources = env.arg2nodes(source, env.fs.Entry) - tgt = [] - for dnode in dnodes: - for src in sources: - # Prepend './' so the lookup doesn't interpret an initial - # '#' on the file name portion as meaning the Node should - # be relative to the top-level SConstruct directory. - target = env.fs.Entry('.'+os.sep+src.name, dnode) - tgt.extend(BaseInstallBuilder(env, target, src)) - return tgt - - InstallBuilder = InstallBuilderWrapper - - global InstallAsBuilder - if InstallAsBuilder is None: - def InstallAsBuilderWrapper(env, target, source): - result = [] - for src, tgt in map(lambda x, y: (x, y), source, target): - result.extend(BaseInstallBuilder(env, tgt, src)) - return result - - InstallAsBuilder = InstallAsBuilderWrapper - - env['BUILDERS']['Install'] = InstallBuilder - env['BUILDERS']['InstallAs'] = InstallAsBuilder + BaseInstallBuilder = SCons.Builder.Builder( + action = install_action, + target_factory = target_factory.Entry, + source_factory = env.fs.Entry, + multi = 1, + emitter = [ add_targets_to_INSTALLED_FILES, ], + name = 'InstallBuilder') + + try: + env['BUILDERS']['Install'] + except KeyError, e: + env['BUILDERS']['Install'] = InstallBuilderWrapper + + try: + env['BUILDERS']['InstallAs'] + except KeyError, e: + env['BUILDERS']['InstallAs'] = InstallAsBuilderWrapper # We'd like to initialize this doing something like the following, # but there isn't yet support for a ${SOURCE.type} expansion that diff --git a/src/engine/SCons/Tool/intelc.py b/src/engine/SCons/Tool/intelc.py index 2d0a5a5..673c848 100644 --- a/src/engine/SCons/Tool/intelc.py +++ b/src/engine/SCons/Tool/intelc.py @@ -37,6 +37,9 @@ __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" import math, sys, os.path, glob, string, re is_windows = sys.platform == 'win32' +is_win64 = is_windows and (os.environ['PROCESSOR_ARCHITECTURE'] == 'AMD64' or + (os.environ.has_key('PROCESSOR_ARCHITEW6432') and + os.environ['PROCESSOR_ARCHITEW6432'] == 'AMD64')) is_linux = sys.platform == 'linux2' if is_windows: @@ -138,7 +141,10 @@ def get_intel_registry_value(valuename, version=None, abi=None): Return a value from the Intel compiler registry tree. (Windows only) """ # Open the key: - K = 'Software\\Intel\\Compilers\\C++\\' + version + '\\'+abi.upper() + if is_win64: + K = 'Software\\Wow6432Node\\Intel\\Compilers\\C++\\' + version + '\\'+abi.upper() + else: + K = 'Software\\Intel\\Compilers\\C++\\' + version + '\\'+abi.upper() try: k = SCons.Util.RegOpenKeyEx(SCons.Util.HKEY_LOCAL_MACHINE, K) except SCons.Util.RegError: @@ -160,7 +166,10 @@ def get_all_compiler_versions(): """ versions=[] if is_windows: - keyname = 'Software\\Intel\\Compilers\\C++' + if is_win64: + keyname = 'Software\\WoW6432Node\\Intel\\Compilers\\C++' + else: + keyname = 'Software\\Intel\\Compilers\\C++' try: k = SCons.Util.RegOpenKeyEx(SCons.Util.HKEY_LOCAL_MACHINE, keyname) @@ -178,7 +187,7 @@ def get_all_compiler_versions(): # than uninstalling properly), so the registry values # are still there. ok = False - for try_abi in ('IA32', 'IA32e', 'IA64'): + for try_abi in ('IA32', 'IA32e', 'IA64', 'EM64T'): try: d = get_intel_registry_value('ProductDir', subkey, try_abi) except MissingRegistryError: @@ -212,6 +221,7 @@ def get_intel_compiler_top(version, abi): The compiler will be in <top>/bin/icl.exe (icc on linux), the include dir is <top>/include, etc. """ + if is_windows: if not SCons.Util.can_read_reg: raise NoRegistryModuleError, "No Windows registry module was found" @@ -282,8 +292,10 @@ def generate(env, version=None, abi=None, topdir=None, verbose=0): else: abi = 'ia32' else: - # XXX: how would we do the same test on Windows? - abi = "ia32" + if is_win64: + abi = 'em64t' + else: + abi = 'ia32' if version and not topdir: try: diff --git a/src/engine/SCons/Tool/javac.py b/src/engine/SCons/Tool/javac.py index 5703b2d..a020727 100644 --- a/src/engine/SCons/Tool/javac.py +++ b/src/engine/SCons/Tool/javac.py @@ -133,20 +133,28 @@ JavaBuilder = SCons.Builder.Builder(action = JavaAction, target_factory = SCons.Node.FS.Entry, source_factory = SCons.Node.FS.Entry) -def getClassPath(env,target, source, for_signature): - path = "" - if env.has_key('JAVACLASSPATH') and env['JAVACLASSPATH']: - path = SCons.Util.AppendPath(path, env['JAVACLASSPATH']) - return "-classpath %s" % (path) - else: - return "" - -def getSourcePath(env,target, source, for_signature): - path = "" - if env.has_key('JAVASOURCEPATH') and env['JAVASOURCEPATH']: - path = SCons.Util.AppendPath(path, env['JAVASOURCEPATH']) - path = SCons.Util.AppendPath(path,['${TARGET.attributes.java_sourcedir}']) - return "-sourcepath %s" % (path) +class pathopt: + """ + Callable object for generating javac-style path options from + a construction variable (e.g. -classpath, -sourcepath). + """ + def __init__(self, opt, var, default=None): + self.opt = opt + self.var = var + self.default = default + + def __call__(self, target, source, env, for_signature): + path = env[self.var] + if path and not SCons.Util.is_List(path): + path = [path] + if self.default: + path = path + [ env[self.default] ] + if path: + return [self.opt, string.join(path, os.pathsep)] + #return self.opt + " " + string.join(path, os.pathsep) + else: + return [] + #return "" def Java(env, target, source, *args, **kw): """ @@ -195,16 +203,20 @@ def generate(env): env.AddMethod(Java) - env['JAVAC'] = 'javac' - env['JAVACFLAGS'] = SCons.Util.CLVar('') - env['JAVACLASSPATH'] = [] - env['JAVASOURCEPATH'] = [] - env['_JAVACLASSPATH'] = getClassPath - env['_JAVASOURCEPATH'] = getSourcePath - env['_JAVACCOM'] = '$JAVAC $JAVACFLAGS $_JAVACLASSPATH -d ${TARGET.attributes.java_classdir} $_JAVASOURCEPATH $SOURCES' - env['JAVACCOM'] = "${TEMPFILE('$_JAVACCOM')}" - env['JAVACLASSSUFFIX'] = '.class' - env['JAVASUFFIX'] = '.java' + env['JAVAC'] = 'javac' + env['JAVACFLAGS'] = SCons.Util.CLVar('') + env['JAVABOOTCLASSPATH'] = [] + env['JAVACLASSPATH'] = [] + env['JAVASOURCEPATH'] = [] + env['_javapathopt'] = pathopt + env['_JAVABOOTCLASSPATH'] = '${_javapathopt("-bootclasspath", "JAVABOOTCLASSPATH")} ' + env['_JAVACLASSPATH'] = '${_javapathopt("-classpath", "JAVACLASSPATH")} ' + env['_JAVASOURCEPATH'] = '${_javapathopt("-sourcepath", "JAVASOURCEPATH", "_JAVASOURCEPATHDEFAULT")} ' + env['_JAVASOURCEPATHDEFAULT'] = '${TARGET.attributes.java_sourcedir}' + env['_JAVACCOM'] = '$JAVAC $JAVACFLAGS $_JAVABOOTCLASSPATH $_JAVACLASSPATH -d ${TARGET.attributes.java_classdir} $_JAVASOURCEPATH $SOURCES' + env['JAVACCOM'] = "${TEMPFILE('$_JAVACCOM')}" + env['JAVACLASSSUFFIX'] = '.class' + env['JAVASUFFIX'] = '.java' def exists(env): return 1 diff --git a/src/engine/SCons/Tool/javac.xml b/src/engine/SCons/Tool/javac.xml index f5af975..ee93685 100644 --- a/src/engine/SCons/Tool/javac.xml +++ b/src/engine/SCons/Tool/javac.xml @@ -14,6 +14,7 @@ JAVACFLAGS JAVACCOM JAVACLASSSUFFIX JAVASUFFIX +JAVABOOTCLASSPATH JAVACLASSPATH JAVASOURCEPATH </sets> @@ -70,6 +71,19 @@ env.Java(target = 'classes', source = ['File1.java', 'File2.java']) </summary> </builder> +<cvar name="JAVABOOTCLASSPATH"> +<summary> +Specifies the list of directories that +will be added to the +&javac; command line +via the <option>-bootclasspath</option> option. +The individual directory names will be +separated by the operating system's path separate character +(<filename>:</filename> on UNIX/Linux/POSIX, +<filename>;</filename> on Windows). +</summary> +</cvar> + <cvar name="JAVAC"> <summary> The Java compiler. diff --git a/src/engine/SCons/Tool/midl.py b/src/engine/SCons/Tool/midl.py index 811d573..a025701 100644 --- a/src/engine/SCons/Tool/midl.py +++ b/src/engine/SCons/Tool/midl.py @@ -40,7 +40,6 @@ import SCons.Builder import SCons.Defaults import SCons.Scanner.IDL import SCons.Util -import SCons.Tool.msvs def midl_emitter(target, source, env): """Produces a list of outputs from the MIDL compiler""" @@ -80,6 +79,10 @@ def generate(env): env['BUILDERS']['TypeLibrary'] = midl_builder def exists(env): + if not env['PLATFORM'] in ('win32', 'cygwin'): + return 0 + + import SCons.Tool.msvs if SCons.Tool.msvs.is_msvs_installed(): # there's at least one version of MSVS installed, which comes with midl: return 1 diff --git a/src/engine/SCons/Tool/msvs.py b/src/engine/SCons/Tool/msvs.py index 138f920..d4efa74 100644 --- a/src/engine/SCons/Tool/msvs.py +++ b/src/engine/SCons/Tool/msvs.py @@ -1763,6 +1763,9 @@ def generate(env): env['SCONS_HOME'] = os.environ.get('SCONS_HOME') def exists(env): + if not env['PLATFORM'] in ('win32', 'cygwin'): + return 0 + try: v = SCons.Tool.msvs.get_visualstudio_versions() except (SCons.Util.RegError, SCons.Errors.InternalError): diff --git a/src/engine/SCons/Tool/msvsTests.py b/src/engine/SCons/Tool/msvsTests.py index b1125e7..8296287 100644 --- a/src/engine/SCons/Tool/msvsTests.py +++ b/src/engine/SCons/Tool/msvsTests.py @@ -606,6 +606,7 @@ if __name__ == "__main__": # only makes sense to test this on win32 if sys.platform != 'win32': + sys.stdout.write("NO RESULT for msvsTests.py: '%s' is not win32\n" % sys.platform) sys.exit(0) SCons.Util.RegOpenKeyEx = DummyOpenKeyEx diff --git a/src/engine/SCons/Tool/packaging/__init__.py b/src/engine/SCons/Tool/packaging/__init__.py index e8f6a09..72bbff0 100644 --- a/src/engine/SCons/Tool/packaging/__init__.py +++ b/src/engine/SCons/Tool/packaging/__init__.py @@ -32,6 +32,7 @@ import SCons.Environment from SCons.Options import * from SCons.Errors import * from SCons.Util import is_List, make_path_relative +from SCons.Warnings import warn, Warning import os, imp import SCons.Defaults @@ -96,7 +97,11 @@ def Package(env, target=None, source=None, **kw): try: kw['PACKAGETYPE']=env['PACKAGETYPE'] except KeyError: pass - if not kw.has_key('PACKAGETYPE') or kw['PACKAGETYPE']==None: + if not kw.get('PACKAGETYPE'): + from SCons.Script import GetOption + kw['PACKAGETYPE'] = GetOption('package_type') + + if kw['PACKAGETYPE'] == None: if env['BUILDERS'].has_key('Tar'): kw['PACKAGETYPE']='targz' elif env['BUILDERS'].has_key('Zip'): @@ -175,76 +180,48 @@ def Package(env, target=None, source=None, **kw): targets.extend(env.Alias( 'package', targets )) return targets -def build_source(ss, sources): - for s in ss: - if s.__class__==SCons.Node.FS.Dir: - build_source(s.all_children()) - elif not s.has_builder() and s.__class__==SCons.Node.FS.File: - sources.append(s) - else: - build_source(s.sources) - -def FindSourceFiles(env, target=None, source=None ): - """ returns a list of all children of the target nodes, which have no - children. This selects all leaves of the DAG that gets build by SCons for - handling dependencies. - """ - if target==None: target = '.' - - nodes = env.arg2nodes(target, env.fs.Entry) - - sources = [] - for node in nodes: - build_source(node.all_children(), sources) - - # now strip the build_node from the sources by calling the srcnode - # function - def get_final_srcnode(file): - srcnode = file.srcnode() - while srcnode != file.srcnode(): - srcnode = file.srcnode() - return srcnode - - # get the final srcnode for all nodes, this means stripping any - # attached build node. - map( get_final_srcnode, sources ) - - # remove duplicates - return list(set(sources)) - -def FindInstalledFiles(env, source=[], target=[]): - """ returns the list of all targets of the Install and InstallAs Builder. - """ - from SCons.Tool import install - return install._INSTALLED_FILES - # # SCons tool initialization functions # + +added = None + def generate(env): + from SCons.Script import AddOption + global added + if not added: + added = 1 + AddOption('--package-type', + dest='package_type', + default=None, + type="string", + action="store", + help='The type of package to create.') + try: env['BUILDERS']['Package'] env['BUILDERS']['Tag'] - env['BUILDERS']['FindSourceFiles'] - env['BUILDERS']['FindInstalledFiles'] except KeyError: env['BUILDERS']['Package'] = Package env['BUILDERS']['Tag'] = Tag - env['BUILDERS']['FindSourceFiles'] = FindSourceFiles - env['BUILDERS']['FindInstalledFiles'] = FindInstalledFiles def exists(env): return 1 +# XXX def options(opts): opts.AddOptions( - EnumOption( [ 'PACKAGETYPE', '--package-type' ], + EnumOption( 'PACKAGETYPE', 'the type of package to create.', None, allowed_values=map( str, __all__ ), ignorecase=2 ) ) +# +# Internal utility functions +# + def copy_attr(f1, f2): """ copies the special packaging file attributes from f1 to f2. """ @@ -254,79 +231,69 @@ def copy_attr(f1, f2): pattrs = filter(copyit, dir(f1)) for attr in pattrs: setattr(f2, attr, getattr(f1, attr)) -# -# Emitter functions which are reused by the various packagers -# -def packageroot_emitter(pkg_root, honor_install_location=1): - """ creates the packageroot emitter. - - The package root emitter uses the CopyAs builder to copy all source files - to the directory given in pkg_root. +def putintopackageroot(target, source, env, pkgroot, honor_install_location=1): + """ Uses the CopyAs builder to copy all source files to the directory given + in pkgroot. If honor_install_location is set and the copied source file has an PACKAGING_INSTALL_LOCATION attribute, the PACKAGING_INSTALL_LOCATION is - used as the new name of the source file under pkg_root. + used as the new name of the source file under pkgroot. - The source file will not be copied if it is already under the the pkg_root + The source file will not be copied if it is already under the the pkgroot directory. All attributes of the source file will be copied to the new file. """ - def package_root_emitter(target, source, env, pkg_root=pkg_root, honor_install_location=honor_install_location): - pkgroot = pkg_root - # make sure the packageroot is a Dir object. - if SCons.Util.is_String(pkgroot): pkgroot=env.Dir(pkgroot) - - def copy_file_to_pkg_root(file, env=env, pkgroot=pkgroot, honor_install_location=honor_install_location): - if file.is_under(pkgroot): - return file - else: - if hasattr(file, 'PACKAGING_INSTALL_LOCATION') and\ - honor_install_location: - new_name=make_path_relative(file.PACKAGING_INSTALL_LOCATION) - else: - new_name=make_path_relative(file.get_path()) + # make sure the packageroot is a Dir object. + if SCons.Util.is_String(pkgroot): pkgroot=env.Dir(pkgroot) + if not SCons.Util.is_List(source): source=[source] - new_file=pkgroot.File(new_name) - new_file=env.CopyAs(new_file, file)[0] + new_source = [] + for file in source: + if SCons.Util.is_String(file): file = env.File(file) - copy_attr(file, new_file) + if file.is_under(pkgroot): + new_source.append(file) + else: + if hasattr(file, 'PACKAGING_INSTALL_LOCATION') and\ + honor_install_location: + new_name=make_path_relative(file.PACKAGING_INSTALL_LOCATION) + else: + new_name=make_path_relative(file.get_path()) - return new_file - return (target, map(copy_file_to_pkg_root, source)) - return package_root_emitter + new_file=pkgroot.File(new_name) + new_file=env.CopyAs(new_file, file)[0] + copy_attr(file, new_file) + new_source.append(new_file) -from SCons.Warnings import warn, Warning + return (target, new_source) -def stripinstall_emitter(): - """ create the a emitter which: - * strips of the Install Builder of the source target, and stores the - install location as the "PACKAGING_INSTALL_LOCATION" of the given source - File object. This effectively avoids having to execute the Install - Action while storing the needed install location. - * warns about files that are mangled by this emitter which have no - Install Builder. +def stripinstallbuilder(target, source, env): + """ strips the install builder action from the source list and stores + the final installation location as the "PACKAGING_INSTALL_LOCATION" of + the source of the source file. This effectively removes the final installed + files from the source list while remembering the installation location. + + It also warns about files which have no install builder attached. """ - def strip_install_emitter(target, source, env): - def has_no_install_location(file): - return not (file.has_builder() and\ - hasattr(file.builder, 'name') and\ - (file.builder.name=="InstallBuilder" or\ - file.builder.name=="InstallAsBuilder")) - - if len(filter(has_no_install_location, source)): - warn(Warning, "there are file to package which have no\ - InstallBuilder attached, this might lead to irreproducible packages") - - n_source=[] - for s in source: - if has_no_install_location(s): - n_source.append(s) - else: - for ss in s.sources: - n_source.append(ss) - copy_attr(s, ss) - setattr(ss, 'PACKAGING_INSTALL_LOCATION', s.get_path()) + def has_no_install_location(file): + return not (file.has_builder() and\ + hasattr(file.builder, 'name') and\ + (file.builder.name=="InstallBuilder" or\ + file.builder.name=="InstallAsBuilder")) + + if len(filter(has_no_install_location, source)): + warn(Warning, "there are files to package which have no\ + InstallBuilder attached, this might lead to irreproducible packages") + + n_source=[] + for s in source: + if has_no_install_location(s): + n_source.append(s) + else: + for ss in s.sources: + n_source.append(ss) + copy_attr(s, ss) + setattr(ss, 'PACKAGING_INSTALL_LOCATION', s.get_path()) - return (target, n_source) - return strip_install_emitter + return (target, n_source) diff --git a/src/engine/SCons/Tool/packaging/ipk.py b/src/engine/SCons/Tool/packaging/ipk.py index 267fe3c..e3ae277 100644 --- a/src/engine/SCons/Tool/packaging/ipk.py +++ b/src/engine/SCons/Tool/packaging/ipk.py @@ -30,7 +30,7 @@ import SCons.Builder import SCons.Node.FS import os -from SCons.Tool.packaging import stripinstall_emitter, packageroot_emitter +from SCons.Tool.packaging import stripinstallbuilder, putintopackageroot def package(env, target, source, PACKAGEROOT, NAME, VERSION, DESCRIPTION, SUMMARY, X_IPK_PRIORITY, X_IPK_SECTION, SOURCE_URL, @@ -42,8 +42,8 @@ def package(env, target, source, PACKAGEROOT, NAME, VERSION, DESCRIPTION, # setup the Ipkg builder bld = env['BUILDERS']['Ipkg'] - bld.push_emitter(packageroot_emitter(PACKAGEROOT)) - bld.push_emitter(stripinstall_emitter()) + target, source = stripinstallbuilder(target, source, env) + target, source = putintopackageroot(target, source, env, PACKAGEROOT) # This should be overridable from the construction environment, # which it is by using ARCHITECTURE=. @@ -85,7 +85,7 @@ def gen_ipk_dir(proot, source, env, kw): # create the specfile builder s_bld=SCons.Builder.Builder( action = build_specfiles, - emitter = [stripinstall_emitter(), packageroot_emitter(proot)]) + ) # create the specfile targets spec_target=[] diff --git a/src/engine/SCons/Tool/packaging/msi.py b/src/engine/SCons/Tool/packaging/msi.py index 7fc7892..1ce2b1a 100644 --- a/src/engine/SCons/Tool/packaging/msi.py +++ b/src/engine/SCons/Tool/packaging/msi.py @@ -36,7 +36,7 @@ from SCons.Builder import Builder from xml.dom.minidom import * from xml.sax.saxutils import escape -from SCons.Tool.packaging import stripinstall_emitter +from SCons.Tool.packaging import stripinstallbuilder # # Utility functions @@ -223,7 +223,7 @@ def build_wxsfile(target, source, env): # # setup function # -def create_default_directory_layout(root, NAME, VERSION, vendor, filename_set): +def create_default_directory_layout(root, NAME, VERSION, VENDOR, filename_set): """ Create the wix default target directory layout and return the innermost directory. @@ -231,7 +231,7 @@ def create_default_directory_layout(root, NAME, VERSION, vendor, filename_set): the Product tag. Everything is put under the PFiles directory property defined by WiX. - After that a directory with the 'vendor' tag is placed and then a + After that a directory with the 'VENDOR' tag is placed and then a directory with the name of the project and its VERSION. This leads to the following TARGET Directory Layout: C:\<PFiles>\<Vendor>\<Projectname-Version>\ @@ -247,9 +247,9 @@ def create_default_directory_layout(root, NAME, VERSION, vendor, filename_set): d2.attributes['Name'] = 'PFiles' d3 = doc.createElement( 'Directory' ) - d3.attributes['Id'] = 'vendor_folder' - d3.attributes['Name'] = escape( gen_dos_short_file_name( vendor, filename_set ) ) - d3.attributes['LongName'] = escape( vendor ) + d3.attributes['Id'] = 'VENDOR_folder' + d3.attributes['Name'] = escape( gen_dos_short_file_name( VENDOR, filename_set ) ) + d3.attributes['LongName'] = escape( VENDOR ) d4 = doc.createElement( 'Directory' ) project_folder = "%s-%s" % ( NAME, VERSION ) @@ -268,7 +268,7 @@ def create_default_directory_layout(root, NAME, VERSION, vendor, filename_set): # # mandatory and optional file tags # -def build_wxsfile_file_section(root, files, NAME, VERSION, vendor, filename_set, id_set): +def build_wxsfile_file_section(root, files, NAME, VERSION, VENDOR, filename_set, id_set): """ builds the Component sections of the wxs file with their included files. Files need to be specified in 8.3 format and in the long name format, long @@ -276,7 +276,7 @@ def build_wxsfile_file_section(root, files, NAME, VERSION, vendor, filename_set, Features are specficied with the 'X_MSI_FEATURE' or 'DOC' FileTag. """ - root = create_default_directory_layout( root, NAME, VERSION, vendor, filename_set ) + root = create_default_directory_layout( root, NAME, VERSION, VENDOR, filename_set ) components = create_feature_dict( files ) factory = Document() @@ -470,7 +470,7 @@ def build_wxsfile_header_section(root, spec): # mandatory sections, will throw a KeyError if the tag is not available Product.attributes['Name'] = escape( spec['NAME'] ) Product.attributes['Version'] = escape( spec['VERSION'] ) - Product.attributes['Manufacturer'] = escape( spec['vendor'] ) + Product.attributes['Manufacturer'] = escape( spec['VENDOR'] ) Product.attributes['Language'] = escape( spec['X_MSI_LANGUAGE'] ) Package.attributes['Description'] = escape( spec['SUMMARY'] ) @@ -490,12 +490,11 @@ def build_wxsfile_header_section(root, spec): # this builder is the entry-point for .wxs file compiler. wxs_builder = Builder( - action = Action( build_wxsfile, string_wxsfile ), - emitter = stripinstall_emitter(), - suffix = '.wxs' ) + action = Action( build_wxsfile, string_wxsfile ), + ensure_suffix = '.wxs' ) def package(env, target, source, PACKAGEROOT, NAME, VERSION, - DESCRIPTION, SUMMARY, **kw): + DESCRIPTION, SUMMARY, VENDOR, X_MSI_LANGUAGE, **kw): # make sure that the Wix Builder is in the environment SCons.Tool.Tool('wix').generate(env) @@ -507,6 +506,9 @@ def package(env, target, source, PACKAGEROOT, NAME, VERSION, kw.update(loc) del kw['source'], kw['target'], kw['env'] + # strip the install builder from the source files + target, source = stripinstallbuilder(target, source, env) + # put the arguments into the env and call the specfile builder. env['msi_spec'] = kw specfile = apply( wxs_builder, [env, target, source], kw ) diff --git a/src/engine/SCons/Tool/packaging/packager.py b/src/engine/SCons/Tool/packaging/packager.py deleted file mode 100644 index 7aca2b6..0000000 --- a/src/engine/SCons/Tool/packaging/packager.py +++ /dev/null @@ -1,218 +0,0 @@ -"""SCons.Tool.Packaging.packager -""" - -# -# __COPYRIGHT__ -# -# Permission is hereby granted, free of charge, to any person obtaining -# a copy of this software and associated documentation files (the -# "Software"), to deal in the Software without restriction, including -# without limitation the rights to use, copy, modify, merge, publish, -# distribute, sublicense, and/or sell copies of the Software, and to -# permit persons to whom the Software is furnished to do so, subject to -# the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY -# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE -# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" - -import os -import SCons.Defaults -from SCons.Util import strip_abs_path - -class Packager: - """ abstract superclass of all packagers. - - Defines the minimal set of function which need to be implemented in order - to create a new packager. - """ - def __init__(self): - self.specfile_suffix = '.spec' - - def create_builder(self, env, kw): - raise Exception( "%s does not implement create_builder()" % (self.__class__.__name_) ) - - def add_targets(self, kw): - """ In the absence of a target this method creates a default one of - the spec given in the kw argument. - """ - if not kw.has_key('target'): - NAME, VERSION = kw['projectname'], kw['version'] - kw['target'] = [ "%s-%s"%(NAME,VERSION) ] - - return kw - - def strip_install_emitter(self, target, source, env): - """ this emitter assert that all files in source have an InstallBuilder - attached. We take their sources and copy over the file tags, so the - the builder this emitter is attached to is independent of the *installed* - files. - """ - tag_factories = [ LocationTagFactory() ] - - def has_no_install_location(file): - return not ( file.has_builder() and (file.builder.name == 'InstallBuilder' or file.builder.name == 'InstallAsBuilder') ) - - # check if all source file belong into this package. - files = filter( has_no_install_location, source ) - if len( files ) != 0: - raise SCons.Errors.UserError( "There are files which have no Install() Builder attached and are therefore not packageable\n%s" % map( lambda x: x.get_path(), files ) ) - - # All source files have an InstallBuilder attached and we don't want our - # package to be dependent on the *installed* files but only on the - # files that will be installed. Therefore we only care for sources of - # the files in the source list. - n_source = [] - - for s in source: - n_s = s.sources[0] - n_s.set_tags( s.get_tags(tag_factories) ) - n_source.append( n_s ) - - return ( target, n_source ) - - -class BinaryPackager(Packager): - """ abstract superclass for all packagers creating a binary package. - - Binary packagers are seperated from source packager by their requirement to - create a specfile or manifest file. This file contains the contents of the - binary packager together with some information about specific files. - - This superclass provides two needed facilities: - * its specfile_emitter function sets up the correct list of source file - and warns about files with no InstallBuilder attached. - """ - def create_specfile_targets(self, kw): - """ returns the specfile target name(s). - - This function is called by specfile_emitter to find out the specfiles - target name. - """ - p, v = kw['NAME'], kw['VERSION'] - return '%s-%s' % ( p, v ) - - def specfile_emitter(self, target, source, env): - """ adds the to build specfile to the source list. - """ - # create a specfile action that is executed for building the specfile - specfile_action = SCons.Action.Action( self.build_specfile, - self.string_specfile, - varlist=[ 'SPEC' ] ) - - # create a specfile Builder with the right sources attached. - specfile_builder = SCons.Builder.Builder( action = self.build_specfile, - suffix = self.specfile_suffix ) - - specfile = apply( specfile_builder, [ env ], { - 'target' : self.create_specfile_targets(env), - 'source' : source } ) - - specfile.extend( source ) - return ( target, specfile ) - - def string_specfile(self, target, source, env): - return "building specfile %s"%(target[0].abspath) - - def build_specfile(self, target, source, env): - """ this function is called to build the specfile of name "target" - from the source list and the settings in "env" - """ - raise Exception( 'class does not implement build_specfile()' ) - -class SourcePackager(Packager): - """ abstract superclass for all packagers which generate a source package. - - They are seperated from other packager by the their package_root attribute. - Since before a source package is created with the help of a Tar or Zip - builder their content needs to be moved to a package_root. For example the - project foo with VERSION 1.2.3, will get its files placed in foo-1.2.3/. - """ - def create_package_root(self,kw): - """ creates the package_r oot for a given specification dict. - """ - try: - return kw['package_root'] - except KeyError: - NAME, VERSION = kw['projectname'], kw['version'] - return "%s-%s"%(NAME,VERSION) - - def package_root_emitter(self, pkg_root, honor_install_location=1): - def package_root_emitter(target, source, env): - """ This emitter copies the sources to the src_package_root directory: - * if a source has an install_location, not its original name is - used but the one specified in the 'install_location' tag. - * else its original name is used. - * if the source file is already in the src_package_root directory, - nothing will be done. - """ - new_source = [] - for s in source: - if os.path.dirname(s.get_path()).rfind(pkg_root) != -1: - new_source.append(s) - else: - tags = s.get_tags() - new_s = None - - if tags.has_key( 'install_location' ) and honor_install_location: - my_target = strip_abs_path(tags['install_location']) - else: - my_target = strip_abs_path(s.get_path()) - - new_s = env.CopyAs( os.path.join( pkg_root, my_target ), s )[0] - - # store the tags of our original file in the new file. - new_s.set_tags( s.get_tags() ) - new_source.append( new_s ) - - return (target, new_source) - - return package_root_emitter - -class TagFactory: - """An instance of this class has the responsibility to generate additional - tags for a SCons.Node.FS.File instance. - - Subclasses have to be callable. This class definition is informally - describing the interface. - """ - - def __call__(self, file, current_tag_dict): - """ This call has to return additional tags in the form of a dict. - """ - pass - - def attach_additional_info(self, info=None): - pass - -class LocationTagFactory(TagFactory): - """ This class creates the "location" tag, which describes the install - location of a given file. - - This is done by analyzing the builder of a given file for a InstallBuilder, - from this builder the install location is deduced. - """ - - def __call__(self, file, current_tag_dict): - if current_tag_dict.has_key('install_location'): - return {} - - if file.has_builder() and\ - (file.builder.name == "InstallBuilder" or\ - file.builder.name == "InstallAsBuilder") and\ - file.has_explicit_builder(): - return { 'install_location' : file.get_path() } - else: - return {} - - diff --git a/src/engine/SCons/Tool/packaging/rpm.py b/src/engine/SCons/Tool/packaging/rpm.py index 94b7b7a..5f97608 100644 --- a/src/engine/SCons/Tool/packaging/rpm.py +++ b/src/engine/SCons/Tool/packaging/rpm.py @@ -33,7 +33,9 @@ import string import SCons.Builder -from SCons.Tool.packaging import stripinstall_emitter, packageroot_emitter, src_targz +from SCons.Environment import OverrideEnvironment +from SCons.Tool.packaging import stripinstallbuilder, src_targz +from SCons.Errors import UserError def package(env, target, source, PACKAGEROOT, NAME, VERSION, PACKAGEVERSION, DESCRIPTION, SUMMARY, X_RPM_GROUP, LICENSE, @@ -43,12 +45,13 @@ def package(env, target, source, PACKAGEROOT, NAME, VERSION, bld = env['BUILDERS']['Rpm'] - bld.push_emitter(targz_emitter) - bld.push_emitter(specfile_emitter) - bld.push_emitter(stripinstall_emitter()) - - # override the default target, with the rpm specific ones. - if str(target[0])=="%s-%s"%(NAME, VERSION): + # Generate a UserError whenever the target name has been set explicitly, + # since rpm does not allow for controlling it. This is detected by + # checking if the target has been set to the default by the Package() + # Environment function. + if str(target[0])!="%s-%s"%(NAME, VERSION): + raise UserError( "Setting target is not supported for rpm." ) + else: # This should be overridable from the construction environment, # which it is by using ARCHITECTURE=. # Guessing based on what os.uname() returns at least allows it @@ -79,13 +82,19 @@ def package(env, target, source, PACKAGEROOT, NAME, VERSION, # if no "SOURCE_URL" tag is given add a default one. if not kw.has_key('SOURCE_URL'): - kw['SOURCE_URL']=(str(target[0])+".tar.gz").replace('.rpm', '') + #kw['SOURCE_URL']=(str(target[0])+".tar.gz").replace('.rpm', '') + kw['SOURCE_URL']=string.replace(str(target[0])+".tar.gz", '.rpm', '') + + # mangle the source and target list for the rpmbuild + env = OverrideEnvironment(env, kw) + target, source = stripinstallbuilder(target, source, env) + target, source = addspecfile(target, source, env) + target, source = collectintargz(target, source, env) # now call the rpm builder to actually build the packet. return apply(bld, [env, target, source], kw) - -def targz_emitter(target, source, env): +def collectintargz(target, source, env): """ Puts all source files into a tar.gz file. """ # the rpm tool depends on a source package, until this is chagned # this hack needs to be here that tries to pack all sources in. @@ -116,7 +125,7 @@ def targz_emitter(target, source, env): return (target, tarball) -def specfile_emitter(target, source, env): +def addspecfile(target, source, env): specfile = "%s-%s" % (env['NAME'], env['VERSION']) bld = SCons.Builder.Builder(action = build_specfile, @@ -180,7 +189,7 @@ def build_specfile_sections(spec): # Default prep, build, install and clean rules # TODO: optimize those build steps, to not compile the project a second time if not spec.has_key('X_RPM_PREP'): - spec['X_RPM_PREP'] = 'rm -rf "$RPM_BUILD_ROOT"' + '\n%setup -q' + spec['X_RPM_PREP'] = '[ -n "$RPM_BUILD_ROOT" -a "$RPM_BUILD_ROOT" != / ] && rm -rf "$RPM_BUILD_ROOT"' + '\n%setup -q' if not spec.has_key('X_RPM_BUILD'): spec['X_RPM_BUILD'] = 'mkdir "$RPM_BUILD_ROOT"' @@ -189,7 +198,7 @@ def build_specfile_sections(spec): spec['X_RPM_INSTALL'] = 'scons --install-sandbox="$RPM_BUILD_ROOT" "$RPM_BUILD_ROOT"' if not spec.has_key('X_RPM_CLEAN'): - spec['X_RPM_CLEAN'] = 'rm -rf "$RPM_BUILD_ROOT"' + spec['X_RPM_CLEAN'] = '[ -n "$RPM_BUILD_ROOT" -a "$RPM_BUILD_ROOT" != / ] && rm -rf "$RPM_BUILD_ROOT"' str = str + SimpleTagCompiler(optional_sections, mandatory=0).compile( spec ) diff --git a/src/engine/SCons/Tool/packaging/src_tarbz2.py b/src/engine/SCons/Tool/packaging/src_tarbz2.py index 7d876e4..8323107 100644 --- a/src/engine/SCons/Tool/packaging/src_tarbz2.py +++ b/src/engine/SCons/Tool/packaging/src_tarbz2.py @@ -28,10 +28,10 @@ The tarbz2 SRC packager. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -from SCons.Tool.packaging import packageroot_emitter +from SCons.Tool.packaging import putintopackageroot def package(env, target, source, PACKAGEROOT, **kw): bld = env['BUILDERS']['Tar'] bld.set_suffix('.tar.bz2') - bld.push_emitter(packageroot_emitter(PACKAGEROOT, honor_install_location=0)) + target, source = putintopackageroot(target, source, env, PACKAGEROOT, honor_install_location=0) return bld(env, target, source, TARFLAGS='-jc') diff --git a/src/engine/SCons/Tool/packaging/src_targz.py b/src/engine/SCons/Tool/packaging/src_targz.py index d84976e..db51279 100644 --- a/src/engine/SCons/Tool/packaging/src_targz.py +++ b/src/engine/SCons/Tool/packaging/src_targz.py @@ -28,10 +28,10 @@ The targz SRC packager. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -from SCons.Tool.packaging import packageroot_emitter +from SCons.Tool.packaging import putintopackageroot def package(env, target, source, PACKAGEROOT, **kw): bld = env['BUILDERS']['Tar'] bld.set_suffix('.tar.gz') - bld.push_emitter(packageroot_emitter(PACKAGEROOT, honor_install_location=0)) + target, source = putintopackageroot(target, source, env, PACKAGEROOT, honor_install_location=0) return bld(env, target, source, TARFLAGS='-zc') diff --git a/src/engine/SCons/Tool/packaging/src_zip.py b/src/engine/SCons/Tool/packaging/src_zip.py index d60fe85..01bd42e 100644 --- a/src/engine/SCons/Tool/packaging/src_zip.py +++ b/src/engine/SCons/Tool/packaging/src_zip.py @@ -28,10 +28,10 @@ The zip SRC packager. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -from SCons.Tool.packaging import packageroot_emitter +from SCons.Tool.packaging import putintopackageroot def package(env, target, source, PACKAGEROOT, **kw): bld = env['BUILDERS']['Zip'] bld.set_suffix('.zip') - bld.push_emitter(packageroot_emitter(PACKAGEROOT, honor_install_location=0)) + target, source = putintopackageroot(target, source, env, PACKAGEROOT, honor_install_location=0) return bld(env, target, source) diff --git a/src/engine/SCons/Tool/packaging/tarbz2.py b/src/engine/SCons/Tool/packaging/tarbz2.py index 7127896..52252da 100644 --- a/src/engine/SCons/Tool/packaging/tarbz2.py +++ b/src/engine/SCons/Tool/packaging/tarbz2.py @@ -28,11 +28,11 @@ The tarbz2 SRC packager. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -from SCons.Tool.packaging import stripinstall_emitter, packageroot_emitter +from SCons.Tool.packaging import stripinstallbuilder, putintopackageroot def package(env, target, source, PACKAGEROOT, **kw): bld = env['BUILDERS']['Tar'] bld.set_suffix('.tar.gz') - bld.push_emitter(packageroot_emitter(PACKAGEROOT)) - bld.push_emitter(stripinstall_emitter()) + target, source = putintopackageroot(target, source, env, PACKAGEROOT) + target, source = stripinstallbuilder(target, source, env) return bld(env, target, source, TARFLAGS='-jc') diff --git a/src/engine/SCons/Tool/packaging/targz.py b/src/engine/SCons/Tool/packaging/targz.py index 798e570..4713840 100644 --- a/src/engine/SCons/Tool/packaging/targz.py +++ b/src/engine/SCons/Tool/packaging/targz.py @@ -28,11 +28,11 @@ The targz SRC packager. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -from SCons.Tool.packaging import stripinstall_emitter, packageroot_emitter +from SCons.Tool.packaging import stripinstallbuilder, putintopackageroot def package(env, target, source, PACKAGEROOT, **kw): bld = env['BUILDERS']['Tar'] bld.set_suffix('.tar.gz') - bld.push_emitter(packageroot_emitter(PACKAGEROOT)) - bld.push_emitter(stripinstall_emitter()) + target, source = stripinstallbuilder(target, source, env) + target, source = putintopackageroot(target, source, env, PACKAGEROOT) return bld(env, target, source, TARFLAGS='-zc') diff --git a/src/engine/SCons/Tool/packaging/zip.py b/src/engine/SCons/Tool/packaging/zip.py index 7663a4a..639d569 100644 --- a/src/engine/SCons/Tool/packaging/zip.py +++ b/src/engine/SCons/Tool/packaging/zip.py @@ -28,11 +28,11 @@ The zip SRC packager. __revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" -from SCons.Tool.packaging import stripinstall_emitter, packageroot_emitter +from SCons.Tool.packaging import stripinstallbuilder, putintopackageroot def package(env, target, source, PACKAGEROOT, **kw): bld = env['BUILDERS']['Zip'] bld.set_suffix('.zip') - bld.push_emitter(packageroot_emitter(PACKAGEROOT)) - bld.push_emitter(stripinstall_emitter()) + target, source = stripinstallbuilder(target, source, env) + target, source = putintopackageroot(target, source, env, PACKAGEROOT) return bld(env, target, source) diff --git a/src/engine/SCons/Tool/rpm.py b/src/engine/SCons/Tool/rpm.py index 2fc4b58..47759ea 100644 --- a/src/engine/SCons/Tool/rpm.py +++ b/src/engine/SCons/Tool/rpm.py @@ -103,7 +103,6 @@ def string_rpm(target, source, env): rpmAction = SCons.Action.Action(build_rpm, string_rpm) RpmBuilder = SCons.Builder.Builder(action = SCons.Action.Action('$RPMCOM', '$RPMCOMSTR'), - source_factory = SCons.Node.FS.Entry, source_scanner = SCons.Defaults.DirScanner, suffix = '$RPMSUFFIX') diff --git a/src/engine/SCons/Tool/sgic++.py b/src/engine/SCons/Tool/sgic++.py index 5461c4d..930e29a 100644 --- a/src/engine/SCons/Tool/sgic++.py +++ b/src/engine/SCons/Tool/sgic++.py @@ -44,7 +44,7 @@ def generate(env): env['CXX'] = 'CC' env['CXXFLAGS'] = SCons.Util.CLVar('$CCFLAGS -LANG:std') - env['SHCXX'] = 'CC' + env['SHCXX'] = '$CXX' env['SHOBJSUFFIX'] = '.o' env['STATIC_AND_SHARED_OBJECTS_ARE_THE_SAME'] = 1 diff --git a/src/engine/SCons/Tool/tex.py b/src/engine/SCons/Tool/tex.py index 0e3d4b1..bbae25e 100644 --- a/src/engine/SCons/Tool/tex.py +++ b/src/engine/SCons/Tool/tex.py @@ -51,6 +51,7 @@ undefined_references_str = '(^LaTeX Warning:.*undefined references)|(^Package \w undefined_references_re = re.compile(undefined_references_str, re.MULTILINE) openout_aux_re = re.compile(r"\\openout.*`(.*\.aux)'") +openout_re = re.compile(r"\\openout.*`(.*)'") makeindex_re = re.compile(r"^[^%]*\\makeindex", re.MULTILINE) tableofcontents_re = re.compile(r"^[^%]*\\tableofcontents", re.MULTILINE) @@ -193,17 +194,18 @@ def tex_emitter(target, source, env): env.Precious(base + '.bbl') target.append(base + '.blg') - # read log file to get all .aux files + # read log file to get all output file (include .aux files) logfilename = base + '.log' dir, base_nodir = os.path.split(base) if os.path.exists(logfilename): content = open(logfilename, "rb").read() - aux_files = openout_aux_re.findall(content) - aux_files = filter(lambda f, b=base_nodir+'.aux': f != b, aux_files) - aux_files = map(lambda f, d=dir: d+os.sep+f, aux_files) - target.extend(aux_files) - for a in aux_files: - env.Precious( a ) + out_files = openout_re.findall(content) + out_files = filter(lambda f, b=base_nodir+'.aux': f != b, out_files) + if dir != '': + out_files = map(lambda f, d=dir: d+os.sep+f, out_files) + target.extend(out_files) + for f in out_files: + env.Precious( f ) return (target, source) diff --git a/src/engine/SCons/Tool/wix.py b/src/engine/SCons/Tool/wix.py index 3aa3375..8d6d3fd 100644 --- a/src/engine/SCons/Tool/wix.py +++ b/src/engine/SCons/Tool/wix.py @@ -50,12 +50,12 @@ def generate(env): env['WIXLIGHTCOM'] = "$WIXLIGHT $WIXLIGHTFLAGS -out ${TARGET} ${SOURCES}" object_builder = SCons.Builder.Builder( - action = '$WIXCANDLECOM', - suffix = '.wxiobj', - src_suffix = '.wxs') + action = '$WIXCANDLECOM', + suffix = '.wxiobj', + src_suffix = '.wxs') linker_builder = SCons.Builder.Builder( - action = '$WIXLIGHTCOM', + action = '$WIXLIGHTCOM', src_suffix = '.wxiobj', src_builder = object_builder) diff --git a/src/engine/SCons/Util.py b/src/engine/SCons/Util.py index 3dfa287..04d263b 100644 --- a/src/engine/SCons/Util.py +++ b/src/engine/SCons/Util.py @@ -51,6 +51,11 @@ ListType = types.ListType StringType = types.StringType TupleType = types.TupleType +def dictify(keys, values, result={}): + for k, v in zip(keys, values): + result[k] = v + return result + _altsep = os.altsep if _altsep is None and sys.platform == 'win32': # My ActivePython 2.0.1 doesn't set os.altsep! What gives? @@ -300,7 +305,7 @@ def print_tree(root, child_func, prune=0, showtags=0, margin=[0], visited={}): tags.append(' S'[IDX(root.side_effect)]) tags.append(' P'[IDX(root.precious)]) tags.append(' A'[IDX(root.always_build)]) - tags.append(' C'[IDX(root.current())]) + tags.append(' C'[IDX(root.is_up_to_date())]) tags.append(' N'[IDX(root.noclean)]) tags.append(' H'[IDX(root.nocache)]) tags.append(']') @@ -886,17 +891,26 @@ else: def case_sensitive_suffixes(s1, s2): return (os.path.normcase(s1) != os.path.normcase(s2)) -def adjustixes(fname, pre, suf): +def adjustixes(fname, pre, suf, ensure_suffix=False): if pre: path, fn = os.path.split(os.path.normpath(fname)) if fn[:len(pre)] != pre: fname = os.path.join(path, pre + fn) - # Only append a suffix if the file does not have one. - if suf and not splitext(fname)[1] and fname[-len(suf):] != suf: + # Only append a suffix if the suffix we're going to add isn't already + # there, and if either we've been asked to ensure the specific suffix + # is present or there's no suffix on it at all. + if suf and fname[-len(suf):] != suf and \ + (ensure_suffix or not splitext(fname)[1]): fname = fname + suf return fname + +# From Tim Peters, +# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560 +# ASPN: Python Cookbook: Remove duplicates from a sequence +# (Also in the printed Python Cookbook.) + def unique(s): """Return a list of the elements in s, but without duplicates. @@ -967,6 +981,45 @@ def unique(s): u.append(x) return u + + +# From Alex Martelli, +# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560 +# ASPN: Python Cookbook: Remove duplicates from a sequence +# First comment, dated 2001/10/13. +# (Also in the printed Python Cookbook.) + +def uniquer(seq, idfun=None): + if idfun is None: + def idfun(x): return x + seen = {} + result = [] + for item in seq: + marker = idfun(item) + # in old Python versions: + # if seen.has_key(marker) + # but in new ones: + if marker in seen: continue + seen[marker] = 1 + result.append(item) + return result + +# A more efficient implementation of Alex's uniquer(), this avoids the +# idfun() argument and function-call overhead by assuming that all +# items in the sequence are hashable. + +def uniquer_hashables(seq): + seen = {} + result = [] + for item in seq: + #if not item in seen: + if not seen.has_key(item): + seen[item] = 1 + result.append(item) + return result + + + # Much of the logic here was originally based on recipe 4.9 from the # Python CookBook, but we had to dumb it way down for Python 1.5.2. class LogicalLines: @@ -996,6 +1049,8 @@ class LogicalLines: result.append(line) return result + + class Unbuffered: """ A proxy class that wraps a file object, flushing after every write, @@ -1101,6 +1156,35 @@ def RenameFunction(function, name): func_defaults) +md5 = False +def MD5signature(s): + return str(s) + +try: + import hashlib +except ImportError: + pass +else: + if hasattr(hashlib, 'md5'): + md5 = True + def MD5signature(s): + m = hashlib.md5() + m.update(str(s)) + return m.hexdigest() + +def MD5collect(signatures): + """ + Collects a list of signatures into an aggregate signature. + + signatures - a list of signatures + returns - the aggregate signature + """ + if len(signatures) == 1: + return signatures[0] + else: + return MD5signature(string.join(signatures, ', ')) + + # From Dinu C. Gherman, # Python Cookbook, second edition, recipe 6.17, p. 277. diff --git a/src/engine/SCons/UtilTests.py b/src/engine/SCons/UtilTests.py index 1149f35..3e8085b 100644 --- a/src/engine/SCons/UtilTests.py +++ b/src/engine/SCons/UtilTests.py @@ -46,6 +46,18 @@ class OutBuffer: def write(self, str): self.buffer = self.buffer + str +class dictifyTestCase(unittest.TestCase): + def test_dictify(self): + """Test the dictify() function""" + r = SCons.Util.dictify(['a', 'b', 'c'], [1, 2, 3]) + assert r == {'a':1, 'b':2, 'c':3}, r + + r = {} + SCons.Util.dictify(['a'], [1], r) + SCons.Util.dictify(['b'], [2], r) + SCons.Util.dictify(['c'], [3], r) + assert r == {'a':1, 'b':2, 'c':3}, r + class UtilTestCase(unittest.TestCase): def test_splitext(self): assert splitext('foo') == ('foo','') @@ -73,7 +85,7 @@ class UtilTestCase(unittest.TestCase): return 1 def always_build(self): return 1 - def current(self): + def is_up_to_date(self): return 1 def noclean(self): return 1 @@ -687,7 +699,33 @@ bling 'bling\n', ], lines +class MD5TestCase(unittest.TestCase): + + def test_collect(self): + """Test collecting a list of signatures into a new signature value + """ + s = map(MD5signature, ('111', '222', '333')) + + assert '698d51a19d8a121ce581499d7b701668' == MD5collect(s[0:1]) + assert '8980c988edc2c78cc43ccb718c06efd5' == MD5collect(s[0:2]) + assert '53fd88c84ff8a285eb6e0a687e55b8c7' == MD5collect(s) + + def test_MD5signature(self): + """Test generating a signature""" + s = MD5signature('111') + assert '698d51a19d8a121ce581499d7b701668' == s, s + + s = MD5signature('222') + assert 'bcbe3365e6ac95ea2c0343a2395834dd' == s, s + if __name__ == "__main__": - suite = unittest.makeSuite(UtilTestCase, 'test_') + suite = unittest.TestSuite() + tclasses = [ dictifyTestCase, + MD5TestCase, + UtilTestCase, + ] + for tclass in tclasses: + names = unittest.getTestCaseNames(tclass, 'test_') + suite.addTests(map(tclass, names)) if not unittest.TextTestRunner().run(suite).wasSuccessful(): sys.exit(1) diff --git a/src/engine/SCons/compat/__init__.py b/src/engine/SCons/compat/__init__.py index 0dfb18e..5e095d1 100644 --- a/src/engine/SCons/compat/__init__.py +++ b/src/engine/SCons/compat/__init__.py @@ -60,7 +60,7 @@ _scons_subprocess.py is our compatibility module for subprocess) so that we can still try to import the real module name and fall back to our compatibility module if we get an ImportError. The import_as() function defined below loads the module as the "real" name (without the -underscore), after which all of the "import {module}" statements in the +'_scons'), after which all of the "import {module}" statements in the rest of our code will find our pre-loaded compatibility module. """ @@ -80,6 +80,19 @@ def import_as(module, name): import builtins try: + import hashlib +except ImportError: + # Pre-2.5 Python has no hashlib module. + try: + import_as('_scons_hashlib', 'hashlib') + except ImportError: + # If we failed importing our compatibility module, it probably + # means this version of Python has no md5 module. Don't do + # anything and let the higher layer discover this fact, so it + # can fall back to using timestamp. + pass + +try: set except NameError: # Pre-2.4 Python has no native set type @@ -112,6 +125,25 @@ except ImportError: # Pre-2.3 Python has no optparse module. import_as('_scons_optparse', 'optparse') +import shlex +try: + shlex.split +except AttributeError: + # Pre-2.3 Python has no shlex.split function. + def split(s, comments=False): + import StringIO + lex = shlex.shlex(StringIO.StringIO(s)) + lex.wordchars = lex.wordchars + '/\\-+,=:' + result = [] + while True: + tt = lex.get_token() + if not tt: + break + result.append(tt) + return result + shlex.split = split + del split + try: import subprocess except ImportError: diff --git a/src/engine/SCons/compat/_scons_hashlib.py b/src/engine/SCons/compat/_scons_hashlib.py new file mode 100644 index 0000000..9445ddb --- /dev/null +++ b/src/engine/SCons/compat/_scons_hashlib.py @@ -0,0 +1,85 @@ +# +# __COPYRIGHT__ +# +# Permission is hereby granted, free of charge, to any person obtaining +# a copy of this software and associated documentation files (the +# "Software"), to deal in the Software without restriction, including +# without limitation the rights to use, copy, modify, merge, publish, +# distribute, sublicense, and/or sell copies of the Software, and to +# permit persons to whom the Software is furnished to do so, subject to +# the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +# + +__doc__ = """ +hashlib backwards-compatibility module for older (pre-2.5) Python versions + +This does not not NOT (repeat, *NOT*) provide complete hashlib +functionality. It only wraps the portions of MD5 functionality used +by SCons, in an interface that looks like hashlib (or enough for our +purposes, anyway). In fact, this module will raise an ImportError if +the underlying md5 module isn't available. +""" + +__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" + +import md5 +import string + +class md5obj: + + md5_module = md5 + + def __init__(self, name, string=''): + if not name in ('MD5', 'md5'): + raise ValueError, "unsupported hash type" + self.name = 'md5' + self.m = self.md5_module.md5() + + def __repr__(self): + return '<%s HASH object @ %#x>' % (self.name, id(self)) + + def copy(self): + import copy + result = copy.copy(self) + result.m = self.m.copy() + return result + + def digest(self): + return self.m.digest() + + def update(self, arg): + return self.m.update(arg) + + if hasattr(md5.md5(), 'hexdigest'): + + def hexdigest(self): + return self.m.hexdigest() + + else: + + # Objects created by the underlying md5 module have no native + # hexdigest() method (*cough* 1.5.2 *cough*), so provide an + # equivalent lifted from elsewhere. + def hexdigest(self): + h = string.hexdigits + r = '' + for c in self.digest(): + i = ord(c) + r = r + h[(i >> 4) & 0xF] + h[i & 0xF] + return r + +new = md5obj + +def md5(string=''): + return md5obj('md5', string) diff --git a/src/engine/SCons/compat/_scons_subprocess.py b/src/engine/SCons/compat/_scons_subprocess.py index df7e702..4cb9e30 100644 --- a/src/engine/SCons/compat/_scons_subprocess.py +++ b/src/engine/SCons/compat/_scons_subprocess.py @@ -373,7 +373,13 @@ class CalledProcessError(Exception): if mswindows: - import threading + try: + import threading + except ImportError: + # SCons: the threading module is only used by the communicate() + # method, which we don't actually use, so don't worry if we + # can't import it. + pass import msvcrt if 0: # <-- change this to use pywin32 instead of the _subprocess driver import pywintypes |