diff options
author | William Deegan <bill@baddogconsulting.com> | 2022-06-14 22:28:08 (GMT) |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-06-14 22:28:08 (GMT) |
commit | dbdd63d8b462879d817a024afaa79875377a8b33 (patch) | |
tree | 4a7f8f3fc928cb03884ce7e280ee354072092cc7 | |
parent | b1ede4ce01b6163086c748f509c51616b9dec051 (diff) | |
parent | 49e50671de9309fd38e5006071332a802161e7c1 (diff) | |
download | SCons-dbdd63d8b462879d817a024afaa79875377a8b33.zip SCons-dbdd63d8b462879d817a024afaa79875377a8b33.tar.gz SCons-dbdd63d8b462879d817a024afaa79875377a8b33.tar.bz2 |
Merge branch 'master' into ninja_determinism
31 files changed, 1059 insertions, 229 deletions
diff --git a/.github/workflows/experimental_tests.yml b/.github/workflows/experimental_tests.yml index 3672144..52078d5 100644 --- a/.github/workflows/experimental_tests.yml +++ b/.github/workflows/experimental_tests.yml @@ -43,8 +43,7 @@ jobs: - name: Install dependencies including ninja ${{ matrix.os }} run: | python -m pip install --upgrade pip setuptools wheel - python -m pip install ninja - # if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + python -m pip install ninja psutil # sudo apt-get update - name: Test experimental packages ${{ matrix.os }} run: | diff --git a/.github/workflows/runtest.yml b/.github/workflows/runtest.yml index cfc585e..dd18013 100644 --- a/.github/workflows/runtest.yml +++ b/.github/workflows/runtest.yml @@ -37,8 +37,7 @@ jobs: - name: Install dependencies including ninja ${{ matrix.os }} run: | python -m pip install --upgrade pip setuptools wheel - python -m pip install ninja - # if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + python -m pip install ninja psutil # sudo apt-get update - name: runtest ${{ matrix.os }} run: | diff --git a/CHANGES.txt b/CHANGES.txt index 4541168..226102f 100755 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -48,6 +48,25 @@ RELEASE VERSION/DATE TO BE FILLED IN LATER require delayed expansion to be enabled which is currently not supported and is typically not enabled by default on the host system. The batch files may also require environment variables that are not included by default in the msvc environment. + - Suppress issuing a warning when there are no installed Visual Studio instances for the default + tools configuration (issue #2813). When msvc is the default compiler because there are no + compilers installed, a build may fail due to the cl.exe command not being recognized. At + present, there is no easy way to detect during msvc initialization if the default environment + will be used later to build a program and/or library. There is no error/warning issued for the + default tools as there are legitimate SCons uses that do not require a c compiler. + - Added a global policy setting and an environment policy variable for specifying the action to + be taken when an msvc request cannot be satisfied. The available options are "error", + "exception", "warning", "warn", "ignore", and "suppress". The global policy variable may be + set and retrieved via the functions set_msvc_notfound_policy and get_msvc_notfound_policy, + respectively. These two methods may be imported from SCons.Tool.MSCommon. The environment + policy variable introduced is MSVC_NOTFOUND_POLICY. When defined, the environment policy + variable overrides the global policy setting for a given environment. When the active policy + is "error" or "exception", an MSVCVersionNotFound exception is raised. When the active policy + is "warning" or "warn", a VisualCMissingWarning warning is issued and the constructed + environment is likely incomplete. When the active policy is "ignore" or "suppress", no action + is taken and the constructed environment is likely incomplete. As implemented, the default + global policy is "warning". The ability to set the global policy via an SCons command-line + option may be added in a future enhancement. From William Deegan: - Fix check for unsupported Python version. It was broken. Also now the error message @@ -118,6 +137,11 @@ RELEASE VERSION/DATE TO BE FILLED IN LATER to connect to the server during start up. - Ninja: added option to skip ninja regeneration if scons can determine the ninja file does not need to be regenerated. + - Ninja: Added new alias "shutdown-ninja-scons-daemon" to allow ninja to shutdown the daemon. + Also added cleanup to test framework to kill ninja scons daemons and clean ip daemon logs. + NOTE: Test for this requires python psutil module. It will be skipped if not present. + - Ninja: Added command line variable NINJA_CMD_ARGS that allows to pass through ninja command line args. + This can also be set in your Environment(). From Mats Wichmann: - Tweak the way default site_scons paths on Windows are expressed to @@ -164,6 +188,10 @@ RELEASE VERSION/DATE TO BE FILLED IN LATER that link has been modified (issue #3880) - Modernize a few tests that use now-deprecated unittest.getTestCaseNames and unittest.makeSuite - Python itself suggests the replacements. + - SCons.Tool.find_program_path now takes an optional add_path argument + to add a path to the execution environment if it was discovered in + default_paths. Previously, the routine, called by many tool modules, + never altered the execution environment, leaving it to the tools. From Zhichang Yu: - Added MSVC_USE_SCRIPT_ARGS variable to pass arguments to MSVC_USE_SCRIPT. diff --git a/RELEASE.txt b/RELEASE.txt index 0dae549..6e0d30d 100755 --- a/RELEASE.txt +++ b/RELEASE.txt @@ -14,7 +14,8 @@ NOTE: If you build with Python 3.10.0 and then rebuild with 3.10.1 (or higher), NEW FUNCTIONALITY ----------------- -- Added MSVC_USE_SCRIPT_ARGS variable to pass arguments to MSVC_USE_SCRIPT. +- Added MSVC_USE_SCRIPT_ARGS Environment variable which specifies command line arguments + to pass to the script specified by MSVC_USE_SCRIPT. - Added Configure.CheckMember() checker to check if struct/class has the specified member. - Added SHELL_ENV_GENERATORS construction variable. This variable should be set to a list (or an iterable) which contains functions to be called in order @@ -24,6 +25,24 @@ NEW FUNCTIONALITY performance impact if not used carefully. - Added MSVC_USE_SETTINGS variable to pass a dictionary to configure the msvc compiler system environment as an alternative to bypassing Visual Studio autodetection entirely. +- Ninja: Added new alias "shutdown-ninja-scons-daemon" to allow ninja to shutdown the daemon. + Also added cleanup to test framework to kill ninja scons daemons and clean ip daemon logs. + NOTE: Test for this requires python psutil module. It will be skipped if not present. +- Ninja: Added command line variable NINJA_CMD_ARGS that allows to pass through ninja command line args. + This can also be set in your Environment(). +- Added a global policy setting and an environment policy variable for specifying the action to + be taken when an msvc request cannot be satisfied. The available options are "error", + "exception", "warning", "warn", "ignore", and "suppress". The global policy variable may be + set and retrieved via the functions set_msvc_notfound_policy and get_msvc_notfound_policy, + respectively. These two methods may be imported from SCons.Tool.MSCommon. The environment + policy variable introduced is MSVC_NOTFOUND_POLICY. When defined, the environment policy + variable overrides the global policy setting for a given environment. When the active policy + is "error" or "exception", an MSVCVersionNotFound exception is raised. When the active policy + is "warning" or "warn", a VisualCMissingWarning warning is issued and the constructed + environment is likely incomplete. When the active policy is "ignore" or "suppress", no action + is taken and the constructed environment is likely incomplete. As implemented, the default + global policy is "warning". The ability to set the global policy via an SCons command-line + option may be added in a future enhancement. DEPRECATED FUNCTIONALITY @@ -136,6 +155,12 @@ FIXES - The system environment variable names imported for MSVC 7.0 and 6.0 were updated to be consistent with the variables names defined by their respective installers. This fixes an error caused when bypassing MSVC detection by specifying the MSVC 7.0 batch file directly. +- Suppress issuing a warning when there are no installed Visual Studio instances for the default + tools configuration (issue #2813). When msvc is the default compiler because there are no + compilers installed, a build may fail due to the cl.exe command not being recognized. At + present, there is no easy way to detect during msvc initialization if the default environment + will be used later to build a program and/or library. There is no error/warning issued for the + default tools as there are legitimate SCons uses that do not require a c compiler. IMPROVEMENTS ------------ diff --git a/SCons/Tool/MSCommon/__init__.py b/SCons/Tool/MSCommon/__init__.py index 50eed73..9d8a8ff 100644 --- a/SCons/Tool/MSCommon/__init__.py +++ b/SCons/Tool/MSCommon/__init__.py @@ -34,10 +34,12 @@ from SCons.Tool.MSCommon.sdk import mssdk_exists, mssdk_setup_env from SCons.Tool.MSCommon.vc import ( msvc_exists, - msvc_setup_env, + msvc_setup_env_tool, msvc_setup_env_once, msvc_version_to_maj_min, msvc_find_vswhere, + set_msvc_notfound_policy, + get_msvc_notfound_policy, ) from SCons.Tool.MSCommon.vs import ( diff --git a/SCons/Tool/MSCommon/vc.py b/SCons/Tool/MSCommon/vc.py index a3fd183..b542d15 100644 --- a/SCons/Tool/MSCommon/vc.py +++ b/SCons/Tool/MSCommon/vc.py @@ -88,6 +88,40 @@ class MSVCScriptNotFound(VisualCException): class MSVCUseSettingsError(VisualCException): pass +class MSVCVersionNotFound(VisualCException): + pass + +# MSVC_NOTFOUND_POLICY definition: +# error: raise exception +# warning: issue warning and continue +# ignore: continue + +_MSVC_NOTFOUND_POLICY_DEFINITION = namedtuple('MSVCNotFoundPolicyDefinition', [ + 'value', + 'symbol', +]) + +_MSVC_NOTFOUND_POLICY_INTERNAL = {} +_MSVC_NOTFOUND_POLICY_EXTERNAL = {} + +for policy_value, policy_symbol_list in [ + (True, ['Error', 'Exception']), + (False, ['Warning', 'Warn']), + (None, ['Ignore', 'Suppress']), +]: + + policy_symbol = policy_symbol_list[0].lower() + policy_def = _MSVC_NOTFOUND_POLICY_DEFINITION(policy_value, policy_symbol) + + _MSVC_NOTFOUND_POLICY_INTERNAL[policy_symbol] = policy_def + + for policy_symbol in policy_symbol_list: + _MSVC_NOTFOUND_POLICY_EXTERNAL[policy_symbol.lower()] = policy_def + _MSVC_NOTFOUND_POLICY_EXTERNAL[policy_symbol] = policy_def + _MSVC_NOTFOUND_POLICY_EXTERNAL[policy_symbol.upper()] = policy_def + +_MSVC_NOTFOUND_POLICY_DEF = _MSVC_NOTFOUND_POLICY_INTERNAL['warning'] + # Dict to 'canonalize' the arch _ARCH_TO_CANONICAL = { "amd64" : "amd64", @@ -959,6 +993,7 @@ def reset_installed_vcs(): """Make it try again to find VC. This is just for the tests.""" global __INSTALLED_VCS_RUN __INSTALLED_VCS_RUN = None + _MSVCSetupEnvDefault.reset() # Running these batch files isn't cheap: most of the time spent in # msvs.generate() is due to vcvars*.bat. In a build that uses "tools='msvs'" @@ -1020,6 +1055,293 @@ def script_env(script, args=None): return cache_data +def _msvc_notfound_policy_lookup(symbol): + + try: + notfound_policy_def = _MSVC_NOTFOUND_POLICY_EXTERNAL[symbol] + except KeyError: + err_msg = "Value specified for MSVC_NOTFOUND_POLICY is not supported: {}.\n" \ + " Valid values are: {}".format( + repr(symbol), + ', '.join([repr(s) for s in _MSVC_NOTFOUND_POLICY_EXTERNAL.keys()]) + ) + raise ValueError(err_msg) + + return notfound_policy_def + +def set_msvc_notfound_policy(MSVC_NOTFOUND_POLICY=None): + """ Set the default policy when MSVC is not found. + + Args: + MSVC_NOTFOUND_POLICY: + string representing the policy behavior + when MSVC is not found or None + + Returns: + The previous policy is returned when the MSVC_NOTFOUND_POLICY argument + is not None. The active policy is returned when the MSVC_NOTFOUND_POLICY + argument is None. + + """ + global _MSVC_NOTFOUND_POLICY_DEF + + prev_policy = _MSVC_NOTFOUND_POLICY_DEF.symbol + + policy = MSVC_NOTFOUND_POLICY + if policy is not None: + _MSVC_NOTFOUND_POLICY_DEF = _msvc_notfound_policy_lookup(policy) + + debug( + 'prev_policy=%s, set_policy=%s, policy.symbol=%s, policy.value=%s', + repr(prev_policy), repr(policy), + repr(_MSVC_NOTFOUND_POLICY_DEF.symbol), repr(_MSVC_NOTFOUND_POLICY_DEF.value) + ) + + return prev_policy + +def get_msvc_notfound_policy(): + """Return the active policy when MSVC is not found.""" + debug( + 'policy.symbol=%s, policy.value=%s', + repr(_MSVC_NOTFOUND_POLICY_DEF.symbol), repr(_MSVC_NOTFOUND_POLICY_DEF.value) + ) + return _MSVC_NOTFOUND_POLICY_DEF.symbol + +def _msvc_notfound_policy_handler(env, msg): + + if env and 'MSVC_NOTFOUND_POLICY' in env: + # environment setting + notfound_policy_src = 'environment' + policy = env['MSVC_NOTFOUND_POLICY'] + if policy is not None: + # user policy request + notfound_policy_def = _msvc_notfound_policy_lookup(policy) + else: + # active global setting + notfound_policy_def = _MSVC_NOTFOUND_POLICY_DEF + else: + # active global setting + notfound_policy_src = 'default' + policy = None + notfound_policy_def = _MSVC_NOTFOUND_POLICY_DEF + + debug( + 'source=%s, set_policy=%s, policy.symbol=%s, policy.value=%s', + notfound_policy_src, repr(policy), repr(notfound_policy_def.symbol), repr(notfound_policy_def.value) + ) + + if notfound_policy_def.value is None: + # ignore + pass + elif notfound_policy_def.value: + raise MSVCVersionNotFound(msg) + else: + SCons.Warnings.warn(SCons.Warnings.VisualCMissingWarning, msg) + +class _MSVCSetupEnvDefault: + """ + Determine if and/or when an error/warning should be issued when there + are no versions of msvc installed. If there is at least one version of + msvc installed, these routines do (almost) nothing. + + Notes: + * When msvc is the default compiler because there are no compilers + installed, a build may fail due to the cl.exe command not being + recognized. Currently, there is no easy way to detect during + msvc initialization if the default environment will be used later + to build a program and/or library. There is no error/warning + as there are legitimate SCons uses that do not require a c compiler. + * As implemented, the default is that a warning is issued. This can + be changed globally via the function set_msvc_notfound_policy and/or + through the environment via the MSVC_NOTFOUND_POLICY variable. + """ + + separator = r';' + + need_init = True + + @classmethod + def reset(cls): + debug('msvc default:init') + cls.n_setup = 0 # number of calls to msvc_setup_env_once + cls.default_ismsvc = False # is msvc the default compiler + cls.default_tools_re_list = [] # list of default tools regular expressions + cls.msvc_tools_init = set() # tools registered via msvc_exists + cls.msvc_tools = None # tools registered via msvc_setup_env_once + cls.msvc_installed = False # is msvc installed (vcs_installed > 0) + cls.msvc_nodefault = False # is there a default version of msvc + cls.need_init = True # reset initialization indicator + + @classmethod + def _initialize(cls, env): + if cls.need_init: + cls.reset() + cls.need_init = False + vcs = get_installed_vcs(env) + cls.msvc_installed = len(vcs) > 0 + debug('msvc default:msvc_installed=%s', cls.msvc_installed) + + @classmethod + def register_tool(cls, env, tool): + debug('msvc default:tool=%s', tool) + if cls.need_init: + cls._initialize(env) + if cls.msvc_installed: + return None + if not tool: + return None + if cls.n_setup == 0: + if tool not in cls.msvc_tools_init: + cls.msvc_tools_init.add(tool) + debug('msvc default:tool=%s, msvc_tools_init=%s', tool, cls.msvc_tools_init) + return None + if tool not in cls.msvc_tools: + cls.msvc_tools.add(tool) + debug('msvc default:tool=%s, msvc_tools=%s', tool, cls.msvc_tools) + + @classmethod + def register_setup(cls, env): + debug('msvc default') + if cls.need_init: + cls._initialize(env) + cls.n_setup += 1 + if not cls.msvc_installed: + cls.msvc_tools = set(cls.msvc_tools_init) + if cls.n_setup == 1: + tool_list = env.get('TOOLS', None) + if tool_list and tool_list[0] == 'default': + if len(tool_list) > 1 and tool_list[1] in cls.msvc_tools: + # msvc tools are the default compiler + cls.default_ismsvc = True + cls.msvc_nodefault = False + debug( + 'msvc default:n_setup=%d, msvc_installed=%s, default_ismsvc=%s', + cls.n_setup, cls.msvc_installed, cls.default_ismsvc + ) + + @classmethod + def set_nodefault(cls): + # default msvc version, msvc not installed + cls.msvc_nodefault = True + debug('msvc default:msvc_nodefault=%s', cls.msvc_nodefault) + + @classmethod + def register_iserror(cls, env, tool): + + cls.register_tool(env, tool) + + if cls.msvc_installed: + # msvc installed + return None + + if not cls.msvc_nodefault: + # msvc version specified + return None + + tool_list = env.get('TOOLS', None) + if not tool_list: + # tool list is empty + return None + + debug( + 'msvc default:n_setup=%s, default_ismsvc=%s, msvc_tools=%s, tool_list=%s', + cls.n_setup, cls.default_ismsvc, cls.msvc_tools, tool_list + ) + + if not cls.default_ismsvc: + + # Summary: + # * msvc is not installed + # * msvc version not specified (default) + # * msvc is not the default compiler + + # construct tools set + tools_set = set(tool_list) + + else: + + if cls.n_setup == 1: + # first setup and msvc is default compiler: + # build default tools regex for current tool state + tools = cls.separator.join(tool_list) + tools_nchar = len(tools) + debug('msvc default:add regex:nchar=%d, tools=%s', tools_nchar, tools) + re_default_tools = re.compile(re.escape(tools)) + cls.default_tools_re_list.insert(0, (tools_nchar, re_default_tools)) + # early exit: no error for default environment when msvc is not installed + return None + + # Summary: + # * msvc is not installed + # * msvc version not specified (default) + # * environment tools list is not empty + # * default tools regex list constructed + # * msvc tools set constructed + # + # Algorithm using tools string and sets: + # * convert environment tools list to a string + # * iteratively remove default tools sequences via regex + # substition list built from longest sequence (first) + # to shortest sequence (last) + # * build environment tools set with remaining tools + # * compute intersection of environment tools and msvc tools sets + # * if the intersection is: + # empty - no error: default tools and/or no additional msvc tools + # not empty - error: user specified one or more msvc tool(s) + # + # This will not produce an error or warning when there are no + # msvc installed instances nor any other recognized compilers + # and the default environment is needed for a build. The msvc + # compiler is forcibly added to the environment tools list when + # there are no compilers installed on win32. In this case, cl.exe + # will not be found on the path resulting in a failed build. + + # construct tools string + tools = cls.separator.join(tool_list) + tools_nchar = len(tools) + + debug('msvc default:check tools:nchar=%d, tools=%s', tools_nchar, tools) + + # iteratively remove default tool sequences (longest to shortest) + re_nchar_min, re_tools_min = cls.default_tools_re_list[-1] + if tools_nchar >= re_nchar_min and re_tools_min.search(tools): + # minimum characters satisfied and minimum pattern exists + for re_nchar, re_default_tool in cls.default_tools_re_list: + if tools_nchar < re_nchar: + # not enough characters for pattern + continue + tools = re_default_tool.sub('', tools).strip(cls.separator) + tools_nchar = len(tools) + debug('msvc default:check tools:nchar=%d, tools=%s', tools_nchar, tools) + if tools_nchar < re_nchar_min or not re_tools_min.search(tools): + # less than minimum characters or minimum pattern does not exist + break + + # construct non-default list(s) tools set + tools_set = {msvc_tool for msvc_tool in tools.split(cls.separator) if msvc_tool} + + debug('msvc default:tools=%s', tools_set) + if not tools_set: + return None + + # compute intersection of remaining tools set and msvc tools set + tools_found = cls.msvc_tools.intersection(tools_set) + debug('msvc default:tools_exist=%s', tools_found) + if not tools_found: + return None + + # construct in same order as tools list + tools_found_list = [] + seen_tool = set() + for tool in tool_list: + if tool not in seen_tool: + seen_tool.add(tool) + if tool in tools_found: + tools_found_list.append(tool) + + # return tool list in order presented + return tools_found_list + def get_default_version(env): msvc_version = env.get('MSVC_VERSION') msvs_version = env.get('MSVS_VERSION') @@ -1054,16 +1376,25 @@ def get_default_version(env): return msvc_version -def msvc_setup_env_once(env): +def msvc_setup_env_once(env, tool=None): try: has_run = env["MSVC_SETUP_RUN"] except KeyError: has_run = False if not has_run: + debug('tool=%s', repr(tool)) + _MSVCSetupEnvDefault.register_setup(env) msvc_setup_env(env) env["MSVC_SETUP_RUN"] = True + req_tools = _MSVCSetupEnvDefault.register_iserror(env, tool) + if req_tools: + msg = "No versions of the MSVC compiler were found.\n" \ + " Visual Studio C/C++ compilers may not be set correctly.\n" \ + " Requested tool(s) are: {}".format(req_tools) + _msvc_notfound_policy_handler(env, msg) + def msvc_find_valid_batch_script(env, version): """Find and execute appropriate batch script to set up build env. @@ -1083,6 +1414,7 @@ def msvc_find_valid_batch_script(env, version): host_platform, target_platform, host_target_list = platforms d = None + version_installed = False for host_arch, target_arch, in host_target_list: # Set to current arch. env['TARGET_ARCH'] = target_arch @@ -1091,14 +1423,11 @@ def msvc_find_valid_batch_script(env, version): try: (vc_script, arg, sdk_script) = find_batch_file(env, version, host_arch, target_arch) debug('vc_script:%s vc_script_arg:%s sdk_script:%s', vc_script, arg, sdk_script) + version_installed = True except VisualCException as e: msg = str(e) debug('Caught exception while looking for batch file (%s)', msg) - warn_msg = "VC version %s not installed. " + \ - "C/C++ compilers are most likely not set correctly.\n" + \ - " Installed versions are: %s" - warn_msg = warn_msg % (version, get_installed_vcs(env)) - SCons.Warnings.warn(SCons.Warnings.VisualCMissingWarning, warn_msg) + version_installed = False continue # Try to use the located batch file for this host/target platform combo @@ -1140,6 +1469,25 @@ def msvc_find_valid_batch_script(env, version): if not d: env['TARGET_ARCH'] = target_platform + if version_installed: + msg = "MSVC version '{}' working host/target script was not found.\n" \ + " Host = '{}', Target = '{}'\n" \ + " Visual Studio C/C++ compilers may not be set correctly".format( + version, host_platform, target_platform + ) + else: + installed_vcs = get_installed_vcs(env) + if installed_vcs: + msg = "MSVC version '{}' was not found.\n" \ + " Visual Studio C/C++ compilers may not be set correctly.\n" \ + " Installed versions are: {}".format(version, installed_vcs) + else: + msg = "MSVC version '{}' was not found.\n" \ + " No versions of the MSVC compiler were found.\n" \ + " Visual Studio C/C++ compilers may not be set correctly".format(version) + + _msvc_notfound_policy_handler(env, msg) + return d _undefined = None @@ -1173,14 +1521,12 @@ def get_use_script_use_settings(env): # use script undefined, use_settings undefined or None return (True, None) - def msvc_setup_env(env): debug('called') version = get_default_version(env) if version is None: - warn_msg = "No version of Visual Studio compiler found - C/C++ " \ - "compilers most likely not set correctly" - SCons.Warnings.warn(SCons.Warnings.VisualCMissingWarning, warn_msg) + if not msvc_setup_env_user(env): + _MSVCSetupEnvDefault.set_nodefault() return None # XXX: we set-up both MSVS version for backward @@ -1189,7 +1535,6 @@ def msvc_setup_env(env): env['MSVS_VERSION'] = version env['MSVS'] = {} - use_script, use_settings = get_use_script_use_settings(env) if SCons.Util.is_String(use_script): use_script = use_script.strip() @@ -1231,7 +1576,56 @@ def msvc_setup_env(env): SCons.Warnings.warn(SCons.Warnings.VisualCMissingWarning, warn_msg) def msvc_exists(env=None, version=None): + debug('version=%s', repr(version)) vcs = get_installed_vcs(env) if version is None: - return len(vcs) > 0 - return version in vcs + rval = len(vcs) > 0 + else: + rval = version in vcs + debug('version=%s, return=%s', repr(version), rval) + return rval + +def msvc_setup_env_user(env=None): + rval = False + if env: + + # Intent is to use msvc tools: + # MSVC_VERSION or MSVS_VERSION: defined and is True + # MSVC_USE_SCRIPT: defined and (is string or is False) + # MSVC_USE_SETTINGS: defined and is not None + + # defined and is True + for key in ['MSVC_VERSION', 'MSVS_VERSION']: + if key in env and env[key]: + rval = True + debug('key=%s, return=%s', repr(key), rval) + return rval + + # defined and (is string or is False) + for key in ['MSVC_USE_SCRIPT']: + if key in env and (SCons.Util.is_String(env[key]) or not env[key]): + rval = True + debug('key=%s, return=%s', repr(key), rval) + return rval + + # defined and is not None + for key in ['MSVC_USE_SETTINGS']: + if key in env and env[key] is not None: + rval = True + debug('key=%s, return=%s', repr(key), rval) + return rval + + debug('return=%s', rval) + return rval + +def msvc_setup_env_tool(env=None, version=None, tool=None): + debug('tool=%s, version=%s', repr(tool), repr(version)) + _MSVCSetupEnvDefault.register_tool(env, tool) + rval = False + if not rval and msvc_exists(env, version): + rval = True + if not rval and msvc_setup_env_user(env): + rval = True + debug('tool=%s, version=%s, return=%s', repr(tool), repr(version), rval) + return rval + diff --git a/SCons/Tool/ToolTests.py b/SCons/Tool/ToolTests.py index 9338c2d..59d093b 100644 --- a/SCons/Tool/ToolTests.py +++ b/SCons/Tool/ToolTests.py @@ -1,5 +1,6 @@ +# MIT License # -# __COPYRIGHT__ +# Copyright The SCons Foundation # # Permission is hereby granted, free of charge, to any person obtaining # a copy of this software and associated documentation files (the @@ -19,9 +20,6 @@ # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# - -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" import os import unittest @@ -99,14 +97,14 @@ class ToolTestCase(unittest.TestCase): def test_pathfind(self): - """Test that find_program_path() does not alter PATH""" + """Test that find_program_path() alters PATH only if add_path is true""" env = DummyEnvironment() PHONY_PATHS = [ r'C:\cygwin64\bin', r'C:\cygwin\bin', '/usr/local/dummy/bin', - env.PHONY_PATH, # will be recognized by dummy WhereIs + env.PHONY_PATH, # will be recognized by dummy WhereIs ] env['ENV'] = {} env['ENV']['PATH'] = '/usr/local/bin:/opt/bin:/bin:/usr/bin' @@ -114,6 +112,9 @@ class ToolTestCase(unittest.TestCase): _ = SCons.Tool.find_program_path(env, 'no_tool', default_paths=PHONY_PATHS) assert env['ENV']['PATH'] == pre_path, env['ENV']['PATH'] + _ = SCons.Tool.find_program_path(env, 'no_tool', default_paths=PHONY_PATHS, add_path=True) + assert env.PHONY_PATH in env['ENV']['PATH'], env['ENV']['PATH'] + if __name__ == "__main__": loader = unittest.TestLoader() diff --git a/SCons/Tool/__init__.py b/SCons/Tool/__init__.py index eda9b0d..8e4a22d 100644 --- a/SCons/Tool/__init__.py +++ b/SCons/Tool/__init__.py @@ -21,13 +21,10 @@ # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -"""SCons.Tool +"""SCons tool selection. -SCons tool selection. - -This looks for modules that define a callable object that can modify -a construction environment as appropriate for a given tool (or tool -chain). +Looks for modules that define a callable object that can modify a +construction environment as appropriate for a given tool (or tool chain). Note that because this subsystem just *selects* a callable that can modify a construction environment, it's possible for people to define @@ -36,8 +33,6 @@ one needs to use or tie in to this subsystem in order to roll their own tool specifications. """ - - import sys import os import importlib.util @@ -834,15 +829,17 @@ def tool_list(platform, env): return [x for x in tools if x] -def find_program_path(env, key_program, default_paths=None): +def find_program_path(env, key_program, default_paths=None, add_path=False) -> str: """ Find the location of a tool using various means. Mainly for windows where tools aren't all installed in /usr/bin, etc. - :param env: Current Construction Environment. - :param key_program: Tool to locate. - :param default_paths: List of additional paths this tool might be found in. + Args: + env: Current Construction Environment. + key_program: Tool to locate. + default_paths: List of additional paths this tool might be found in. + add_path: If true, add path found if it was from *default_paths*. """ # First search in the SCons path path = env.WhereIs(key_program) @@ -854,15 +851,22 @@ def find_program_path(env, key_program, default_paths=None): if path: return path - # Finally, add the defaults and check again. Do not change - # ['ENV']['PATH'] permananetly, the caller can do that if needed. + # Finally, add the defaults and check again. if default_paths is None: return path + save_path = env['ENV']['PATH'] for p in default_paths: env.AppendENVPath('PATH', p) path = env.WhereIs(key_program) + + # By default, do not change ['ENV']['PATH'] permananetly + # leave that to the caller, unless add_path is true. env['ENV']['PATH'] = save_path + if path and add_path: + bin_dir = os.path.dirname(path) + env.AppendENVPath('PATH', bin_dir) + return path # Local Variables: diff --git a/SCons/Tool/fortran.xml b/SCons/Tool/fortran.xml index a4f0cec..4f0517b 100644 --- a/SCons/Tool/fortran.xml +++ b/SCons/Tool/fortran.xml @@ -1,6 +1,6 @@ <?xml version="1.0"?> <!-- -__COPYRIGHT__ +Copyright The SCons Foundation This file is processed by the bin/SConsDoc.py module. See its __doc__ string for a discussion of the format. @@ -26,7 +26,7 @@ See its __doc__ string for a discussion of the format. <tool name="fortran"> <summary> <para> -Set construction variables for generic POSIX Fortran compilers. +Set &consvars; for generic POSIX Fortran compilers. </para> </summary> <sets> @@ -43,6 +43,8 @@ Set construction variables for generic POSIX Fortran compilers. <item>FORTRANPPCOMSTR</item> <item>SHFORTRANCOMSTR</item> <item>SHFORTRANPPCOMSTR</item> +<item>CPPFLAGS</item> +<item>_CPPDEFFLAGS</item> </uses> </tool> @@ -61,11 +63,9 @@ for all versions of Fortran. The command line used to compile a Fortran source file to an object file. By default, any options specified in the &cv-link-FORTRANFLAGS;, -&cv-link-CPPFLAGS;, -&cv-link-_CPPDEFFLAGS;, &cv-link-_FORTRANMODFLAG;, and -&cv-link-_FORTRANINCFLAGS; construction variables -are included on this command line. +&cv-link-_FORTRANINCFLAGS; +&consvars; are included on this command line. </para> </summary> </cvar> @@ -120,7 +120,7 @@ for the variables that expand those options. <cvar name="_FORTRANINCFLAGS"> <summary> <para> -An automatically-generated construction variable +An automatically-generated &consvar; containing the Fortran compiler command-line options for specifying directories to be searched for include files and module files. @@ -150,7 +150,7 @@ for module files, as well. The prefix used to specify a module directory on the Fortran compiler command line. This will be prepended to the beginning of the directory -in the &cv-link-FORTRANMODDIR; construction variables +in the &cv-link-FORTRANMODDIR; &consvars; when the &cv-link-_FORTRANMODFLAG; variables is automatically generated. </para> </summary> @@ -162,7 +162,7 @@ when the &cv-link-_FORTRANMODFLAG; variables is automatically generated. The suffix used to specify a module directory on the Fortran compiler command line. This will be appended to the end of the directory -in the &cv-link-FORTRANMODDIR; construction variables +in the &cv-link-FORTRANMODDIR; &consvars; when the &cv-link-_FORTRANMODFLAG; variables is automatically generated. </para> </summary> @@ -171,7 +171,7 @@ when the &cv-link-_FORTRANMODFLAG; variables is automatically generated. <cvar name="_FORTRANMODFLAG"> <summary> <para> -An automatically-generated construction variable +An automatically-generated &consvar; containing the Fortran compiler command-line option for specifying the directory location where the Fortran compiler should place any module files that happen to get @@ -249,11 +249,11 @@ env = Environment(FORTRANPATH=include) The directory list will be added to command lines through the automatically-generated &cv-link-_FORTRANINCFLAGS; -construction variable, +&consvar;, which is constructed by respectively prepending and appending the values of the &cv-link-INCPREFIX; and &cv-link-INCSUFFIX; -construction variables +&consvars; to the beginning and end of each directory in &cv-link-FORTRANPATH;. Any command lines you define that need @@ -277,7 +277,7 @@ By default, any options specified in the &cv-link-FORTRANFLAGS;, &cv-link-_CPPDEFFLAGS;, &cv-link-_FORTRANMODFLAG;, and &cv-link-_FORTRANINCFLAGS; -construction variables are included on this command line. +&consvars; are included on this command line. </para> </summary> </cvar> @@ -323,6 +323,12 @@ The default Fortran compiler used for generating shared-library objects. <para> The command line used to compile a Fortran source file to a shared-library object file. +By default, any options specified +in the &cv-link-SHFORTRANFLAGS;, +&cv-link-_FORTRANMODFLAG;, and +&cv-link-_FORTRANINCFLAGS; +&consvars; are included on this command line. +See also &cv-link-FORTRANCOM;. </para> </summary> </cvar> @@ -353,10 +359,13 @@ to generate shared-library objects. The command line used to compile a Fortran source file to a shared-library object file after first running the file through the C preprocessor. -Any options specified -in the &cv-link-SHFORTRANFLAGS; and -&cv-link-CPPFLAGS; construction variables -are included on this command line. +By default, any options specified in the &cv-link-SHFORTRANFLAGS;, +&cv-link-CPPFLAGS;, +&cv-link-_CPPDEFFLAGS;, +&cv-link-_FORTRANMODFLAG;, and +&cv-link-_FORTRANINCFLAGS; +&consvars; are included on this command line. +See also &cv-link-SHFORTRANCOM;. </para> </summary> </cvar> diff --git a/SCons/Tool/midl.py b/SCons/Tool/midl.py index 2757c34..c25ce69 100644 --- a/SCons/Tool/midl.py +++ b/SCons/Tool/midl.py @@ -1,15 +1,6 @@ -"""SCons.Tool.midl - -Tool-specific initialization for midl (Microsoft IDL compiler). - -There normally shouldn't be any need to import this module directly. -It will usually be imported through the generic SCons.Tool.Tool() -selection method. - -""" - +# MIT License # -# __COPYRIGHT__ +# Copyright The SCons Foundation # # Permission is hereby granted, free of charge, to any person obtaining # a copy of this software and associated documentation files (the @@ -29,9 +20,16 @@ selection method. # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -# -__revision__ = "__FILE__ __REVISION__ __DATE__ __DEVELOPER__" +"""SCons.Tool.midl + +Tool-specific initialization for midl (Microsoft IDL compiler). + +There normally shouldn't be any need to import this module directly. +It will usually be imported through the generic SCons.Tool.Tool() +selection method. + +""" import SCons.Action import SCons.Builder @@ -39,7 +37,10 @@ import SCons.Defaults import SCons.Scanner.IDL import SCons.Util -from .MSCommon import msvc_exists +from .MSCommon import msvc_setup_env_tool + +tool_name = 'midl' + def midl_emitter(target, source, env): """Produces a list of outputs from the MIDL compiler""" @@ -58,28 +59,31 @@ def midl_emitter(target, source, env): dlldata = base + '_data.c' targets.append(dlldata) - return (targets, source) + return targets, source + idl_scanner = SCons.Scanner.IDL.IDLScan() midl_action = SCons.Action.Action('$MIDLCOM', '$MIDLCOMSTR') -midl_builder = SCons.Builder.Builder(action = midl_action, - src_suffix = '.idl', +midl_builder = SCons.Builder.Builder(action=midl_action, + src_suffix='.idl', suffix='.tlb', - emitter = midl_emitter, - source_scanner = idl_scanner) + emitter=midl_emitter, + source_scanner=idl_scanner) + def generate(env): """Add Builders and construction variables for midl to an Environment.""" - env['MIDL'] = 'MIDL.EXE' - env['MIDLFLAGS'] = SCons.Util.CLVar('/nologo') - env['MIDLCOM'] = '$MIDL $MIDLFLAGS /tlb ${TARGETS[0]} /h ${TARGETS[1]} /iid ${TARGETS[2]} /proxy ${TARGETS[3]} /dlldata ${TARGETS[4]} $SOURCE 2> NUL' + env['MIDL'] = 'MIDL.EXE' + env['MIDLFLAGS'] = SCons.Util.CLVar('/nologo') + env['MIDLCOM'] = '$MIDL $MIDLFLAGS /tlb ${TARGETS[0]} /h ${TARGETS[1]} /iid ${TARGETS[2]} /proxy ${TARGETS[3]} /dlldata ${TARGETS[4]} $SOURCE 2> NUL' env['BUILDERS']['TypeLibrary'] = midl_builder + def exists(env): - return msvc_exists(env) + return msvc_setup_env_tool(env, tool=tool_name) # Local Variables: # tab-width:4 diff --git a/SCons/Tool/mslib.py b/SCons/Tool/mslib.py index 354f5cf..be4088b 100644 --- a/SCons/Tool/mslib.py +++ b/SCons/Tool/mslib.py @@ -41,14 +41,16 @@ import SCons.Tool.msvs import SCons.Tool.msvc import SCons.Util -from .MSCommon import msvc_exists, msvc_setup_env_once +from .MSCommon import msvc_setup_env_tool, msvc_setup_env_once + +tool_name = 'mslib' def generate(env): """Add Builders and construction variables for lib to an Environment.""" SCons.Tool.createStaticLibBuilder(env) # Set-up ms tools paths - msvc_setup_env_once(env) + msvc_setup_env_once(env, tool=tool_name) env['AR'] = 'lib' env['ARFLAGS'] = SCons.Util.CLVar('/nologo') @@ -64,7 +66,7 @@ def generate(env): def exists(env): - return msvc_exists(env) + return msvc_setup_env_tool(env, tool=tool_name) # Local Variables: # tab-width:4 diff --git a/SCons/Tool/mslink.py b/SCons/Tool/mslink.py index 3dac7f0..2a90e17 100644 --- a/SCons/Tool/mslink.py +++ b/SCons/Tool/mslink.py @@ -43,9 +43,11 @@ import SCons.Tool.msvc import SCons.Tool.msvs import SCons.Util -from .MSCommon import msvc_setup_env_once, msvc_exists +from .MSCommon import msvc_setup_env_once, msvc_setup_env_tool from .MSCommon.common import get_pch_node +tool_name = 'mslink' + def pdbGenerator(env, target, source, for_signature): try: return ['/PDB:%s' % target[0].attributes.pdb, '/DEBUG'] @@ -307,7 +309,7 @@ def generate(env): env['_MANIFEST_SOURCES'] = None # _windowsManifestSources # Set-up ms tools paths - msvc_setup_env_once(env) + msvc_setup_env_once(env, tool=tool_name) # Loadable modules are on Windows the same as shared libraries, but they # are subject to different build parameters (LDMODULE* variables). @@ -330,7 +332,7 @@ def generate(env): env['TEMPFILEARGJOIN'] = os.linesep def exists(env): - return msvc_exists(env) + return msvc_setup_env_tool(env, tool=tool_name) # Local Variables: # tab-width:4 diff --git a/SCons/Tool/msvc.py b/SCons/Tool/msvc.py index 58b7463..191d2cc 100644 --- a/SCons/Tool/msvc.py +++ b/SCons/Tool/msvc.py @@ -44,9 +44,11 @@ import SCons.Util import SCons.Warnings import SCons.Scanner.RC -from .MSCommon import msvc_exists, msvc_setup_env_once, msvc_version_to_maj_min, msvc_find_vswhere +from .MSCommon import msvc_setup_env_tool, msvc_setup_env_once, msvc_version_to_maj_min, msvc_find_vswhere from .MSCommon.common import get_pch_node +tool_name = 'msvc' + CSuffixes = ['.c', '.C'] CXXSuffixes = ['.cc', '.cpp', '.cxx', '.c++', '.C++'] @@ -293,7 +295,7 @@ def generate(env): env['VSWHERE'] = env.get('VSWHERE', msvc_find_vswhere()) # Set-up ms tools paths - msvc_setup_env_once(env) + msvc_setup_env_once(env, tool=tool_name) env['CFILESUFFIX'] = '.c' env['CXXFILESUFFIX'] = '.cc' @@ -320,7 +322,7 @@ def generate(env): def exists(env): - return msvc_exists(env) + return msvc_setup_env_tool(env, tool=tool_name) # Local Variables: # tab-width:4 diff --git a/SCons/Tool/msvc.xml b/SCons/Tool/msvc.xml index 4a45d26..e8df128 100644 --- a/SCons/Tool/msvc.xml +++ b/SCons/Tool/msvc.xml @@ -74,6 +74,7 @@ Sets construction variables for the Microsoft Visual C/C++ compiler. <item>PCH</item> <item>PCHSTOP</item> <item>PDB</item> +<item>MSVC_NOTFOUND_POLICY</item> </uses> </tool> @@ -578,5 +579,98 @@ and also before &f-link-env-Tool; is called to ininitialize any of those tools: </summary> </cvar> +<cvar name="MSVC_NOTFOUND_POLICY"> +<summary> +<para> +Specify the scons behavior when the Microsoft Visual C/C++ compiler is not detected. +</para> + +<para> + The <envar>MSVC_NOTFOUND_POLICY</envar> specifies the &scons; behavior when no msvc versions are detected or + when the requested msvc version is not detected. +</para> + +<para> +The valid values for <envar>MSVC_NOTFOUND_POLICY</envar> and the corresponding &scons; behavior are: +</para> + +<variablelist> + +<varlistentry> +<term><parameter>'Error' or 'Exception'</parameter></term> +<listitem> +<para> +Raise an exception when no msvc versions are detected or when the requested msvc version is not detected. +</para> +</listitem> +</varlistentry> + +<varlistentry> +<term><parameter>'Warning' or 'Warn'</parameter></term> +<listitem> +<para> +Issue a warning and continue when no msvc versions are detected or when the requested msvc version is not detected. +Depending on usage, this could result in build failure(s). +</para> +</listitem> +</varlistentry> + +<varlistentry> +<term><parameter>'Ignore' or 'Suppress'</parameter></term> +<listitem> +<para> +Take no action and continue when no msvc versions are detected or when the requested msvc version is not detected. +Depending on usage, this could result in build failure(s). +</para> +</listitem> +</varlistentry> + +</variablelist> + +<para> +Note: in addition to the camel case values shown above, lower case and upper case values are accepted as well. +</para> + +<para> +The <envar>MSVC_NOTFOUND_POLICY</envar> is applied when any of the following conditions are satisfied: +<itemizedlist> +<listitem><para> +&cv-MSVC_VERSION; is specified, the default tools list is implicitly defined (i.e., the tools list is not specified), +and the default tools list contains one or more of the msvc tools. +</para></listitem> +<listitem><para> +&cv-MSVC_VERSION; is specified, the default tools list is explicitly specified (e.g., <literal>tools=['default']</literal>), +and the default tools list contains one or more of the msvc tools. +</para></listitem> +<listitem><para> +A non-default tools list is specified that contains one or more of the msvc tools (e.g., <literal>tools=['msvc', 'mslink']</literal>). +</para></listitem> +</itemizedlist> +</para> + +<para> +The <envar>MSVC_NOTFOUND_POLICY</envar> is ignored when any of the following conditions are satisfied: +<itemizedlist> +<listitem><para> +&cv-MSVC_VERSION; is not specified and the default tools list is implicitly defined (i.e., the tools list is not specified). +</para></listitem> +<listitem><para> +&cv-MSVC_VERSION; is not specified and the default tools list is explicitly specified (e.g., <literal>tools=['default']</literal>). +</para></listitem> +<listitem><para> +A non-default tool list is specified that does not contain any of the msvc tools (e.g., <literal>tools=['mingw']</literal>). +</para></listitem> +</itemizedlist> +</para> + +<para> +When <envar>MSVC_NOTFOUND_POLICY</envar> is not specified, the default &scons; behavior is to issue a warning and continue +subject to the conditions listed above. The default &scons; behavior may change in the future. +</para> + + +</summary> +</cvar> + </sconsdoc> diff --git a/SCons/Tool/msvs.py b/SCons/Tool/msvs.py index 887cb59..86df1ef 100644 --- a/SCons/Tool/msvs.py +++ b/SCons/Tool/msvs.py @@ -45,7 +45,9 @@ import SCons.Util import SCons.Warnings from SCons.Defaults import processDefines from SCons.compat import PICKLE_PROTOCOL -from .MSCommon import msvc_exists, msvc_setup_env_once +from .MSCommon import msvc_setup_env_tool, msvc_setup_env_once + +tool_name = 'msvs' ############################################################################## # Below here are the classes and functions for generation of @@ -2077,7 +2079,7 @@ def generate(env): env['MSVSCLEANCOM'] = '$MSVSSCONSCOM -c "$MSVSBUILDTARGET"' # Set-up ms tools paths for default version - msvc_setup_env_once(env) + msvc_setup_env_once(env, tool=tool_name) if 'MSVS_VERSION' in env: version_num, suite = msvs_parse_version(env['MSVS_VERSION']) @@ -2107,7 +2109,7 @@ def generate(env): env['SCONS_HOME'] = os.environ.get('SCONS_HOME') def exists(env): - return msvc_exists(env) + return msvc_setup_env_tool(env, tool=tool_name) # Local Variables: # tab-width:4 diff --git a/SCons/Tool/ninja/NinjaState.py b/SCons/Tool/ninja/NinjaState.py index c75c966..5e7c289 100644 --- a/SCons/Tool/ninja/NinjaState.py +++ b/SCons/Tool/ninja/NinjaState.py @@ -235,6 +235,12 @@ class NinjaState: "pool": "local_pool", "restat": 1 }, + "EXIT_SCONS_DAEMON": { + "command": "$PYTHON_BIN $NINJA_TOOL_DIR/ninja_daemon_build.py $PORT $NINJA_DIR_PATH --exit", + "description": "Shutting down ninja scons daemon server", + "pool": "local_pool", + "restat": 1 + }, "SCONS": { "command": "$SCONS_INVOCATION $out", "description": "$SCONS_INVOCATION $out", @@ -643,6 +649,11 @@ class NinjaState: rule="SCONS_DAEMON", ) + ninja.build( + "shutdown_ninja_scons_daemon_phony", + rule="EXIT_SCONS_DAEMON", + ) + if all_targets is None: # Look in SCons's list of DEFAULT_TARGETS, find the ones that diff --git a/SCons/Tool/ninja/__init__.py b/SCons/Tool/ninja/__init__.py index 7c32676..61f98f5 100644 --- a/SCons/Tool/ninja/__init__.py +++ b/SCons/Tool/ninja/__init__.py @@ -91,7 +91,7 @@ def ninja_builder(env, target, source): if str(env.get("NINJA_DISABLE_AUTO_RUN")).lower() not in ['1', 'true']: num_jobs = env.get('NINJA_MAX_JOBS', env.GetOption("num_jobs")) - cmd += ['-j' + str(num_jobs)] + NINJA_CMDLINE_TARGETS + cmd += ['-j' + str(num_jobs)] + env.get('NINJA_CMD_ARGS', '').split() + NINJA_CMDLINE_TARGETS print(f"ninja will be run with command line targets: {' '.join(NINJA_CMDLINE_TARGETS)}") print("Executing:", str(' '.join(cmd))) @@ -129,6 +129,14 @@ def ninja_builder(env, target, source): sys.stdout.write("\n") +def options(opts): + """ + Add command line Variables for Ninja builder. + """ + opts.AddVariables( + ("NINJA_CMD_ARGS", "Arguments to pass to ninja"), + ) + def exists(env): """Enable if called.""" @@ -200,7 +208,7 @@ def generate(env): env["NINJA_SCONS_DAEMON_KEEP_ALIVE"] = env.get("NINJA_SCONS_DAEMON_KEEP_ALIVE", 180000) if GetOption("disable_ninja"): - env.SConsignFile(os.path.join(str(env['NINJA_DIR']),'.ninja.sconsign')) + env.SConsignFile(os.path.join(str(env['NINJA_DIR']), '.ninja.sconsign')) # here we allow multiple environments to construct rules and builds # into the same ninja file @@ -460,7 +468,6 @@ def generate(env): # date-ness. SCons.Script.Main.BuildTask.needs_execute = lambda x: True - def ninja_Set_Default_Targets(env, tlist): """ Record the default targets if they were ever set by the user. Ninja @@ -496,5 +503,5 @@ def generate(env): env['TEMPFILEDIR'] = "$NINJA_DIR/.response_files" env["TEMPFILE"] = NinjaNoResponseFiles - env.Alias('run-ninja-scons-daemon', 'run_ninja_scons_daemon_phony') + env.Alias('shutdown-ninja-scons-daemon', 'shutdown_ninja_scons_daemon_phony') diff --git a/SCons/Tool/ninja/ninja.xml b/SCons/Tool/ninja/ninja.xml index 6b247d0..664d521 100644 --- a/SCons/Tool/ninja/ninja.xml +++ b/SCons/Tool/ninja/ninja.xml @@ -77,7 +77,7 @@ See its __doc__ string for a discussion of the format. <item>IMPLICIT_COMMAND_DEPENDENCIES</item> <item>NINJA_SCONS_DAEMON_KEEP_ALIVE</item> <item>NINJA_SCONS_DAEMON_PORT</item> - + <item>NINJA_CMD_ARGS</item> <!-- TODO: Document these --> <!-- <item>NINJA_RULES</item>--> @@ -395,5 +395,22 @@ python -m pip install ninja </summary> </cvar> + <cvar name="NINJA_CMD_ARGS"> + <summary> + <para> + A string which will pass arguments through SCons to the ninja command when scons executes ninja. + Has no effect if &cv-NINJA_DISABLE_AUTO_RUN; is set. + </para> + <para> + This value can also be passed on the command line: + </para> + <example_commands> +scons NINJA_CMD_ARGS=-v +or +scons NINJA_CMD_ARGS="-v -j 3" + </example_commands> + </summary> + </cvar> + </sconsdoc> diff --git a/SCons/Tool/ninja/ninja_daemon_build.py b/SCons/Tool/ninja/ninja_daemon_build.py index 48156f5..32c375d 100644 --- a/SCons/Tool/ninja/ninja_daemon_build.py +++ b/SCons/Tool/ninja/ninja_daemon_build.py @@ -54,17 +54,30 @@ logging.basicConfig( level=logging.DEBUG, ) + def log_error(msg): logging.debug(msg) sys.stderr.write(msg) + while True: try: + if not os.path.exists(daemon_dir / "pidfile"): + if sys.argv[3] != '--exit': + logging.debug(f"ERROR: Server pid not found {daemon_dir / 'pidfile'} for request {sys.argv[3]}") + exit(1) + else: + logging.debug("WARNING: Unnecessary request to shutdown server, it's already shutdown.") + exit(0) + logging.debug(f"Sending request: {sys.argv[3]}") conn = http.client.HTTPConnection( "127.0.0.1", port=int(sys.argv[1]), timeout=60 ) - conn.request("GET", "/?build=" + sys.argv[3]) + if sys.argv[3] == '--exit': + conn.request("GET", "/?exit=1") + else: + conn.request("GET", "/?build=" + sys.argv[3]) response = None while not response: @@ -81,6 +94,7 @@ while True: if status != 200: log_error(msg.decode("utf-8")) exit(1) + logging.debug(f"Request Done: {sys.argv[3]}") exit(0) diff --git a/SCons/Tool/ninja/ninja_scons_daemon.py b/SCons/Tool/ninja/ninja_scons_daemon.py index c4a1d11..6802af2 100644 --- a/SCons/Tool/ninja/ninja_scons_daemon.py +++ b/SCons/Tool/ninja/ninja_scons_daemon.py @@ -168,6 +168,7 @@ def daemon_thread_func(): te.start() daemon_ready = False + building_node = None startup_complete = False @@ -218,12 +219,14 @@ def daemon_thread_func(): except queue.Empty: break if "exit" in building_node: + daemon_log("input: " + "exit") p.stdin.write("exit\n".encode("utf-8")) p.stdin.flush() with building_cv: shared_state.finished_building += [building_node] daemon_ready = False - raise + shared_state.daemon_needs_to_shutdown = True + break else: input_command = "build " + building_node + "\n" diff --git a/SCons/Tool/packaging/__init__.py b/SCons/Tool/packaging/__init__.py index 68cedeb..6fe01c1 100644 --- a/SCons/Tool/packaging/__init__.py +++ b/SCons/Tool/packaging/__init__.py @@ -217,10 +217,11 @@ def generate(env): env['BUILDERS']['Package'] = Package env['BUILDERS']['Tag'] = Tag + def exists(env): return 1 -# XXX + def options(opts): opts.AddVariables( EnumVariable('PACKAGETYPE', diff --git a/requirements.txt b/requirements.txt index 1be070c..e05c610 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,4 +14,5 @@ rst2pdf ninja # Needed for test/Parallel/failed-build/failed-build.py +# Also for test/ninja/shutdown_scons_daemon.py psutil diff --git a/test/MSVC/msvc_badversion.py b/test/MSVC/msvc_badversion.py new file mode 100644 index 0000000..65dc789 --- /dev/null +++ b/test/MSVC/msvc_badversion.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python +# +# MIT License +# +# Copyright The SCons Foundation +# +# Permission is hereby granted, free of charge, to any person obtaining +# a copy of this software and associated documentation files (the +# "Software"), to deal in the Software without restriction, including +# without limitation the rights to use, copy, modify, merge, publish, +# distribute, sublicense, and/or sell copies of the Software, and to +# permit persons to whom the Software is furnished to do so, subject to +# the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +""" +Test scons with an invalid MSVC version when at least one MSVC is present. +""" + +import sys + +import TestSCons +import SCons.Tool.MSCommon.vc as msvc + +test = TestSCons.TestSCons() + +if sys.platform != 'win32': + test.skip_test("Not win32 platform. Skipping test\n") + +test.skip_if_not_msvc() + +installed_msvc_versions = msvc.get_installed_vcs() +# MSVC guaranteed to be at least one version on the system or else +# skip_if_not_msvc() function would have skipped the test + +test.write('SConstruct', """\ +DefaultEnvironment(tools=[]) +env = Environment(MSVC_VERSION='12.9') +""") +test.run(arguments='-Q -s', stdout='') + +test.write('SConstruct', """\ +DefaultEnvironment(tools=[]) +env = Environment(MSVC_VERSION='12.9', MSVC_NOTFOUND_POLICY='ignore') +""") +test.run(arguments='-Q -s', stdout='') + +test.write('SConstruct', """\ +DefaultEnvironment(tools=[]) +env = Environment(MSVC_VERSION='12.9', MSVC_NOTFOUND_POLICY='warning') +""") +test.run(arguments='-Q -s', stdout='') + +test.write('SConstruct', """\ +DefaultEnvironment(tools=[]) +env = Environment(MSVC_VERSION='12.9', MSVC_NOTFOUND_POLICY='error') +""") +test.run(arguments='-Q -s', status=2, stderr=r"^.*MSVCVersionNotFound.+", match=TestSCons.match_re_dotall) + +test.write('SConstruct', """\ +env = Environment(MSVC_VERSION='12.9', MSVC_NOTFOUND_POLICY='bad_value') +""") +test.run(arguments='-Q -s', status=2, stderr=r"^.* Value specified for MSVC_NOTFOUND_POLICY.+", match=TestSCons.match_re_dotall) + + +test.pass_test() + +# Local Variables: +# tab-width:4 +# indent-tabs-mode:nil +# End: +# vim: set expandtab tabstop=4 shiftwidth=4: diff --git a/test/MSVC/no_msvc.py b/test/MSVC/no_msvc.py index d1161c6..4ab7dd8 100644 --- a/test/MSVC/no_msvc.py +++ b/test/MSVC/no_msvc.py @@ -48,4 +48,24 @@ test.run(arguments='-Q -s') if 'MSVC_VERSION=None' not in test.stdout(): test.fail_test() -test.pass_test()
\ No newline at end of file +# test msvc version number request with no msvc's +test.file_fixture('no_msvc/no_msvcs_sconstruct_version.py', 'SConstruct') +test.run(arguments='-Q -s', status=2, stderr=r"^.*MSVCVersionNotFound.+", match=TestSCons.match_re_dotall) + +# test that MSVCVersionNotFound is not raised for default msvc tools +# when a non-msvc tool list is used +test.subdir('site_scons', ['site_scons', 'site_tools']) + +test.write(['site_scons', 'site_tools', 'myignoredefaultmsvctool.py'], """ +import SCons.Tool +def generate(env): + env['MYIGNOREDEFAULTMSVCTOOL']='myignoredefaultmsvctool' +def exists(env): + return 1 +""") + +test.file_fixture('no_msvc/no_msvcs_sconstruct_tools.py', 'SConstruct') +test.run(arguments='-Q -s') + +test.pass_test() + diff --git a/test/fixture/no_msvc/no_msvcs_sconstruct_tools.py b/test/fixture/no_msvc/no_msvcs_sconstruct_tools.py new file mode 100644 index 0000000..9aa924b --- /dev/null +++ b/test/fixture/no_msvc/no_msvcs_sconstruct_tools.py @@ -0,0 +1,14 @@ +import SCons +import SCons.Tool.MSCommon + +def DummyVsWhere(msvc_version, env): + # not testing versions with vswhere, so return none + return None + +for key in SCons.Tool.MSCommon.vc._VCVER_TO_PRODUCT_DIR: + SCons.Tool.MSCommon.vc._VCVER_TO_PRODUCT_DIR[key]=[(SCons.Util.HKEY_LOCAL_MACHINE, r'')] + +SCons.Tool.MSCommon.vc.find_vc_pdir_vswhere = DummyVsWhere + +env = SCons.Environment.Environment(tools=['myignoredefaultmsvctool']) + diff --git a/test/fixture/no_msvc/no_msvcs_sconstruct_version.py b/test/fixture/no_msvc/no_msvcs_sconstruct_version.py new file mode 100644 index 0000000..f5cabf7 --- /dev/null +++ b/test/fixture/no_msvc/no_msvcs_sconstruct_version.py @@ -0,0 +1,16 @@ +import SCons +import SCons.Tool.MSCommon + +def DummyVsWhere(msvc_version, env): + # not testing versions with vswhere, so return none + return None + +for key in SCons.Tool.MSCommon.vc._VCVER_TO_PRODUCT_DIR: + SCons.Tool.MSCommon.vc._VCVER_TO_PRODUCT_DIR[key]=[(SCons.Util.HKEY_LOCAL_MACHINE, r'')] + +SCons.Tool.MSCommon.vc.find_vc_pdir_vswhere = DummyVsWhere + +SCons.Tool.MSCommon.set_msvc_notfound_policy('error') + +env = SCons.Environment.Environment(MSVC_VERSION='14.3') + diff --git a/test/ninja/generate_and_build.py b/test/ninja/generate_and_build.py index 91be108..e1c26d4 100644 --- a/test/ninja/generate_and_build.py +++ b/test/ninja/generate_and_build.py @@ -49,12 +49,17 @@ test.dir_fixture('ninja-fixture') test.file_fixture('ninja_test_sconscripts/sconstruct_generate_and_build', 'SConstruct') # generate simple build -test.run(stdout=None) +test.run(stdout=None, arguments='NINJA_CMD_ARGS=-v') test.must_contain_all_lines(test.stdout(), ['Generating: build.ninja']) test.must_contain_all(test.stdout(), 'Executing:') test.must_contain_all(test.stdout(), 'ninja%(_exe)s -f' % locals()) +test.must_contain_all(test.stdout(), ' -j1 -v') test.run(program=test.workpath('foo' + _exe), stdout="foo.c") +# Test multiple args for NINJA_CMD_ARGS +test.run(stdout=None, arguments={'NINJA_CMD_ARGS':"-v -j3"}) +test.must_contain_all(test.stdout(), ' -v -j3') + # clean build and ninja files test.run(arguments='-c', stdout=None) test.must_contain_all_lines(test.stdout(), [ diff --git a/test/ninja/shutdown_scons_daemon.py b/test/ninja/shutdown_scons_daemon.py new file mode 100644 index 0000000..64ec2c7 --- /dev/null +++ b/test/ninja/shutdown_scons_daemon.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python +# +# Copyright The SCons Foundation +# +# Permission is hereby granted, free of charge, to any person obtaining +# a copy of this software and associated documentation files (the +# "Software"), to deal in the Software without restriction, including +# without limitation the rights to use, copy, modify, merge, publish, +# distribute, sublicense, and/or sell copies of the Software, and to +# permit persons to whom the Software is furnished to do so, subject to +# the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY +# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE +# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +# + +import os +from timeit import default_timer as timer + +import TestSCons +from TestCmd import IS_WINDOWS + +test = TestSCons.TestSCons() + +try: + import ninja +except ImportError: + test.skip_test("Could not find ninja module in python") + +try: + import psutil +except ImportError: + test.skip_test("Could not find psutil module in python") + + +_python_ = TestSCons._python_ +_exe = TestSCons._exe + +ninja_bin = os.path.abspath( + os.path.join(ninja.__file__, os.pardir, "data", "bin", "ninja" + _exe) +) + +test.dir_fixture("ninja-fixture") + +test.file_fixture( + "ninja_test_sconscripts/sconstruct_force_scons_callback", "SConstruct" +) + +test.run(stdout=None) +pid = None +test.must_exist(test.workpath('.ninja/scons_daemon_dirty')) +with open(test.workpath('.ninja/scons_daemon_dirty')) as f: + pid = int(f.read()) + if pid not in [proc.pid for proc in psutil.process_iter()]: + test.fail_test(message="daemon not running!") + +program = test.workpath("run_ninja_env.bat") if IS_WINDOWS else ninja_bin +test.run(program=program, arguments='shutdown-ninja-scons-daemon', stdout=None) + +wait_time = 10 +start_time = timer() +while True: + if wait_time > (timer() - start_time): + if pid not in [proc.pid for proc in psutil.process_iter()]: + break + else: + test.fail_test(message=f"daemon still not shutdown after {wait_time} seconds.") + +test.must_not_exist(test.workpath('.ninja/scons_daemon_dirty')) +test.pass_test() + +# Local Variables: +# tab-width:4 +# indent-tabs-mode:nil +# End: +# vim: set expandtab tabstop=4 shiftwidth=4: diff --git a/testing/framework/TestCmd.py b/testing/framework/TestCmd.py index 5759121..fc5e328 100644 --- a/testing/framework/TestCmd.py +++ b/testing/framework/TestCmd.py @@ -299,9 +299,12 @@ __version__ = "1.3" import atexit import difflib import errno +import hashlib import os import re +import psutil import shutil +import signal import stat import subprocess import sys @@ -310,6 +313,7 @@ import threading import time import traceback from collections import UserList, UserString +from pathlib import Path from subprocess import PIPE, STDOUT from typing import Optional @@ -383,6 +387,37 @@ def _caller(tblist, skip): return string +def clean_up_ninja_daemon(self, result_type): + """ + Kill any running scons daemon started by ninja and clean up it's working dir and + temp files. + """ + if self: + for path in Path(self.workdir).rglob('.ninja'): + daemon_dir = Path(tempfile.gettempdir()) / ( + "scons_daemon_" + str(hashlib.md5(str(path.resolve()).encode()).hexdigest()) + ) + pidfiles = [daemon_dir / 'pidfile', path / 'scons_daemon_dirty'] + for pidfile in pidfiles: + if pidfile.exists(): + with open(pidfile) as f: + try: + pid = int(f.read()) + os.kill(pid, signal.SIGINT) + except OSError: + pass + + while True: + if pid not in [proc.pid for proc in psutil.process_iter()]: + break + else: + time.sleep(0.1) + + if not self._preserve[result_type]: + if daemon_dir.exists(): + shutil.rmtree(daemon_dir) + + def fail_test(self=None, condition=True, function=None, skip=0, message=None): """Causes a test to exit with a fail. @@ -402,6 +437,7 @@ def fail_test(self=None, condition=True, function=None, skip=0, message=None): return if function is not None: function() + clean_up_ninja_daemon(self, 'fail_test') of = "" desc = "" sep = " " @@ -447,6 +483,7 @@ def no_result(self=None, condition=True, function=None, skip=0): return if function is not None: function() + clean_up_ninja_daemon(self, 'no_result') of = "" desc = "" sep = " " @@ -483,6 +520,7 @@ def pass_test(self=None, condition=True, function=None): return if function is not None: function() + clean_up_ninja_daemon(self, 'pass_test') sys.stderr.write("PASSED\n") sys.exit(0) @@ -1125,6 +1163,9 @@ class TestCmd: interpreter = [interpreter] cmd = list(interpreter) + cmd if arguments: + if isinstance(arguments, dict): + cmd.extend(["%s=%s" % (k, v) for k, v in arguments.items()]) + return cmd if isinstance(arguments, str): arguments = arguments.split() cmd.extend(arguments) @@ -1569,6 +1610,9 @@ class TestCmd: The specified program will have the original directory prepended unless it is enclosed in a [list]. + + argument: If this is a dict() then will create arguments with KEY+VALUE for + each entry in the dict. """ if self.external: if not program: diff --git a/testing/framework/TestCmdTests.py b/testing/framework/TestCmdTests.py index 3b29091..a760382 100644 --- a/testing/framework/TestCmdTests.py +++ b/testing/framework/TestCmdTests.py @@ -243,7 +243,7 @@ import atexit import sys import TestCmd -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path @atexit.register def cleanup(): @@ -415,7 +415,7 @@ class diff_TestCase(TestCmdTestCase): def test_diff_custom_function(self): """Test diff() using a custom function""" self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def my_diff(a, b): return [ @@ -440,7 +440,7 @@ STDOUT========================================================================== def test_diff_string(self): self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff = 'diff_re') test.diff("a\\nb1\\nc\\n", "a\\nb2\\nc\\n", 'STDOUT') @@ -457,7 +457,7 @@ STDOUT========================================================================== def test_error(self): """Test handling a compilation error in TestCmd.diff_re()""" script_input = """import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd assert TestCmd.diff_re([r"a.*(e"], ["abcde"]) sys.exit(0) @@ -472,7 +472,7 @@ sys.exit(0) def test_simple_diff_static_method(self): """Test calling the TestCmd.TestCmd.simple_diff() static method""" self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd result = TestCmd.TestCmd.simple_diff(['a', 'b', 'c', 'e', 'f1'], ['a', 'c', 'd', 'e', 'f2']) @@ -485,7 +485,7 @@ sys.exit(0) def test_context_diff_static_method(self): """Test calling the TestCmd.TestCmd.context_diff() static method""" self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd result = TestCmd.TestCmd.context_diff(['a\\n', 'b\\n', 'c\\n', 'e\\n', 'f1\\n'], ['a\\n', 'c\\n', 'd\\n', 'e\\n', 'f2\\n']) @@ -514,7 +514,7 @@ sys.exit(0) def test_unified_diff_static_method(self): """Test calling the TestCmd.TestCmd.unified_diff() static method""" self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd result = TestCmd.TestCmd.unified_diff(['a\\n', 'b\\n', 'c\\n', 'e\\n', 'f1\\n'], ['a\\n', 'c\\n', 'd\\n', 'e\\n', 'f2\\n']) @@ -538,7 +538,7 @@ sys.exit(0) def test_diff_re_static_method(self): """Test calling the TestCmd.TestCmd.diff_re() static method""" self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd result = TestCmd.TestCmd.diff_re(['a', 'b', 'c', '.', 'f1'], ['a', 'c', 'd', 'e', 'f2']) @@ -567,7 +567,7 @@ class diff_stderr_TestCase(TestCmdTestCase): def test_diff_stderr_default(self): """Test diff_stderr() default behavior""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd() test.diff_stderr('a\nb1\nc\n', 'a\nb2\nc\n') @@ -584,7 +584,7 @@ sys.exit(0) """Test diff_stderr() not affecting diff_stdout() behavior""" self.popen_python(r""" import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stderr='diff_re') print("diff_stderr:") @@ -605,7 +605,7 @@ diff_stdout: def test_diff_stderr_custom_function(self): """Test diff_stderr() using a custom function""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def my_diff(a, b): return ["a:"] + a + ["b:"] + b @@ -623,7 +623,7 @@ def def test_diff_stderr_TestCmd_function(self): """Test diff_stderr() using a TestCmd function""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stderr = TestCmd.diff_re) test.diff_stderr('a\n.\n', 'b\nc\n') @@ -639,7 +639,7 @@ sys.exit(0) def test_diff_stderr_static_method(self): """Test diff_stderr() using a static method""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stderr=TestCmd.TestCmd.diff_re) test.diff_stderr('a\n.\n', 'b\nc\n') @@ -655,7 +655,7 @@ sys.exit(0) def test_diff_stderr_string(self): """Test diff_stderr() using a string to fetch the diff method""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stderr='diff_re') test.diff_stderr('a\n.\n', 'b\nc\n') @@ -674,7 +674,7 @@ class diff_stdout_TestCase(TestCmdTestCase): def test_diff_stdout_default(self): """Test diff_stdout() default behavior""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd() test.diff_stdout('a\nb1\nc\n', 'a\nb2\nc\n') @@ -691,7 +691,7 @@ sys.exit(0) """Test diff_stdout() not affecting diff_stderr() behavior""" self.popen_python(r""" import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stdout='diff_re') print("diff_stdout:") @@ -712,7 +712,7 @@ diff_stderr: def test_diff_stdout_custom_function(self): """Test diff_stdout() using a custom function""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def my_diff(a, b): return ["a:"] + a + ["b:"] + b @@ -730,7 +730,7 @@ def def test_diff_stdout_TestCmd_function(self): """Test diff_stdout() using a TestCmd function""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stdout = TestCmd.diff_re) test.diff_stdout('a\n.\n', 'b\nc\n') @@ -746,7 +746,7 @@ sys.exit(0) def test_diff_stdout_static_method(self): """Test diff_stdout() using a static method""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stdout=TestCmd.TestCmd.diff_re) test.diff_stdout('a\n.\n', 'b\nc\n') @@ -762,7 +762,7 @@ sys.exit(0) def test_diff_stdout_string(self): """Test diff_stdout() using a string to fetch the diff method""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(diff_stdout='diff_re') test.diff_stdout('a\n.\n', 'b\nc\n') @@ -788,7 +788,7 @@ class exit_TestCase(TestCmdTestCase): 'no_result': "NO RESULT for test at line 5 of <stdin>\n"} global ExitError input = """import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(workdir = '%s') test.%s() @@ -863,13 +863,13 @@ sys.stderr.write("run: STDERR\\n") # Everything before this prepared our "source directory." # Now do the real test. self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd TestCmd.fail_test(condition = 1) """ % self.orig_cwd, status = 1, stderr = "FAILED test at line 4 of <stdin>\n") self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '') test.run() @@ -877,7 +877,7 @@ test.fail_test(condition = (test.status == 0)) """ % self.orig_cwd, status = 1, stderr = "FAILED test of %s\n\tat line 6 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', description = 'xyzzy', workdir = '') test.run() @@ -885,7 +885,7 @@ test.fail_test(condition = (test.status == 0)) """ % self.orig_cwd, status = 1, stderr = "FAILED test of %s [xyzzy]\n\tat line 6 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '') test.run() @@ -895,7 +895,7 @@ test.fail_test(condition = (test.status == 0), function = xxx) """ % self.orig_cwd, status = 1, stderr = "printed on failure\nFAILED test of %s\n\tat line 8 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def test1(self): self.run() @@ -906,7 +906,7 @@ test2(TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '')) """ % self.orig_cwd, status = 1, stderr = "FAILED test of %s\n\tat line 6 of <stdin> (test1)\n\tfrom line 8 of <stdin> (test2)\n\tfrom line 9 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def test1(self): self.run() @@ -1063,7 +1063,7 @@ class match_re_dotall_TestCase(TestCmdTestCase): # Now do the real test. try: script_input = """import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd assert TestCmd.match_re_dotall("abcde", r"a.*(e") sys.exit(0) @@ -1136,7 +1136,7 @@ class match_re_TestCase(TestCmdTestCase): # Now do the real test. try: script_input = """import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd assert TestCmd.match_re("abcde\\n", "a.*(e\\n") sys.exit(0) @@ -1346,13 +1346,13 @@ sys.stderr.write("run: STDERR\\n") # Everything before this prepared our "source directory." # Now do the real test. self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd TestCmd.no_result(condition = 1) """ % self.orig_cwd, status = 2, stderr = "NO RESULT for test at line 4 of <stdin>\n") self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '') test.run() @@ -1360,7 +1360,7 @@ test.no_result(condition = (test.status == 0)) """ % self.orig_cwd, status = 2, stderr = "NO RESULT for test of %s\n\tat line 6 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', description = 'xyzzy', workdir = '') test.run() @@ -1368,7 +1368,7 @@ test.no_result(condition = (test.status == 0)) """ % self.orig_cwd, status = 2, stderr = "NO RESULT for test of %s [xyzzy]\n\tat line 6 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '') test.run() @@ -1378,7 +1378,7 @@ test.no_result(condition = (test.status == 0), function = xxx) """ % self.orig_cwd, status = 2, stderr = "printed on no result\nNO RESULT for test of %s\n\tat line 8 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def test1(self): self.run() @@ -1389,7 +1389,7 @@ test2(TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '')) """ % self.orig_cwd, status = 2, stderr = "NO RESULT for test of %s\n\tat line 6 of <stdin> (test1)\n\tfrom line 8 of <stdin> (test2)\n\tfrom line 9 of <stdin>\n" % run_env.workpath('run')) self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd def test1(self): self.run() @@ -1413,13 +1413,13 @@ sys.stderr.write("run: STDERR\\n") # Everything before this prepared our "source directory." # Now do the real test. self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd TestCmd.pass_test(condition = 1) """ % self.orig_cwd, stderr = "PASSED\n") self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '') test.run() @@ -1427,7 +1427,7 @@ test.pass_test(condition = (test.status == 0)) """ % self.orig_cwd, stderr = "PASSED\n") self.popen_python("""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd(program = 'run', interpreter = 'python', workdir = '') test.run() @@ -2051,7 +2051,7 @@ class set_diff_function_TestCase(TestCmdTestCase): def test_set_diff_function(self): """Test set_diff_function()""" self.popen_python(r"""import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd() test.diff("a\n", "a\n") @@ -2064,7 +2064,7 @@ sys.exit(0) """Test set_diff_function(): stdout""" self.popen_python("""\ import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd() print("diff:") @@ -2093,7 +2093,7 @@ diff_stdout: """Test set_diff_function(): stderr """ self.popen_python("""\ import sys -sys.path = ['%s'] + sys.path +sys.path = [r'%s'] + sys.path import TestCmd test = TestCmd.TestCmd() print("diff:") @@ -2261,6 +2261,12 @@ class command_args_TestCase(TestCmdTestCase): expect = ['PYTHON', default_prog, 'arg3', 'arg4'] assert r == expect, (expect, r) + # Test arguments = dict + r = test.command_args(interpreter='PYTHON', arguments={'VAR1':'1'}) + expect = ['PYTHON', default_prog, 'VAR1=1'] + assert r == expect, (expect, r) + + test.interpreter_set('default_python') r = test.command_args() @@ -2585,7 +2591,7 @@ script_recv: STDERR: input with open(t.recv_out_path, 'rb') as f: result = to_str(f.read()) expect = 'script_recv: ' + input - assert result == expect, repr(result) + assert result == expect, "Result:[%s] should match\nExpected:[%s]" % (result, expect) p = test.start(stdin=1) input = 'send() input to the receive script\n' @@ -3103,7 +3109,7 @@ class workpath_TestCase(TestCmdTestCase): assert wpath == os.path.join(test.workdir, 'foo', 'bar') - +@unittest.skipIf(sys.platform == 'win32', "Don't run on win32") class readable_TestCase(TestCmdTestCase): def test_readable(self): """Test readable()""" @@ -3183,7 +3189,7 @@ class writable_TestCase(TestCmdTestCase): assert not _is_writable(test.workpath('file1')) - +@unittest.skipIf(sys.platform == 'win32', "Don't run on win32") class executable_TestCase(TestCmdTestCase): def test_executable(self): """Test executable()""" @@ -3331,70 +3337,8 @@ class variables_TestCase(TestCmdTestCase): assert stderr == "", stderr - if __name__ == "__main__": - tclasses = [ - __init__TestCase, - basename_TestCase, - cleanup_TestCase, - chmod_TestCase, - combine_TestCase, - command_args_TestCase, - description_TestCase, - diff_TestCase, - diff_stderr_TestCase, - diff_stdout_TestCase, - exit_TestCase, - fail_test_TestCase, - interpreter_TestCase, - match_TestCase, - match_exact_TestCase, - match_re_dotall_TestCase, - match_re_TestCase, - match_stderr_TestCase, - match_stdout_TestCase, - no_result_TestCase, - pass_test_TestCase, - preserve_TestCase, - program_TestCase, - read_TestCase, - rmdir_TestCase, - run_TestCase, - run_verbose_TestCase, - set_diff_function_TestCase, - set_match_function_TestCase, - sleep_TestCase, - start_TestCase, - stderr_TestCase, - stdin_TestCase, - stdout_TestCase, - subdir_TestCase, - symlink_TestCase, - tempdir_TestCase, - timeout_TestCase, - unlink_TestCase, - touch_TestCase, - verbose_TestCase, - workdir_TestCase, - workdirs_TestCase, - workpath_TestCase, - writable_TestCase, - write_TestCase, - variables_TestCase, - ] - if sys.platform != 'win32': - tclasses.extend([ - executable_TestCase, - readable_TestCase, - ]) - suite = unittest.TestSuite() - for tclass in tclasses: - loader = unittest.TestLoader() - loader.testMethodPrefix = 'test_' - names = loader.getTestCaseNames(tclass) - suite.addTests([tclass(n) for n in names]) - if not unittest.TextTestRunner().run(suite).wasSuccessful(): - sys.exit(1) + unittest.main() # Local Variables: # tab-width:4 diff --git a/testing/framework/TestCommonTests.py b/testing/framework/TestCommonTests.py index dff7a50..c8ea130 100644 --- a/testing/framework/TestCommonTests.py +++ b/testing/framework/TestCommonTests.py @@ -1840,7 +1840,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) tc.run(arguments = "arg1 arg2 arg3", @@ -1868,7 +1868,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(fail_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run() """) @@ -1897,7 +1897,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(stderr_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run() """) @@ -1929,8 +1929,8 @@ class run_TestCase(TestCommonTestCase): def raise_exception(*args, **kw): raise TypeError("forced TypeError") TestCmd.TestCmd.start = raise_exception - tc = TestCommon(program='%(pass_script)s', - interpreter='%(python)s', + tc = TestCommon(program=r'%(pass_script)s', + interpreter=r'%(python)s', workdir='') tc.run() """) @@ -1966,7 +1966,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(stderr_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run(stderr = None) """) @@ -1980,7 +1980,7 @@ class run_TestCase(TestCommonTestCase): def my_match_exact(actual, expect): return actual == expect from TestCommon import TestCommon, match_re_dotall tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_re_dotall) tc.run(arguments = "arg1 arg2 arg3", @@ -1997,7 +1997,7 @@ class run_TestCase(TestCommonTestCase): def my_match_exact(actual, expect): return actual == expect from TestCommon import TestCommon, match_re_dotall tc = TestCommon(program=r'%(stderr_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_re_dotall) tc.run(arguments = "arg1 arg2 arg3", @@ -2013,7 +2013,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(fail_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run(status = 1) """) @@ -2026,7 +2026,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) tc.run(stdout = r"%(pass_script)s: STDOUT: []" + "\\n") @@ -2040,7 +2040,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(stderr_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) tc.run(stderr = r"%(stderr_script)s: STDERR: []" + "\\n") @@ -2054,7 +2054,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run(status = 1) """) @@ -2084,7 +2084,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(fail_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run(status = 2) """) @@ -2114,7 +2114,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run(stdout = "Not found\\n") """) @@ -2146,7 +2146,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(stderr_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run(stderr = "Not found\\n") """) @@ -2180,7 +2180,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) tc.run(options = "opt1 opt2 opt3", @@ -2195,7 +2195,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) tc.run(options = "opt1 opt2 opt3", @@ -2217,7 +2217,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon tc = TestCommon(program=r'%(signal_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='') tc.run() """) @@ -2249,7 +2249,7 @@ class run_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(stdin_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir='', match=match_exact) expect_stdout = r"%(stdin_script)s: STDOUT: 'input'" + "\\n" @@ -2277,7 +2277,7 @@ class start_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) p = tc.start(options = "opt1 opt2 opt3") @@ -2293,7 +2293,7 @@ class start_TestCase(TestCommonTestCase): script = lstrip("""\ from TestCommon import TestCommon, match_exact tc = TestCommon(program=r'%(pass_script)s', - interpreter='%(python)s', + interpreter=r'%(python)s', workdir="", match=match_exact) p = tc.start(options = "opt1 opt2 opt3", |