ctest: Optionally avoid starting tests that may exceed a given CPU load

Add a TestLoad setting to CTest that can be set via a new --test-load
command-line option, CTEST_TEST_LOAD variable, or TEST_LOAD option to
the ctest_test command.  Teach cmCTestMultiProcessHandler to measure
the CPU load and avoid starting tests that may take more than the
spare load currently available.  The expression

 <current_load> + <test_processors> <= <max-load>

must be true to start a new test.

Co-Author: Zack Galbreath <zack.galbreath@kitware.com>

6 files changed, 40 insertions, 0 deletions

diff --git a/Help/command/ctest_memcheck.rst b/Help/command/ctest_memcheck.rst
index 2800511..29bdf7d 100644
--- a/Help/command/ctest_memcheck.rst
+++ b/Help/command/ctest_memcheck.rst
@@ -14,6 +14,7 @@ Perform the :ref:`CTest MemCheck Step` as a :ref:`Dashboard Client`.
                  [EXCLUDE_LABEL <label-exclude-regex>]
                  [INCLUDE_LABEL <label-include-regex>]
                  [PARALLEL_LEVEL <level>]
+                 [TEST_LOAD <threshold>]
                  [SCHEDULE_RANDOM <ON|OFF>]
                  [STOP_TIME <time-of-day>]
                  [RETURN_VALUE <result-var>]
diff --git a/Help/command/ctest_test.rst b/Help/command/ctest_test.rst
index af422b6..162db69 100644
--- a/Help/command/ctest_test.rst
+++ b/Help/command/ctest_test.rst
@@ -14,6 +14,7 @@ Perform the :ref:`CTest Test Step` as a :ref:`Dashboard Client`.
              [EXCLUDE_LABEL <label-exclude-regex>]
              [INCLUDE_LABEL <label-include-regex>]
              [PARALLEL_LEVEL <level>]
+             [TEST_LOAD <threshold>]
              [SCHEDULE_RANDOM <ON|OFF>]
              [STOP_TIME <time-of-day>]
              [RETURN_VALUE <result-var>]
@@ -61,6 +62,13 @@ The options are:
   Specify a positive number representing the number of tests to
   be run in parallel.
 
+``TEST_LOAD <threshold>``
+  While running tests in parallel, try not to start tests when they
+  may cause the CPU load to pass above a given threshold.  If not
+  specified the :variable:`CTEST_TEST_LOAD` variable will be checked,
+  and then the ``--test-load`` command-line argument to :manual:`ctest(1)`.
+  See also the ``TestLoad`` setting in the :ref:`CTest Test Step`.
+
 ``SCHEDULE_RANDOM <ON|OFF>``
   Launch tests in a random order.  This may be useful for detecting
   implicit test dependencies.
diff --git a/Help/manual/cmake-variables.7.rst b/Help/manual/cmake-variables.7.rst
index 0e6222f..adbc40b 100644
--- a/Help/manual/cmake-variables.7.rst
+++ b/Help/manual/cmake-variables.7.rst
@@ -386,6 +386,7 @@ Variables for CTest
    /variable/CTEST_SVN_COMMAND
    /variable/CTEST_SVN_OPTIONS
    /variable/CTEST_SVN_UPDATE_OPTIONS
+   /variable/CTEST_TEST_LOAD
    /variable/CTEST_TEST_TIMEOUT
    /variable/CTEST_TRIGGER_SITE
    /variable/CTEST_UPDATE_COMMAND
diff --git a/Help/manual/ctest.1.rst b/Help/manual/ctest.1.rst
index c91321b..50c856a 100644
--- a/Help/manual/ctest.1.rst
+++ b/Help/manual/ctest.1.rst
@@ -66,6 +66,13 @@ Options
  number of jobs.  This option can also be set by setting the
  environment variable ``CTEST_PARALLEL_LEVEL``.
 
+``--test-load <level>``
+ While running tests in parallel (e.g. with ``-j``), try not to start
+ tests when they may cause the CPU load to pass above a given threshold.
+
+ When ``ctest`` is run as a `Dashboard Client`_ this sets the
+ ``TestLoad`` option of the `CTest Test Step`_.
+
 ``-Q,--quiet``
  Make ctest quiet.
 
@@ -776,6 +783,13 @@ Arguments to the command may specify some of the step settings.
 
 Configuration settings include:
 
+``TestLoad``
+  While running tests in parallel (e.g. with ``-j``), try not to start
+  tests when they may cause the CPU load to pass above a given threshold.
+
+  * `CTest Script`_ variable: :variable:`CTEST_TEST_LOAD`
+  * :module:`CTest` module variable: ``CTEST_TEST_LOAD``
+
 ``TimeOut``
   The default timeout for each test if not specified by the
   :prop_test:`TIMEOUT` test property.
diff --git a/Help/release/dev/ctest-test-load-option.rst b/Help/release/dev/ctest-test-load-option.rst
new file mode 100644
index 0000000..069f49a
--- /dev/null
+++ b/Help/release/dev/ctest-test-load-option.rst
@@ -0,0 +1,9 @@
+ctest-test-load-option
+----------------------
+
+* CTest learned to optionally measure the CPU load during parallel
+  testing and avoid starting tests that may cause the load to exceed
+  a given threshold.  See the :manual:`ctest(1)` command ``--test-load``
+  option, the ``TestLoad`` setting of the :ref:`CTest Test Step`,
+  the :variable:`CTEST_TEST_LOAD` variable, and the ``TEST_LOAD``
+  option of the :command:`ctest_test` command.
diff --git a/Help/variable/CTEST_TEST_LOAD.rst b/Help/variable/CTEST_TEST_LOAD.rst
new file mode 100644
index 0000000..80823fe
--- /dev/null
+++ b/Help/variable/CTEST_TEST_LOAD.rst
@@ -0,0 +1,7 @@
+CTEST_TEST_LOAD
+---------------
+
+Specify the ``TestLoad`` setting in the :ref:`CTest Test Step`
+of a :manual:`ctest(1)` dashboard client script.  This sets the
+default value for the ``TEST_LOAD`` option of the :command:`ctest_test`
+command.