diff options
Diffstat (limited to 'Help/prop_test')
23 files changed, 417 insertions, 0 deletions
diff --git a/Help/prop_test/ATTACHED_FILES.rst b/Help/prop_test/ATTACHED_FILES.rst new file mode 100644 index 0000000..496d800 --- /dev/null +++ b/Help/prop_test/ATTACHED_FILES.rst @@ -0,0 +1,7 @@ +ATTACHED_FILES +-------------- + +Attach a list of files to a dashboard submission. + +Set this property to a list of files that will be encoded and +submitted to the dashboard as an addition to the test result. diff --git a/Help/prop_test/ATTACHED_FILES_ON_FAIL.rst b/Help/prop_test/ATTACHED_FILES_ON_FAIL.rst new file mode 100644 index 0000000..add54b2 --- /dev/null +++ b/Help/prop_test/ATTACHED_FILES_ON_FAIL.rst @@ -0,0 +1,7 @@ +ATTACHED_FILES_ON_FAIL +---------------------- + +Attach a list of files to a dashboard submission if the test fails. + +Same as :prop_test:`ATTACHED_FILES`, but these files will only be +included if the test does not pass. diff --git a/Help/prop_test/COST.rst b/Help/prop_test/COST.rst new file mode 100644 index 0000000..0c0fca7 --- /dev/null +++ b/Help/prop_test/COST.rst @@ -0,0 +1,8 @@ +COST +---- + +Set this to a floating point value. Tests in a test set will be run +in descending order of cost. + +This property describes the cost of a test. You can explicitly set +this value; tests with higher ``COST`` values will run first. diff --git a/Help/prop_test/DEPENDS.rst b/Help/prop_test/DEPENDS.rst new file mode 100644 index 0000000..89c7553 --- /dev/null +++ b/Help/prop_test/DEPENDS.rst @@ -0,0 +1,10 @@ +DEPENDS +------- + +Specifies that this test should only be run after the specified list of tests. + +Set this to a list of tests that must finish before this test is run. The +results of those tests are not considered, the dependency relationship is +purely for order of execution (i.e. it is really just a *run after* +relationship). Consider using test fixtures with setup tests if a dependency +with successful completion is required (see :prop_test:`FIXTURES_REQUIRED`). diff --git a/Help/prop_test/DISABLED.rst b/Help/prop_test/DISABLED.rst new file mode 100644 index 0000000..1d469e8 --- /dev/null +++ b/Help/prop_test/DISABLED.rst @@ -0,0 +1,15 @@ +DISABLED +-------- + +If set to ``True``, the test will be skipped and its status will be 'Not Run'. A +``DISABLED`` test will not be counted in the total number of tests and its +completion status will be reported to CDash as ``Disabled``. + +A ``DISABLED`` test does not participate in test fixture dependency resolution. +If a ``DISABLED`` test has fixture requirements defined in its +:prop_test:`FIXTURES_REQUIRED` property, it will not cause setup or cleanup +tests for those fixtures to be added to the test set. + +If a test with the :prop_test:`FIXTURES_SETUP` property set is ``DISABLED``, +the fixture behavior will be as though that setup test was passing and any test +case requiring that fixture will still run. diff --git a/Help/prop_test/ENVIRONMENT.rst b/Help/prop_test/ENVIRONMENT.rst new file mode 100644 index 0000000..102c792 --- /dev/null +++ b/Help/prop_test/ENVIRONMENT.rst @@ -0,0 +1,9 @@ +ENVIRONMENT +----------- + +Specify environment variables that should be defined for running a test. + +If set to a list of environment variables and values of the form +``MYVAR=value`` those environment variables will be defined while running +the test. The environment is restored to its previous state after the +test is done. diff --git a/Help/prop_test/FAIL_REGULAR_EXPRESSION.rst b/Help/prop_test/FAIL_REGULAR_EXPRESSION.rst new file mode 100644 index 0000000..facf902 --- /dev/null +++ b/Help/prop_test/FAIL_REGULAR_EXPRESSION.rst @@ -0,0 +1,15 @@ +FAIL_REGULAR_EXPRESSION +----------------------- + +If the output matches this regular expression the test will fail. + +If set, if the output matches one of specified regular expressions, +the test will fail. Example: + +.. code-block:: cmake + + set_tests_properties(mytest PROPERTIES + FAIL_REGULAR_EXPRESSION "[^a-z]Error;ERROR;Failed" + ) + +``FAIL_REGULAR_EXPRESSION`` expects a list of regular expressions. diff --git a/Help/prop_test/FIXTURES_CLEANUP.rst b/Help/prop_test/FIXTURES_CLEANUP.rst new file mode 100644 index 0000000..3075b4d --- /dev/null +++ b/Help/prop_test/FIXTURES_CLEANUP.rst @@ -0,0 +1,47 @@ +FIXTURES_CLEANUP +---------------- + +Specifies a list of fixtures for which the test is to be treated as a cleanup +test. These fixture names are distinct from test case names and are not +required to have any similarity to the names of tests associated with them. + +Fixture cleanup tests are ordinary tests with all of the usual test +functionality. Setting the ``FIXTURES_CLEANUP`` property for a test has two +primary effects: + +- CTest will ensure the test executes after all other tests which list any of + the fixtures in its :prop_test:`FIXTURES_REQUIRED` property. + +- If CTest is asked to run only a subset of tests (e.g. using regular + expressions or the ``--rerun-failed`` option) and the cleanup test is not in + the set of tests to run, it will automatically be added if any tests in the + set require any fixture listed in ``FIXTURES_CLEANUP``. + +A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP`` +property. It will execute only once for the whole CTest run, not once for each +fixture. A fixture can also have more than one cleanup test defined. If there +are multiple cleanup tests for a fixture, projects can control their order with +the usual :prop_test:`DEPENDS` test property if necessary. + +A cleanup test is allowed to require other fixtures, but not any fixture listed +in its ``FIXTURES_CLEANUP`` property. For example: + +.. code-block:: cmake + + # Ok: Dependent fixture is different to cleanup + set_tests_properties(cleanupFoo PROPERTIES + FIXTURES_CLEANUP Foo + FIXTURES_REQUIRED Bar + ) + + # Error: cannot require same fixture as cleanup + set_tests_properties(cleanupFoo PROPERTIES + FIXTURES_CLEANUP Foo + FIXTURES_REQUIRED Foo + ) + +Cleanup tests will execute even if setup or regular tests for that fixture fail +or are skipped. + +See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use +test fixtures. diff --git a/Help/prop_test/FIXTURES_REQUIRED.rst b/Help/prop_test/FIXTURES_REQUIRED.rst new file mode 100644 index 0000000..e3f60c4 --- /dev/null +++ b/Help/prop_test/FIXTURES_REQUIRED.rst @@ -0,0 +1,96 @@ +FIXTURES_REQUIRED +----------------- + +Specifies a list of fixtures the test requires. Fixture names are case +sensitive and they are not required to have any similarity to test names. + +Fixtures are a way to attach setup and cleanup tasks to a set of tests. If a +test requires a given fixture, then all tests marked as setup tasks for that +fixture will be executed first (once for the whole set of tests, not once per +test requiring the fixture). After all tests requiring a particular fixture +have completed, CTest will ensure all tests marked as cleanup tasks for that +fixture are then executed. Tests are marked as setup tasks with the +:prop_test:`FIXTURES_SETUP` property and as cleanup tasks with the +:prop_test:`FIXTURES_CLEANUP` property. If any of a fixture's setup tests fail, +all tests listing that fixture in their ``FIXTURES_REQUIRED`` property will not +be executed. The cleanup tests for the fixture will always be executed, even if +some setup tests fail. + +When CTest is asked to execute only a subset of tests (e.g. by the use of +regular expressions or when run with the ``--rerun-failed`` command line +option), it will automatically add any setup or cleanup tests for fixtures +required by any of the tests that are in the execution set. This behavior can +be overridden with the ``-FS``, ``-FC`` and ``-FA`` command line options to +:manual:`ctest(1)` if desired. + +Since setup and cleanup tasks are also tests, they can have an ordering +specified by the :prop_test:`DEPENDS` test property just like any other tests. +This can be exploited to implement setup or cleanup using multiple tests for a +single fixture to modularise setup or cleanup logic. + +The concept of a fixture is different to that of a resource specified by +:prop_test:`RESOURCE_LOCK`, but they may be used together. A fixture defines a +set of tests which share setup and cleanup requirements, whereas a resource +lock has the effect of ensuring a particular set of tests do not run in +parallel. Some situations may need both, such as setting up a database, +serialising test access to that database and deleting the database again at the +end. For such cases, tests would populate both ``FIXTURES_REQUIRED`` and +:prop_test:`RESOURCE_LOCK` to combine the two behaviours. Names used for +:prop_test:`RESOURCE_LOCK` have no relationship with names of fixtures, so note +that a resource lock does not imply a fixture and vice versa. + +Consider the following example which represents a database test scenario +similar to that mentioned above: + +.. code-block:: cmake + + add_test(NAME testsDone COMMAND emailResults) + add_test(NAME fooOnly COMMAND testFoo) + add_test(NAME dbOnly COMMAND testDb) + add_test(NAME dbWithFoo COMMAND testDbWithFoo) + add_test(NAME createDB COMMAND initDB) + add_test(NAME setupUsers COMMAND userCreation) + add_test(NAME cleanupDB COMMAND deleteDB) + add_test(NAME cleanupFoo COMMAND removeFoos) + + set_tests_properties(setupUsers PROPERTIES DEPENDS createDB) + + set_tests_properties(createDB PROPERTIES FIXTURES_SETUP DB) + set_tests_properties(setupUsers PROPERTIES FIXTURES_SETUP DB) + set_tests_properties(cleanupDB PROPERTIES FIXTURES_CLEANUP DB) + set_tests_properties(cleanupFoo PROPERTIES FIXTURES_CLEANUP Foo) + set_tests_properties(testsDone PROPERTIES FIXTURES_CLEANUP "DB;Foo") + + set_tests_properties(fooOnly PROPERTIES FIXTURES_REQUIRED Foo) + set_tests_properties(dbOnly PROPERTIES FIXTURES_REQUIRED DB) + set_tests_properties(dbWithFoo PROPERTIES FIXTURES_REQUIRED "DB;Foo") + + set_tests_properties(dbOnly dbWithFoo createDB setupUsers cleanupDB + PROPERTIES RESOURCE_LOCK DbAccess) + +Key points from this example: + +- Two fixtures are defined: ``DB`` and ``Foo``. Tests can require a single + fixture as ``fooOnly`` and ``dbOnly`` do, or they can depend on multiple + fixtures like ``dbWithFoo`` does. + +- A ``DEPENDS`` relationship is set up to ensure ``setupUsers`` happens after + ``createDB``, both of which are setup tests for the ``DB`` fixture and will + therefore be executed before the ``dbOnly`` and ``dbWithFoo`` tests + automatically. + +- No explicit ``DEPENDS`` relationships were needed to make the setup tests run + before or the cleanup tests run after the regular tests. + +- The ``Foo`` fixture has no setup tests defined, only a single cleanup test. + +- ``testsDone`` is a cleanup test for both the ``DB`` and ``Foo`` fixtures. + Therefore, it will only execute once regular tests for both fixtures have + finished (i.e. after ``fooOnly``, ``dbOnly`` and ``dbWithFoo``). No + ``DEPENDS`` relationship was specified for ``testsDone``, so it is free to + run before, after or concurrently with other cleanup tests for either + fixture. + +- The setup and cleanup tests never list the fixtures they are for in their own + ``FIXTURES_REQUIRED`` property, as that would result in a dependency on + themselves and be considered an error. diff --git a/Help/prop_test/FIXTURES_SETUP.rst b/Help/prop_test/FIXTURES_SETUP.rst new file mode 100644 index 0000000..fdb21cc --- /dev/null +++ b/Help/prop_test/FIXTURES_SETUP.rst @@ -0,0 +1,48 @@ +FIXTURES_SETUP +-------------- + +Specifies a list of fixtures for which the test is to be treated as a setup +test. These fixture names are distinct from test case names and are not +required to have any similarity to the names of tests associated with them. + +Fixture setup tests are ordinary tests with all of the usual test +functionality. Setting the ``FIXTURES_SETUP`` property for a test has two +primary effects: + +- CTest will ensure the test executes before any other test which lists the + fixture name(s) in its :prop_test:`FIXTURES_REQUIRED` property. + +- If CTest is asked to run only a subset of tests (e.g. using regular + expressions or the ``--rerun-failed`` option) and the setup test is not in + the set of tests to run, it will automatically be added if any tests in the + set require any fixture listed in ``FIXTURES_SETUP``. + +A setup test can have multiple fixtures listed in its ``FIXTURES_SETUP`` +property. It will execute only once for the whole CTest run, not once for each +fixture. A fixture can also have more than one setup test defined. If there are +multiple setup tests for a fixture, projects can control their order with the +usual :prop_test:`DEPENDS` test property if necessary. + +A setup test is allowed to require other fixtures, but not any fixture listed +in its ``FIXTURES_SETUP`` property. For example: + +.. code-block:: cmake + + # Ok: dependent fixture is different to setup + set_tests_properties(setupFoo PROPERTIES + FIXTURES_SETUP Foo + FIXTURES_REQUIRED Bar + ) + + # Error: cannot require same fixture as setup + set_tests_properties(setupFoo PROPERTIES + FIXTURES_SETUP Foo + FIXTURES_REQUIRED Foo + ) + +If any of a fixture's setup tests fail, none of the tests listing that fixture +in its :prop_test:`FIXTURES_REQUIRED` property will be run. Cleanup tests will, +however, still be executed. + +See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use +test fixtures. diff --git a/Help/prop_test/LABELS.rst b/Help/prop_test/LABELS.rst new file mode 100644 index 0000000..8d75570 --- /dev/null +++ b/Help/prop_test/LABELS.rst @@ -0,0 +1,6 @@ +LABELS +------ + +Specify a list of text labels associated with a test. + +The list is reported in dashboard submissions. diff --git a/Help/prop_test/MEASUREMENT.rst b/Help/prop_test/MEASUREMENT.rst new file mode 100644 index 0000000..de459ed --- /dev/null +++ b/Help/prop_test/MEASUREMENT.rst @@ -0,0 +1,8 @@ +MEASUREMENT +----------- + +Specify a ``CDASH`` measurement and value to be reported for a test. + +If set to a name then that name will be reported to ``CDASH`` as a named +measurement with a value of ``1``. You may also specify a value by +setting ``MEASUREMENT`` to ``measurement=value``. diff --git a/Help/prop_test/PASS_REGULAR_EXPRESSION.rst b/Help/prop_test/PASS_REGULAR_EXPRESSION.rst new file mode 100644 index 0000000..0cd6215 --- /dev/null +++ b/Help/prop_test/PASS_REGULAR_EXPRESSION.rst @@ -0,0 +1,16 @@ +PASS_REGULAR_EXPRESSION +----------------------- + +The output must match this regular expression for the test to pass. + +If set, the test output will be checked against the specified regular +expressions and at least one of the regular expressions has to match, +otherwise the test will fail. Example: + +.. code-block:: cmake + + set_tests_properties(mytest PROPERTIES + PASS_REGULAR_EXPRESSION "TestPassed;All ok" + ) + +``PASS_REGULAR_EXPRESSION`` expects a list of regular expressions. diff --git a/Help/prop_test/PROCESSORS.rst b/Help/prop_test/PROCESSORS.rst new file mode 100644 index 0000000..a927c10 --- /dev/null +++ b/Help/prop_test/PROCESSORS.rst @@ -0,0 +1,16 @@ +PROCESSORS +---------- + +Set to specify how many process slots this test requires. +If not set, the default is ``1`` processor. + +Denotes the number of processors that this test will require. This is +typically used for MPI tests, and should be used in conjunction with +the :command:`ctest_test` ``PARALLEL_LEVEL`` option. + +This will also be used to display a weighted test timing result in label and +subproject summaries in the command line output of :manual:`ctest(1)`. The wall +clock time for the test run will be multiplied by this property to give a +better idea of how much cpu resource CTest allocated for the test. + +See also the :prop_test:`PROCESSOR_AFFINITY` test property. diff --git a/Help/prop_test/PROCESSOR_AFFINITY.rst b/Help/prop_test/PROCESSOR_AFFINITY.rst new file mode 100644 index 0000000..38ec179 --- /dev/null +++ b/Help/prop_test/PROCESSOR_AFFINITY.rst @@ -0,0 +1,11 @@ +PROCESSOR_AFFINITY +------------------ + +Set to a true value to ask CTest to launch the test process with CPU affinity +for a fixed set of processors. If enabled and supported for the current +platform, CTest will choose a set of processors to place in the CPU affinity +mask when launching the test process. The number of processors in the set is +determined by the :prop_test:`PROCESSORS` test property or the number of +processors available to CTest, whichever is smaller. The set of processors +chosen will be disjoint from the processors assigned to other concurrently +running tests that also have the ``PROCESSOR_AFFINITY`` property enabled. diff --git a/Help/prop_test/REQUIRED_FILES.rst b/Help/prop_test/REQUIRED_FILES.rst new file mode 100644 index 0000000..fac357c --- /dev/null +++ b/Help/prop_test/REQUIRED_FILES.rst @@ -0,0 +1,7 @@ +REQUIRED_FILES +-------------- + +List of files required to run the test. + +If set to a list of files, the test will not be run unless all of the +files exist. diff --git a/Help/prop_test/RESOURCE_LOCK.rst b/Help/prop_test/RESOURCE_LOCK.rst new file mode 100644 index 0000000..755e0aa --- /dev/null +++ b/Help/prop_test/RESOURCE_LOCK.rst @@ -0,0 +1,10 @@ +RESOURCE_LOCK +------------- + +Specify a list of resources that are locked by this test. + +If multiple tests specify the same resource lock, they are guaranteed +not to run concurrently. + +See also :prop_test:`FIXTURES_REQUIRED` if the resource requires any setup or +cleanup steps. diff --git a/Help/prop_test/RUN_SERIAL.rst b/Help/prop_test/RUN_SERIAL.rst new file mode 100644 index 0000000..ab4c542 --- /dev/null +++ b/Help/prop_test/RUN_SERIAL.rst @@ -0,0 +1,8 @@ +RUN_SERIAL +---------- + +Do not run this test in parallel with any other test. + +Use this option in conjunction with the ctest_test ``PARALLEL_LEVEL`` +option to specify that this test should not be run in parallel with +any other tests. diff --git a/Help/prop_test/SKIP_RETURN_CODE.rst b/Help/prop_test/SKIP_RETURN_CODE.rst new file mode 100644 index 0000000..a05fbf3 --- /dev/null +++ b/Help/prop_test/SKIP_RETURN_CODE.rst @@ -0,0 +1,9 @@ +SKIP_RETURN_CODE +---------------- + +Return code to mark a test as skipped. + +Sometimes only a test itself can determine if all requirements for the +test are met. If such a situation should not be considered a hard failure +a return code of the process can be specified that will mark the test as +``Not Run`` if it is encountered. diff --git a/Help/prop_test/TIMEOUT.rst b/Help/prop_test/TIMEOUT.rst new file mode 100644 index 0000000..d1cb90d --- /dev/null +++ b/Help/prop_test/TIMEOUT.rst @@ -0,0 +1,9 @@ +TIMEOUT +------- + +How many seconds to allow for this test. + +This property if set will limit a test to not take more than the +specified number of seconds to run. If it exceeds that the test +process will be killed and ctest will move to the next test. This +setting takes precedence over :variable:`CTEST_TEST_TIMEOUT`. diff --git a/Help/prop_test/TIMEOUT_AFTER_MATCH.rst b/Help/prop_test/TIMEOUT_AFTER_MATCH.rst new file mode 100644 index 0000000..d607992 --- /dev/null +++ b/Help/prop_test/TIMEOUT_AFTER_MATCH.rst @@ -0,0 +1,39 @@ +TIMEOUT_AFTER_MATCH +------------------- + +Change a test's timeout duration after a matching line is encountered +in its output. + +Usage +^^^^^ + +.. code-block:: cmake + + add_test(mytest ...) + set_property(TEST mytest PROPERTY TIMEOUT_AFTER_MATCH "${seconds}" "${regex}") + +Description +^^^^^^^^^^^ + +Allow a test ``seconds`` to complete after ``regex`` is encountered in +its output. + +When the test outputs a line that matches ``regex`` its start time is +reset to the current time and its timeout duration is changed to +``seconds``. Prior to this, the timeout duration is determined by the +:prop_test:`TIMEOUT` property or the :variable:`CTEST_TEST_TIMEOUT` +variable if either of these are set. Because the test's start time is +reset, its execution time will not include any time that was spent +waiting for the matching output. + +:prop_test:`TIMEOUT_AFTER_MATCH` is useful for avoiding spurious +timeouts when your test must wait for some system resource to become +available before it can execute. Set :prop_test:`TIMEOUT` to a longer +duration that accounts for resource acquisition and use +:prop_test:`TIMEOUT_AFTER_MATCH` to control how long the actual test +is allowed to run. + +If the required resource can be controlled by CTest you should use +:prop_test:`RESOURCE_LOCK` instead of :prop_test:`TIMEOUT_AFTER_MATCH`. +This property should be used when only the test itself can determine +when its required resources are available. diff --git a/Help/prop_test/WILL_FAIL.rst b/Help/prop_test/WILL_FAIL.rst new file mode 100644 index 0000000..f1f94a4 --- /dev/null +++ b/Help/prop_test/WILL_FAIL.rst @@ -0,0 +1,7 @@ +WILL_FAIL +--------- + +If set to true, this will invert the pass/fail flag of the test. + +This property can be used for tests that are expected to fail and +return a non zero return code. diff --git a/Help/prop_test/WORKING_DIRECTORY.rst b/Help/prop_test/WORKING_DIRECTORY.rst new file mode 100644 index 0000000..92a0409 --- /dev/null +++ b/Help/prop_test/WORKING_DIRECTORY.rst @@ -0,0 +1,9 @@ +WORKING_DIRECTORY +----------------- + +The directory from which the test executable will be called. + +If this is not set, the test will be run with the working directory set to the +binary directory associated with where the test was created (i.e. the +:variable:`CMAKE_CURRENT_BINARY_DIR` for where :command:`add_test` was +called). |