diff options
Diffstat (limited to 'Help')
| -rw-r--r-- | Help/manual/cmake-properties.7.rst | 3 | ||||
| -rw-r--r-- | Help/prop_test/DEPENDS.rst | 6 | ||||
| -rw-r--r-- | Help/prop_test/FIXTURES_CLEANUP.rst | 46 | ||||
| -rw-r--r-- | Help/prop_test/FIXTURES_REQUIRED.rst | 94 | ||||
| -rw-r--r-- | Help/prop_test/FIXTURES_SETUP.rst | 47 | ||||
| -rw-r--r-- | Help/prop_test/RESOURCE_LOCK.rst | 3 | ||||
| -rw-r--r-- | Help/release/dev/test-fixtures.rst | 8 |
7 files changed, 206 insertions, 1 deletions
diff --git a/Help/manual/cmake-properties.7.rst b/Help/manual/cmake-properties.7.rst index 2cb6a1a..271f497 100644 --- a/Help/manual/cmake-properties.7.rst +++ b/Help/manual/cmake-properties.7.rst @@ -304,6 +304,9 @@ Properties on Tests /prop_test/DEPENDS /prop_test/ENVIRONMENT /prop_test/FAIL_REGULAR_EXPRESSION + /prop_test/FIXTURES_CLEANUP + /prop_test/FIXTURES_REQUIRED + /prop_test/FIXTURES_SETUP /prop_test/LABELS /prop_test/MEASUREMENT /prop_test/PASS_REGULAR_EXPRESSION diff --git a/Help/prop_test/DEPENDS.rst b/Help/prop_test/DEPENDS.rst index ee946d9..89c7553 100644 --- a/Help/prop_test/DEPENDS.rst +++ b/Help/prop_test/DEPENDS.rst @@ -3,4 +3,8 @@ DEPENDS Specifies that this test should only be run after the specified list of tests. -Set this to a list of tests that must finish before this test is run. +Set this to a list of tests that must finish before this test is run. The +results of those tests are not considered, the dependency relationship is +purely for order of execution (i.e. it is really just a *run after* +relationship). Consider using test fixtures with setup tests if a dependency +with successful completion is required (see :prop_test:`FIXTURES_REQUIRED`). diff --git a/Help/prop_test/FIXTURES_CLEANUP.rst b/Help/prop_test/FIXTURES_CLEANUP.rst new file mode 100644 index 0000000..f0a4be0 --- /dev/null +++ b/Help/prop_test/FIXTURES_CLEANUP.rst @@ -0,0 +1,46 @@ +FIXTURES_CLEANUP +---------------- + +Specifies a list of fixtures for which the test is to be treated as a cleanup +test. + +Fixture cleanup tests are ordinary tests with all of the usual test +functionality. Setting the ``FIXTURES_CLEANUP`` property for a test has two +primary effects: + +- CTest will ensure the test executes after all other tests which list any of + the fixtures in its :prop_test:`FIXTURES_REQUIRED` property. + +- If CTest is asked to run only a subset of tests (e.g. using regular + expressions or the ``--rerun-failed`` option) and the cleanup test is not in + the set of tests to run, it will automatically be added if any tests in the + set require any fixture listed in ``FIXTURES_CLEANUP``. + +A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP`` +property. It will execute only once for the whole CTest run, not once for each +fixture. A fixture can also have more than one cleanup test defined. If there +are multiple cleanup tests for a fixture, projects can control their order with +the usual :prop_test:`DEPENDS` test property if necessary. + +A cleanup test is allowed to require other fixtures, but not any fixture listed +in its ``FIXTURES_CLEANUP`` property. For example: + +.. code-block:: cmake + + # Ok: Dependent fixture is different to cleanup + set_tests_properties(cleanupFoo PROPERTIES + FIXTURES_CLEANUP Foo + FIXTURES_REQUIRED Bar + ) + + # Error: cannot require same fixture as cleanup + set_tests_properties(cleanupFoo PROPERTIES + FIXTURES_CLEANUP Foo + FIXTURES_REQUIRED Foo + ) + +Cleanup tests will execute even if setup or regular tests for that fixture fail +or are skipped. + +See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use +test fixtures. diff --git a/Help/prop_test/FIXTURES_REQUIRED.rst b/Help/prop_test/FIXTURES_REQUIRED.rst new file mode 100644 index 0000000..e37dfb5 --- /dev/null +++ b/Help/prop_test/FIXTURES_REQUIRED.rst @@ -0,0 +1,94 @@ +FIXTURES_REQUIRED +----------------- + +Specifies a list of fixtures the test requires. Fixture names are case +sensitive. + +Fixtures are a way to attach setup and cleanup tasks to a set of tests. If a +test requires a given fixture, then all tests marked as setup tasks for that +fixture will be executed first (once for the whole set of tests, not once per +test requiring the fixture). After all tests requiring a particular fixture +have completed, CTest will ensure all tests marked as cleanup tasks for that +fixture are then executed. Tests are marked as setup tasks with the +:prop_test:`FIXTURES_SETUP` property and as cleanup tasks with the +:prop_test:`FIXTURES_CLEANUP` property. If any of a fixture's setup tests fail, +all tests listing that fixture in their ``FIXTURES_REQUIRED`` property will not +be executed. The cleanup tests for the fixture will always be executed, even if +some setup tests fail. + +When CTest is asked to execute only a subset of tests (e.g. by the use of +regular expressions or when run with the ``--rerun-failed`` command line +option), it will automatically add any setup or cleanup tests for fixtures +required by any of the tests that are in the execution set. + +Since setup and cleanup tasks are also tests, they can have an ordering +specified by the :prop_test:`DEPENDS` test property just like any other tests. +This can be exploited to implement setup or cleanup using multiple tests for a +single fixture to modularise setup or cleanup logic. + +The concept of a fixture is different to that of a resource specified by +:prop_test:`RESOURCE_LOCK`, but they may be used together. A fixture defines a +set of tests which share setup and cleanup requirements, whereas a resource +lock has the effect of ensuring a particular set of tests do not run in +parallel. Some situations may need both, such as setting up a database, +serialising test access to that database and deleting the database again at the +end. For such cases, tests would populate both ``FIXTURES_REQUIRED`` and +:prop_test:`RESOURCE_LOCK` to combine the two behaviours. Names used for +:prop_test:`RESOURCE_LOCK` have no relationship with names of fixtures, so note +that a resource lock does not imply a fixture and vice versa. + +Consider the following example which represents a database test scenario +similar to that mentioned above: + +.. code-block:: cmake + + add_test(NAME testsDone COMMAND emailResults) + add_test(NAME fooOnly COMMAND testFoo) + add_test(NAME dbOnly COMMAND testDb) + add_test(NAME dbWithFoo COMMAND testDbWithFoo) + add_test(NAME createDB COMMAND initDB) + add_test(NAME setupUsers COMMAND userCreation) + add_test(NAME cleanupDB COMMAND deleteDB) + add_test(NAME cleanupFoo COMMAND removeFoos) + + set_tests_properties(setupUsers PROPERTIES DEPENDS createDB) + + set_tests_properties(createDB PROPERTIES FIXTURES_SETUP DB) + set_tests_properties(setupUsers PROPERTIES FIXTURES_SETUP DB) + set_tests_properties(cleanupDB PROPERTIES FIXTURES_CLEANUP DB) + set_tests_properties(cleanupFoo PROPERTIES FIXTURES_CLEANUP Foo) + set_tests_properties(testsDone PROPERTIES FIXTURES_CLEANUP "DB;Foo") + + set_tests_properties(fooOnly PROPERTIES FIXTURES_REQUIRED Foo) + set_tests_properties(dbOnly PROPERTIES FIXTURES_REQUIRED DB) + set_tests_properties(dbWithFoo PROPERTIES FIXTURES_REQUIRED "DB;Foo") + + set_tests_properties(dbOnly dbWithFoo createDB setupUsers cleanupDB + PROPERTIES RESOURCE_LOCK DbAccess) + +Key points from this example: + +- Two fixtures are defined: ``DB`` and ``Foo``. Tests can require a single + fixture as ``fooOnly`` and ``dbOnly`` do, or they can depend on multiple + fixtures like ``dbWithFoo`` does. + +- A ``DEPENDS`` relationship is set up to ensure ``setupUsers`` happens after + ``createDB``, both of which are setup tests for the ``DB`` fixture and will + therefore be executed before the ``dbOnly`` and ``dbWithFoo`` tests + automatically. + +- No explicit ``DEPENDS`` relationships were needed to make the setup tests run + before or the cleanup tests run after the regular tests. + +- The ``Foo`` fixture has no setup tests defined, only a single cleanup test. + +- ``testsDone`` is a cleanup test for both the ``DB`` and ``Foo`` fixtures. + Therefore, it will only execute once regular tests for both fixtures have + finished (i.e. after ``fooOnly``, ``dbOnly`` and ``dbWithFoo``). No + ``DEPENDS`` relationship was specified for ``testsDone``, so it is free to + run before, after or concurrently with other cleanup tests for either + fixture. + +- The setup and cleanup tests never list the fixtures they are for in their own + ``FIXTURES_REQUIRED`` property, as that would result in a dependency on + themselves and be considered an error. diff --git a/Help/prop_test/FIXTURES_SETUP.rst b/Help/prop_test/FIXTURES_SETUP.rst new file mode 100644 index 0000000..a220215 --- /dev/null +++ b/Help/prop_test/FIXTURES_SETUP.rst @@ -0,0 +1,47 @@ +FIXTURES_SETUP +-------------- + +Specifies a list of fixtures for which the test is to be treated as a setup +test. + +Fixture setup tests are ordinary tests with all of the usual test +functionality. Setting the ``FIXTURES_SETUP`` property for a test has two +primary effects: + +- CTest will ensure the test executes before any other test which lists the + fixture(s) in its :prop_test:`FIXTURES_REQUIRED` property. + +- If CTest is asked to run only a subset of tests (e.g. using regular + expressions or the ``--rerun-failed`` option) and the setup test is not in + the set of tests to run, it will automatically be added if any tests in the + set require any fixture listed in ``FIXTURES_SETUP``. + +A setup test can have multiple fixtures listed in its ``FIXTURES_SETUP`` +property. It will execute only once for the whole CTest run, not once for each +fixture. A fixture can also have more than one setup test defined. If there are +multiple setup tests for a fixture, projects can control their order with the +usual :prop_test:`DEPENDS` test property if necessary. + +A setup test is allowed to require other fixtures, but not any fixture listed +in its ``FIXTURES_SETUP`` property. For example: + +.. code-block:: cmake + + # Ok: dependent fixture is different to setup + set_tests_properties(setupFoo PROPERTIES + FIXTURES_SETUP Foo + FIXTURES_REQUIRED Bar + ) + + # Error: cannot require same fixture as setup + set_tests_properties(setupFoo PROPERTIES + FIXTURES_SETUP Foo + FIXTURES_REQUIRED Foo + ) + +If any of a fixture's setup tests fail, none of the tests listing that fixture +in its :prop_test:`FIXTURES_REQUIRED` property will be run. Cleanup tests will, +however, still be executed. + +See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use +test fixtures. diff --git a/Help/prop_test/RESOURCE_LOCK.rst b/Help/prop_test/RESOURCE_LOCK.rst index 8c30f01..755e0aa 100644 --- a/Help/prop_test/RESOURCE_LOCK.rst +++ b/Help/prop_test/RESOURCE_LOCK.rst @@ -5,3 +5,6 @@ Specify a list of resources that are locked by this test. If multiple tests specify the same resource lock, they are guaranteed not to run concurrently. + +See also :prop_test:`FIXTURES_REQUIRED` if the resource requires any setup or +cleanup steps. diff --git a/Help/release/dev/test-fixtures.rst b/Help/release/dev/test-fixtures.rst new file mode 100644 index 0000000..a466c46 --- /dev/null +++ b/Help/release/dev/test-fixtures.rst @@ -0,0 +1,8 @@ +test-fixtures +------------- + +* CTest now supports test fixtures through the new :prop_test:`FIXTURES_SETUP`, + :prop_test:`FIXTURES_CLEANUP` and :prop_test:`FIXTURES_REQUIRED` test + properties. When using regular expressions or ``--rerun-failed`` to limit + the tests to be run, a fixture's setup and cleanup tests will automatically + be added to the execution set if any test requires that fixture. |