CTest: Add support for test fixtures

Add new test properties:

* FIXTURES_SETUP
* FIXTURES_CLEANUP
* FIXTURES_REQUIRED

to specify the roles and dependencies of tests providing/using
test fixtures.

5 files changed, 195 insertions, 1 deletions

diff --git a/Help/prop_test/DEPENDS.rst b/Help/prop_test/DEPENDS.rst
index ee946d9..89c7553 100644
--- a/Help/prop_test/DEPENDS.rst
+++ b/Help/prop_test/DEPENDS.rst
@@ -3,4 +3,8 @@ DEPENDS
 
 Specifies that this test should only be run after the specified list of tests.
 
-Set this to a list of tests that must finish before this test is run.
+Set this to a list of tests that must finish before this test is run. The
+results of those tests are not considered, the dependency relationship is
+purely for order of execution (i.e. it is really just a *run after*
+relationship). Consider using test fixtures with setup tests if a dependency
+with successful completion is required (see :prop_test:`FIXTURES_REQUIRED`).
diff --git a/Help/prop_test/FIXTURES_CLEANUP.rst b/Help/prop_test/FIXTURES_CLEANUP.rst
new file mode 100644
index 0000000..f0a4be0
--- /dev/null
+++ b/Help/prop_test/FIXTURES_CLEANUP.rst
@@ -0,0 +1,46 @@
+FIXTURES_CLEANUP
+----------------
+
+Specifies a list of fixtures for which the test is to be treated as a cleanup
+test.
+
+Fixture cleanup tests are ordinary tests with all of the usual test
+functionality. Setting the ``FIXTURES_CLEANUP`` property for a test has two
+primary effects:
+
+- CTest will ensure the test executes after all other tests which list any of
+  the fixtures in its :prop_test:`FIXTURES_REQUIRED` property.
+
+- If CTest is asked to run only a subset of tests (e.g. using regular
+  expressions or the ``--rerun-failed`` option) and the cleanup test is not in
+  the set of tests to run, it will automatically be added if any tests in the
+  set require any fixture listed in ``FIXTURES_CLEANUP``.
+
+A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP``
+property. It will execute only once for the whole CTest run, not once for each
+fixture. A fixture can also have more than one cleanup test defined. If there
+are multiple cleanup tests for a fixture, projects can control their order with
+the usual :prop_test:`DEPENDS` test property if necessary.
+
+A cleanup test is allowed to require other fixtures, but not any fixture listed
+in its ``FIXTURES_CLEANUP`` property. For example:
+
+.. code-block:: cmake
+
+  # Ok: Dependent fixture is different to cleanup
+  set_tests_properties(cleanupFoo PROPERTIES
+    FIXTURES_CLEANUP  Foo
+    FIXTURES_REQUIRED Bar
+  )
+
+  # Error: cannot require same fixture as cleanup
+  set_tests_properties(cleanupFoo PROPERTIES
+    FIXTURES_CLEANUP  Foo
+    FIXTURES_REQUIRED Foo
+  )
+
+Cleanup tests will execute even if setup or regular tests for that fixture fail
+or are skipped.
+
+See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
+test fixtures.
diff --git a/Help/prop_test/FIXTURES_REQUIRED.rst b/Help/prop_test/FIXTURES_REQUIRED.rst
new file mode 100644
index 0000000..e37dfb5
--- /dev/null
+++ b/Help/prop_test/FIXTURES_REQUIRED.rst
@@ -0,0 +1,94 @@
+FIXTURES_REQUIRED
+-----------------
+
+Specifies a list of fixtures the test requires. Fixture names are case
+sensitive.
+
+Fixtures are a way to attach setup and cleanup tasks to a set of tests. If a
+test requires a given fixture, then all tests marked as setup tasks for that
+fixture will be executed first (once for the whole set of tests, not once per
+test requiring the fixture). After all tests requiring a particular fixture
+have completed, CTest will ensure all tests marked as cleanup tasks for that
+fixture are then executed. Tests are marked as setup tasks with the
+:prop_test:`FIXTURES_SETUP` property and as cleanup tasks with the
+:prop_test:`FIXTURES_CLEANUP` property. If any of a fixture's setup tests fail,
+all tests listing that fixture in their ``FIXTURES_REQUIRED`` property will not
+be executed. The cleanup tests for the fixture will always be executed, even if
+some setup tests fail.
+
+When CTest is asked to execute only a subset of tests (e.g. by the use of
+regular expressions or when run with the ``--rerun-failed`` command line
+option), it will automatically add any setup or cleanup tests for fixtures
+required by any of the tests that are in the execution set.
+
+Since setup and cleanup tasks are also tests, they can have an ordering
+specified by the :prop_test:`DEPENDS` test property just like any other tests.
+This can be exploited to implement setup or cleanup using multiple tests for a
+single fixture to modularise setup or cleanup logic.
+
+The concept of a fixture is different to that of a resource specified by
+:prop_test:`RESOURCE_LOCK`, but they may be used together. A fixture defines a
+set of tests which share setup and cleanup requirements, whereas a resource
+lock has the effect of ensuring a particular set of tests do not run in
+parallel. Some situations may need both, such as setting up a database,
+serialising test access to that database and deleting the database again at the
+end. For such cases, tests would populate both ``FIXTURES_REQUIRED`` and
+:prop_test:`RESOURCE_LOCK` to combine the two behaviours. Names used for
+:prop_test:`RESOURCE_LOCK` have no relationship with names of fixtures, so note
+that a resource lock does not imply a fixture and vice versa.
+
+Consider the following example which represents a database test scenario
+similar to that mentioned above:
+
+.. code-block:: cmake
+
+  add_test(NAME testsDone   COMMAND emailResults)
+  add_test(NAME fooOnly     COMMAND testFoo)
+  add_test(NAME dbOnly      COMMAND testDb)
+  add_test(NAME dbWithFoo   COMMAND testDbWithFoo)
+  add_test(NAME createDB    COMMAND initDB)
+  add_test(NAME setupUsers  COMMAND userCreation)
+  add_test(NAME cleanupDB   COMMAND deleteDB)
+  add_test(NAME cleanupFoo  COMMAND removeFoos)
+
+  set_tests_properties(setupUsers PROPERTIES DEPENDS createDB)
+
+  set_tests_properties(createDB   PROPERTIES FIXTURES_SETUP    DB)
+  set_tests_properties(setupUsers PROPERTIES FIXTURES_SETUP    DB)
+  set_tests_properties(cleanupDB  PROPERTIES FIXTURES_CLEANUP  DB)
+  set_tests_properties(cleanupFoo PROPERTIES FIXTURES_CLEANUP  Foo)
+  set_tests_properties(testsDone  PROPERTIES FIXTURES_CLEANUP  "DB;Foo")
+
+  set_tests_properties(fooOnly    PROPERTIES FIXTURES_REQUIRED Foo)
+  set_tests_properties(dbOnly     PROPERTIES FIXTURES_REQUIRED DB)
+  set_tests_properties(dbWithFoo  PROPERTIES FIXTURES_REQUIRED "DB;Foo")
+
+  set_tests_properties(dbOnly dbWithFoo createDB setupUsers cleanupDB
+                       PROPERTIES RESOURCE_LOCK DbAccess)
+
+Key points from this example:
+
+- Two fixtures are defined: ``DB`` and ``Foo``. Tests can require a single
+  fixture as ``fooOnly`` and ``dbOnly`` do, or they can depend on multiple
+  fixtures like ``dbWithFoo`` does.
+
+- A ``DEPENDS`` relationship is set up to ensure ``setupUsers`` happens after
+  ``createDB``, both of which are setup tests for the ``DB`` fixture and will
+  therefore be executed before the ``dbOnly`` and ``dbWithFoo`` tests
+  automatically.
+
+- No explicit ``DEPENDS`` relationships were needed to make the setup tests run
+  before or the cleanup tests run after the regular tests.
+
+- The ``Foo`` fixture has no setup tests defined, only a single cleanup test.
+
+- ``testsDone`` is a cleanup test for both the ``DB`` and ``Foo`` fixtures.
+  Therefore, it will only execute once regular tests for both fixtures have
+  finished (i.e. after ``fooOnly``, ``dbOnly`` and ``dbWithFoo``). No
+  ``DEPENDS`` relationship was specified for ``testsDone``, so it is free to
+  run before, after or concurrently with other cleanup tests for either
+  fixture.
+
+- The setup and cleanup tests never list the fixtures they are for in their own
+  ``FIXTURES_REQUIRED`` property, as that would result in a dependency on
+  themselves and be considered an error.
diff --git a/Help/prop_test/FIXTURES_SETUP.rst b/Help/prop_test/FIXTURES_SETUP.rst
new file mode 100644
index 0000000..a220215
--- /dev/null
+++ b/Help/prop_test/FIXTURES_SETUP.rst
@@ -0,0 +1,47 @@
+FIXTURES_SETUP
+--------------
+
+Specifies a list of fixtures for which the test is to be treated as a setup
+test.
+
+Fixture setup tests are ordinary tests with all of the usual test
+functionality. Setting the ``FIXTURES_SETUP`` property for a test has two
+primary effects:
+
+- CTest will ensure the test executes before any other test which lists the
+  fixture(s) in its :prop_test:`FIXTURES_REQUIRED` property.
+
+- If CTest is asked to run only a subset of tests (e.g. using regular
+  expressions or the ``--rerun-failed`` option) and the setup test is not in
+  the set of tests to run, it will automatically be added if any tests in the
+  set require any fixture listed in ``FIXTURES_SETUP``.
+
+A setup test can have multiple fixtures listed in its ``FIXTURES_SETUP``
+property. It will execute only once for the whole CTest run, not once for each
+fixture. A fixture can also have more than one setup test defined. If there are
+multiple setup tests for a fixture, projects can control their order with the
+usual :prop_test:`DEPENDS` test property if necessary.
+
+A setup test is allowed to require other fixtures, but not any fixture listed
+in its ``FIXTURES_SETUP`` property. For example:
+
+.. code-block:: cmake
+
+  # Ok: dependent fixture is different to setup
+  set_tests_properties(setupFoo PROPERTIES
+    FIXTURES_SETUP    Foo
+    FIXTURES_REQUIRED Bar
+  )
+
+  # Error: cannot require same fixture as setup
+  set_tests_properties(setupFoo PROPERTIES
+    FIXTURES_SETUP    Foo
+    FIXTURES_REQUIRED Foo
+  )
+
+If any of a fixture's setup tests fail, none of the tests listing that fixture
+in its :prop_test:`FIXTURES_REQUIRED` property will be run. Cleanup tests will,
+however, still be executed.
+
+See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
+test fixtures.
diff --git a/Help/prop_test/RESOURCE_LOCK.rst b/Help/prop_test/RESOURCE_LOCK.rst
index 8c30f01..755e0aa 100644
--- a/Help/prop_test/RESOURCE_LOCK.rst
+++ b/Help/prop_test/RESOURCE_LOCK.rst
@@ -5,3 +5,6 @@ Specify a list of resources that are locked by this test.
 
 If multiple tests specify the same resource lock, they are guaranteed
 not to run concurrently.
+
+See also :prop_test:`FIXTURES_REQUIRED` if the resource requires any setup or
+cleanup steps.