summaryrefslogtreecommitdiffstats
path: root/Help/prop_test
diff options
context:
space:
mode:
authorCraig Scott <craig.scott@crascit.com>2016-09-07 04:04:07 (GMT)
committerBrad King <brad.king@kitware.com>2016-09-20 18:37:38 (GMT)
commit73f47c9e46d42513cd7eeb08e27c4d1a424949ad (patch)
treef7f275e022e3c6debf4450015d1ae40e46fb56a3 /Help/prop_test
parent6b8812c27e6df1d10fa4bfc30cb3eadd08d7966b (diff)
downloadCMake-73f47c9e46d42513cd7eeb08e27c4d1a424949ad.zip
CMake-73f47c9e46d42513cd7eeb08e27c4d1a424949ad.tar.gz
CMake-73f47c9e46d42513cd7eeb08e27c4d1a424949ad.tar.bz2
CTest: Add support for test fixtures
Add new test properties: * FIXTURES_SETUP * FIXTURES_CLEANUP * FIXTURES_REQUIRED to specify the roles and dependencies of tests providing/using test fixtures.
Diffstat (limited to 'Help/prop_test')
-rw-r--r--Help/prop_test/DEPENDS.rst6
-rw-r--r--Help/prop_test/FIXTURES_CLEANUP.rst46
-rw-r--r--Help/prop_test/FIXTURES_REQUIRED.rst94
-rw-r--r--Help/prop_test/FIXTURES_SETUP.rst47
-rw-r--r--Help/prop_test/RESOURCE_LOCK.rst3
5 files changed, 195 insertions, 1 deletions
diff --git a/Help/prop_test/DEPENDS.rst b/Help/prop_test/DEPENDS.rst
index ee946d9..89c7553 100644
--- a/Help/prop_test/DEPENDS.rst
+++ b/Help/prop_test/DEPENDS.rst
@@ -3,4 +3,8 @@ DEPENDS
Specifies that this test should only be run after the specified list of tests.
-Set this to a list of tests that must finish before this test is run.
+Set this to a list of tests that must finish before this test is run. The
+results of those tests are not considered, the dependency relationship is
+purely for order of execution (i.e. it is really just a *run after*
+relationship). Consider using test fixtures with setup tests if a dependency
+with successful completion is required (see :prop_test:`FIXTURES_REQUIRED`).
diff --git a/Help/prop_test/FIXTURES_CLEANUP.rst b/Help/prop_test/FIXTURES_CLEANUP.rst
new file mode 100644
index 0000000..f0a4be0
--- /dev/null
+++ b/Help/prop_test/FIXTURES_CLEANUP.rst
@@ -0,0 +1,46 @@
+FIXTURES_CLEANUP
+----------------
+
+Specifies a list of fixtures for which the test is to be treated as a cleanup
+test.
+
+Fixture cleanup tests are ordinary tests with all of the usual test
+functionality. Setting the ``FIXTURES_CLEANUP`` property for a test has two
+primary effects:
+
+- CTest will ensure the test executes after all other tests which list any of
+ the fixtures in its :prop_test:`FIXTURES_REQUIRED` property.
+
+- If CTest is asked to run only a subset of tests (e.g. using regular
+ expressions or the ``--rerun-failed`` option) and the cleanup test is not in
+ the set of tests to run, it will automatically be added if any tests in the
+ set require any fixture listed in ``FIXTURES_CLEANUP``.
+
+A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP``
+property. It will execute only once for the whole CTest run, not once for each
+fixture. A fixture can also have more than one cleanup test defined. If there
+are multiple cleanup tests for a fixture, projects can control their order with
+the usual :prop_test:`DEPENDS` test property if necessary.
+
+A cleanup test is allowed to require other fixtures, but not any fixture listed
+in its ``FIXTURES_CLEANUP`` property. For example:
+
+.. code-block:: cmake
+
+ # Ok: Dependent fixture is different to cleanup
+ set_tests_properties(cleanupFoo PROPERTIES
+ FIXTURES_CLEANUP Foo
+ FIXTURES_REQUIRED Bar
+ )
+
+ # Error: cannot require same fixture as cleanup
+ set_tests_properties(cleanupFoo PROPERTIES
+ FIXTURES_CLEANUP Foo
+ FIXTURES_REQUIRED Foo
+ )
+
+Cleanup tests will execute even if setup or regular tests for that fixture fail
+or are skipped.
+
+See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
+test fixtures.
diff --git a/Help/prop_test/FIXTURES_REQUIRED.rst b/Help/prop_test/FIXTURES_REQUIRED.rst
new file mode 100644
index 0000000..e37dfb5
--- /dev/null
+++ b/Help/prop_test/FIXTURES_REQUIRED.rst
@@ -0,0 +1,94 @@
+FIXTURES_REQUIRED
+-----------------
+
+Specifies a list of fixtures the test requires. Fixture names are case
+sensitive.
+
+Fixtures are a way to attach setup and cleanup tasks to a set of tests. If a
+test requires a given fixture, then all tests marked as setup tasks for that
+fixture will be executed first (once for the whole set of tests, not once per
+test requiring the fixture). After all tests requiring a particular fixture
+have completed, CTest will ensure all tests marked as cleanup tasks for that
+fixture are then executed. Tests are marked as setup tasks with the
+:prop_test:`FIXTURES_SETUP` property and as cleanup tasks with the
+:prop_test:`FIXTURES_CLEANUP` property. If any of a fixture's setup tests fail,
+all tests listing that fixture in their ``FIXTURES_REQUIRED`` property will not
+be executed. The cleanup tests for the fixture will always be executed, even if
+some setup tests fail.
+
+When CTest is asked to execute only a subset of tests (e.g. by the use of
+regular expressions or when run with the ``--rerun-failed`` command line
+option), it will automatically add any setup or cleanup tests for fixtures
+required by any of the tests that are in the execution set.
+
+Since setup and cleanup tasks are also tests, they can have an ordering
+specified by the :prop_test:`DEPENDS` test property just like any other tests.
+This can be exploited to implement setup or cleanup using multiple tests for a
+single fixture to modularise setup or cleanup logic.
+
+The concept of a fixture is different to that of a resource specified by
+:prop_test:`RESOURCE_LOCK`, but they may be used together. A fixture defines a
+set of tests which share setup and cleanup requirements, whereas a resource
+lock has the effect of ensuring a particular set of tests do not run in
+parallel. Some situations may need both, such as setting up a database,
+serialising test access to that database and deleting the database again at the
+end. For such cases, tests would populate both ``FIXTURES_REQUIRED`` and
+:prop_test:`RESOURCE_LOCK` to combine the two behaviours. Names used for
+:prop_test:`RESOURCE_LOCK` have no relationship with names of fixtures, so note
+that a resource lock does not imply a fixture and vice versa.
+
+Consider the following example which represents a database test scenario
+similar to that mentioned above:
+
+.. code-block:: cmake
+
+ add_test(NAME testsDone COMMAND emailResults)
+ add_test(NAME fooOnly COMMAND testFoo)
+ add_test(NAME dbOnly COMMAND testDb)
+ add_test(NAME dbWithFoo COMMAND testDbWithFoo)
+ add_test(NAME createDB COMMAND initDB)
+ add_test(NAME setupUsers COMMAND userCreation)
+ add_test(NAME cleanupDB COMMAND deleteDB)
+ add_test(NAME cleanupFoo COMMAND removeFoos)
+
+ set_tests_properties(setupUsers PROPERTIES DEPENDS createDB)
+
+ set_tests_properties(createDB PROPERTIES FIXTURES_SETUP DB)
+ set_tests_properties(setupUsers PROPERTIES FIXTURES_SETUP DB)
+ set_tests_properties(cleanupDB PROPERTIES FIXTURES_CLEANUP DB)
+ set_tests_properties(cleanupFoo PROPERTIES FIXTURES_CLEANUP Foo)
+ set_tests_properties(testsDone PROPERTIES FIXTURES_CLEANUP "DB;Foo")
+
+ set_tests_properties(fooOnly PROPERTIES FIXTURES_REQUIRED Foo)
+ set_tests_properties(dbOnly PROPERTIES FIXTURES_REQUIRED DB)
+ set_tests_properties(dbWithFoo PROPERTIES FIXTURES_REQUIRED "DB;Foo")
+
+ set_tests_properties(dbOnly dbWithFoo createDB setupUsers cleanupDB
+ PROPERTIES RESOURCE_LOCK DbAccess)
+
+Key points from this example:
+
+- Two fixtures are defined: ``DB`` and ``Foo``. Tests can require a single
+ fixture as ``fooOnly`` and ``dbOnly`` do, or they can depend on multiple
+ fixtures like ``dbWithFoo`` does.
+
+- A ``DEPENDS`` relationship is set up to ensure ``setupUsers`` happens after
+ ``createDB``, both of which are setup tests for the ``DB`` fixture and will
+ therefore be executed before the ``dbOnly`` and ``dbWithFoo`` tests
+ automatically.
+
+- No explicit ``DEPENDS`` relationships were needed to make the setup tests run
+ before or the cleanup tests run after the regular tests.
+
+- The ``Foo`` fixture has no setup tests defined, only a single cleanup test.
+
+- ``testsDone`` is a cleanup test for both the ``DB`` and ``Foo`` fixtures.
+ Therefore, it will only execute once regular tests for both fixtures have
+ finished (i.e. after ``fooOnly``, ``dbOnly`` and ``dbWithFoo``). No
+ ``DEPENDS`` relationship was specified for ``testsDone``, so it is free to
+ run before, after or concurrently with other cleanup tests for either
+ fixture.
+
+- The setup and cleanup tests never list the fixtures they are for in their own
+ ``FIXTURES_REQUIRED`` property, as that would result in a dependency on
+ themselves and be considered an error.
diff --git a/Help/prop_test/FIXTURES_SETUP.rst b/Help/prop_test/FIXTURES_SETUP.rst
new file mode 100644
index 0000000..a220215
--- /dev/null
+++ b/Help/prop_test/FIXTURES_SETUP.rst
@@ -0,0 +1,47 @@
+FIXTURES_SETUP
+--------------
+
+Specifies a list of fixtures for which the test is to be treated as a setup
+test.
+
+Fixture setup tests are ordinary tests with all of the usual test
+functionality. Setting the ``FIXTURES_SETUP`` property for a test has two
+primary effects:
+
+- CTest will ensure the test executes before any other test which lists the
+ fixture(s) in its :prop_test:`FIXTURES_REQUIRED` property.
+
+- If CTest is asked to run only a subset of tests (e.g. using regular
+ expressions or the ``--rerun-failed`` option) and the setup test is not in
+ the set of tests to run, it will automatically be added if any tests in the
+ set require any fixture listed in ``FIXTURES_SETUP``.
+
+A setup test can have multiple fixtures listed in its ``FIXTURES_SETUP``
+property. It will execute only once for the whole CTest run, not once for each
+fixture. A fixture can also have more than one setup test defined. If there are
+multiple setup tests for a fixture, projects can control their order with the
+usual :prop_test:`DEPENDS` test property if necessary.
+
+A setup test is allowed to require other fixtures, but not any fixture listed
+in its ``FIXTURES_SETUP`` property. For example:
+
+.. code-block:: cmake
+
+ # Ok: dependent fixture is different to setup
+ set_tests_properties(setupFoo PROPERTIES
+ FIXTURES_SETUP Foo
+ FIXTURES_REQUIRED Bar
+ )
+
+ # Error: cannot require same fixture as setup
+ set_tests_properties(setupFoo PROPERTIES
+ FIXTURES_SETUP Foo
+ FIXTURES_REQUIRED Foo
+ )
+
+If any of a fixture's setup tests fail, none of the tests listing that fixture
+in its :prop_test:`FIXTURES_REQUIRED` property will be run. Cleanup tests will,
+however, still be executed.
+
+See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
+test fixtures.
diff --git a/Help/prop_test/RESOURCE_LOCK.rst b/Help/prop_test/RESOURCE_LOCK.rst
index 8c30f01..755e0aa 100644
--- a/Help/prop_test/RESOURCE_LOCK.rst
+++ b/Help/prop_test/RESOURCE_LOCK.rst
@@ -5,3 +5,6 @@ Specify a list of resources that are locked by this test.
If multiple tests specify the same resource lock, they are guaranteed
not to run concurrently.
+
+See also :prop_test:`FIXTURES_REQUIRED` if the resource requires any setup or
+cleanup steps.