Tests: CPack test documentation facelift

1 files changed, 136 insertions, 32 deletions

diff --git a/Tests/RunCMake/CPack/README.txt b/Tests/RunCMake/CPack/README.txt
index 5c86880..f51d254 100644
--- a/Tests/RunCMake/CPack/README.txt
+++ b/Tests/RunCMake/CPack/README.txt
@@ -1,41 +1,75 @@
 RunCMake.CPack is a test module that is intended for testing of package
 generators that can be validated from command line.
 
-TODO: all tests should cover all packaging types (single package, grouped and
-      component packaging)
-
 -------------
 Adding a test
 -------------
 
-CPack test root directory: 'Tests/RunCMake/CPack'.
+CPack test root directory: 'Tests/RunCMake/CPack/tests'.
 
 All phases are executed separately for each generator that is bound to a test.
 Tests for each generator are subtests of test 'RunCMake.CPack_<generator_name>'.
 
 Each test must also be added to 'RunCMakeTest.cmake' script located in CPack
 test root directory.
+
 Line that adds a test is:
-run_cpack_test(<test_name> "<generator_name>")
+run_cpack_test(<test_name> "<generator_name_list>" <compile_stage>
+               "<packaging_type_list>")
+
+<generator_name_list> may be one generator e.g. "RPM" or multiple e.g. "RPM;DEB"
+and will be run for all listed generators. In test and verification scripts
+current generator can be accessed through GENERATOR_TYPE variable.
+
+<compile_stage> is a boolean variable that enables or disables compile stage -
+most tests don't require compilation as a non binary file can be used for
+package content but sometimes an executable or a library has to be created
+before the packaging stage.
 
-<generator_name> may be one generator e.g. "RPM" or multiple e.g. "RPM;DEB" and
-will be run for all listed generators.
+<packaging_type_list> can be a list of one or more packaging types: MONOLITHIC,
+COMPONENT or GROUP - each type sets per generator default variables which can
+be overwritten in the test if needed
+(see <generator_type>/packaging_<packaging_type>_default.cmake for the variables
+that are set by default for each packaging type).
+Alternatively CUSTOM value can be set which means that default values will not
+be set and that values will be set manually in the test itself.
+
+Alternatively a test with subtests can be added:
+run_cpack_test_subtests(<test_name> "<subtests_list>" "<generator_name_list>"
+                        <compile_stage> "<packaging_type_list>")
+
+<subtests_list> is the only new parameter and it is a list of names that will
+be used for subtests. In test and verification scripts subtest name can be
+accessed through RunCMake_SUBTEST_SUFFIX variable.
+
+Also source package tests can be added:
+run_cpack_source_test(<test_name> "<generator_name_list>" true)
 
 Test consists of
 - test prerequirements phase (optional)
 - CMake execution phase
+- build phase (optional and not available for source package tests)
 - CPack execution phase
 - verification of generated files
 
+The phases are executed once per specified generator, packaging type and subtest
+combinatiion.
+
 test prerequirements phase (optional):
 --------------------------------------
 
 In some cases individual tests for certain generator need additional
 prerequirements met.
 
-In such cases '<generator_name>/<test_name>-Prerequirements.cmake' file
-containing 'function(get_test_prerequirements found_var)' should be created.
-Function should return true if all prerequirements are met.
+In such cases '<test_name>/<generator_name>-Prerequirements.cmake' file
+containing 'function(get_test_prerequirements found_var config_file)' should be
+created. Function should return true in found_var if all prerequirements are
+met. config_file variable contains the location of config file generated by the
+generator so that this function can check if prerequired variable is set in the
+file.
+NOTE: All required programs should be searched for in generator prerequirements
+function and only checked for the variable in configure file in per test
+function.
 
 If prerequirements are not met test will be skipped outputting
 '<test_name> - SKIPPED' string. Note that this doesn't fail the entire test
@@ -48,15 +82,21 @@ TODO: skipped tests should provide expected error string so test should fail
 CMake execution phase:
 ----------------------
 
-To add a new CPack test we first create a <test_name>.cmake script that contains
-CMake commands that should be used as a preparation script for generation of
-different types of packages. This script is placed into CPack test root
-directory even if it will be used for only one of the generators.
+To add a new CPack test we first create a '<test_name>/test.cmake' script that
+contains CMake commands that should be used as a preparation script for
+generation of different types of packages. This script is placed into CPack
+test root directory.
 
 If test will be used for multiple generators but some of them require some
-generator specific commands then those commands should be added to a separate
-file that should be located in '<generator_name>/<test_name>-specifics.cmake'
-in CPack test root directory.
+generator specific commands then those commands should be added to 'test.cmake'
+script wrapped with 'if(GENERATOR_TYPE STREQUAL <name_of_the_generator>)'.
+
+build phase (optional and not available for source package tests)
+-----------------------------------------------------------------
+
+This phase only runs make command.
+
+NOTE: By default all tests have CMAKE_BUILD_TYPE variable set to Debug.
 
 CPack execution phase:
 ----------------------
@@ -64,6 +104,8 @@ CPack execution phase:
 Only executes CPack for content that was generated during CMake execution
 phase.
 
+NOTE: By default CPACK_PACKAGE_NAME variable is set to lower case test name.
+
 Verification of generated files:
 --------------------------------
 
@@ -76,24 +118,49 @@ contain expected files.
 Mandatory verification phase also checks that no other unexpected package files
 were generated (this is executed even if EXPECTED_FILES_COUNT contains 0 in
 order to verify that no files were generated).
-CMake script '<generator_name>/<test_name>-ExpectedFiles.cmake' is required by
+CMake script '<test_name>/ExpectedFiles.cmake' is required by
 this step and must contain
 - EXPECTED_FILES_COUNT variable that contains the number of expected files that
   will be generated (0 or more)
+
 - EXPECTED_FILE_<file_number_starting_with_1> that contains globing expression
   that uniquely defines expected file name (will be used to find expected file)
-  and should be present once for each expected file
+  and should be present once for each expected file.
+  NOTE: This variable should be used only as last resort as it sets generator
+        specific globbing expression. Each generator can recreate file name from
+        parts that are already populated by default but can always be
+        overwritten if needed:
+  - EXPECTED_FILE_<file_number_starting_with_1>_NAME is the name component of
+    the file (by default it is set to package name that is same as test name
+    in lowercase)
+  - EXPECTED_FILE_<file_number_starting_with_1>_VERSION is the version of the
+    package (by default it is set to '0.1.1')
+  - EXPECTED_FILE_<file_number_starting_with_1>_REVISION is the revision of the
+    package (by default it is set to '1')
+
 - EXPECTED_FILE_CONTENT_<file_number_starting_with_1> that contains regular
   expression of files that should be present in generated file and should be
   present once for each expected file
+  NOTE: This variable should be used only as last resort as it sets generator
+        specific regular expression.
+        EXPECTED_FILE_CONTENT_<file_number_starting_with_1>_LIST should be
+        prefered as it requires a list of expected files and directories that
+        is later changed automatically depending on the generator so expected
+        package content can be written only once per test for all generators.
 
 Optional verification phase is generator specific and is optionaly executed.
-This phase is executed if '<generator_name>/<test_name>-VerifyResult.cmake'
-script exists.
-In case that the script doesn't exist VerifyResult.cmake script automatically
-prints out standard output and standard error from CPack execution phase that
-is compared with '<generator_name>/<test_name>-stdout.txt' regular expression
-and '<generator_name>/<test_name>-stderr.txt' regular expresson respectively.
+This phase is executed if '<test_name>/VerifyResult.cmake' script exists.
+
+VerifyResult.cmake script also automatically prints out standard output and
+standard error from CPack execution phase that is compared with
+'<test_name>/<generator_name>-stdout.txt' regular expression and
+and '<test_name>/<generator_name>-stderr.txt' regular expresson respectively.
+NOTE: For subtests generator name can also be suffixed with subtest name and in
+      such case subtest file is preferred.
+      File name format: '<generator_name>-<subtest_name>-std<type>.txt' where
+                        <type> can either be 'out' or 'err'.
+NOTE: If none of the comparison files are present then the default generator
+      file is used if present.
 
 ----------------------
 Adding a new generator
@@ -101,13 +168,44 @@ Adding a new generator
 
 To add a new generator we must
 - add new generator directory (e.g. RPM for RPM generator) to CPack test root
-  directory that contains 'Helpers.cmake' script. In this script a function
-  named 'getPackageContent' must exist. This function should list files that
-  are contained in a package. Function should accept two parameters
-  + FILE variable that will contain path to file for which the content should be
-    listed
-  + RESULT_VAR that will tell the function which variable in parent scope should
-    contain the result (list of files inside package file)
+  directory that contains 'Helpers.cmake' script.
+  - In this script some functions must exist:
+    - getPackageContent: This function should list files that are contained in
+                         a package.
+      Function parameters:
+      + FILE variable that will contain path to file for which the content
+        should be listed
+      + RESULT_VAR that will tell the function which variable in parent scope
+        should contain the result (list of files inside package file)
+    - getPackageNameGlobexpr: This function should generate package name
+                              globbing expression.
+      Function parameters:
+      + NAME that will contain the expected package name
+      + COMPONENT that will contain the expected package component
+      + VERSION that will contain the expected package version
+      + REVISION that will contain the expected package revision number
+      + FILE_NO that will contain the file number for which the globbing
+        expression is generated
+      + RESULT_VAR that will tell the function which variable in parent scope
+        should contain the result (file name globbing expression)
+    - getPackageContentList: This function should return a list of files and
+                             directories that are in the provided package.
+      Function parameters:
+      + FILE that will contain the package file for which the package content
+        should be returned.
+      + RESULT_VAR that will tell the function which variable in parent scope
+        should contain the result (list of pacakge content)
+    - toExpectedContentList: This function should convert an expected package
+                             content list into one that is expected for the
+                             generator (e.g. rpm packages have install/relocate
+                             path prefixes which aren't part of the package so
+                             those paths have to be removed from the expected
+                             content list).
+      Function parameters:
+      + FILE_NO that will contain the file number for which the conversion
+        should be performed
+      + CONTENT_VAR that will contain the input list and is also the variable
+        in parent scope which should contain the result (converted content list)
 - add 'Prerequirements.cmake' script to generator directory. In this script a
   function named 'get_test_prerequirements' must exist. This function should
   set a variable in parent scope (name of the variable is the first parameter)
@@ -119,3 +217,9 @@ To add a new generator we must
 - add tests the same way as described above
 - add generator to 'add_RunCMake_test_group' function call that is located in
   RunCMake CMakeLists.txt file
+- if needed add 'packaging_<packaging_type>_default.cmake' scripts that define
+  default variables that will be set for each packaging type (MONOLITHIC,
+  COMPONENT and GROUP)
+- if needed add 'default_expected_std<type>.txt' files where <type> is either
+  'out' or 'err' which will contain default expected output of package
+  generation regular expression.