path: root/Tests/RunCMake/CPack/README.txt

RunCMake.CPack is a test module that is intended for testing of package
generators that can be validated from command line.

-------------
Adding a test
-------------

CPack test root directory: 'Tests/RunCMake/CPack/tests'.

All phases are executed separately for each generator that is bound to a test.
Tests for each generator are subtests of test 'RunCMake.CPack_<generator_name>'.

Each test must also be added to 'RunCMakeTest.cmake' script located in CPack
test root directory.

Line that adds a test is:
run_cpack_test(<test_name> "<generator_name_list>" <compile_stage>
               "<packaging_type_list>")

<generator_name_list> may be one generator e.g. "RPM" or multiple e.g. "RPM;DEB"
and will be run for all listed generators. In test and verification scripts
current generator can be accessed through GENERATOR_TYPE variable.

<compile_stage> is a boolean variable that enables or disables compile stage -
most tests don't require compilation as a non binary file can be used for
package content but sometimes an executable or a library has to be created
before the packaging stage.

<packaging_type_list> can be a list of one or more packaging types: MONOLITHIC,
COMPONENT or GROUP - each type sets per generator default variables which can
be overwritten in the test if needed
(see <generator_type>/packaging_<packaging_type>_default.cmake for the variables
that are set by default for each packaging type).
Alternatively CUSTOM value can be set which means that default values will not
be set and that values will be set manually in the test itself.

Alternatively a test with subtests can be added:
run_cpack_test_subtests(<test_name> "<subtests_list>" "<generator_name_list>"
                        <compile_stage> "<packaging_type_list>")

<subtests_list> is the only new parameter and it is a list of names that will
be used for subtests. In test and verification scripts subtest name can be
accessed through RunCMake_SUBTEST_SUFFIX variable.

Also source package tests can be added:
run_cpack_source_test(<test_name> "<generator_name_list>" true)

Test consists of
- test prerequirements phase (optional)
- CMake execution phase
- build phase (optional and not available for source package tests)
- CPack execution phase
- verification of generated files

The phases are executed once per specified generator, packaging type and subtest
combinatiion.

test prerequirements phase (optional):
--------------------------------------

In some cases individual tests for certain generator need additional
prerequirements met.

In such cases '<test_name>/<generator_name>-Prerequirements.cmake' file
containing 'function(get_test_prerequirements found_var config_file)' should be
created. Function should return true in found_var if all prerequirements are
met. config_file variable contains the location of config file generated by the
generator so that this function can check if prerequired variable is set in the
file.
NOTE: All required programs should be searched for in generator prerequirements
function and only checked for the variable in configure file in per test
function.

If prerequirements are not met test will be skipped outputting
'<test_name> - SKIPPED' string. Note that this doesn't fail the entire test
group.

TODO: skipped tests should provide expected error string so test should fail
      if error string is not found in the output of run test (this would add
      'EXPECTED FAIL' string on success and 'ERROR' on failure).

CMake execution phase:
----------------------

To add a new CPack test we first create a '<test_name>/test.cmake' script that
contains CMake commands that should be used as a preparation script for
generation of different types of packages. This script is placed into CPack
test root directory.

If test will be used for multiple generators but some of them require some
generator specific commands then those commands should be added to 'test.cmake'
script wrapped with 'if(GENERATOR_TYPE STREQUAL <name_of_the_generator>)'.

NOTE: In some cases (for example when testing CPackComponent.cmake functions)
the test has to run some functions after CPack.cmake is included. In such cases
a function run_after_include_cpack can be declared in test.cmake file and that
function will run after the inclusion of CPack.cmake.

NOTE: During CMake configure stage developer warnings may be expected. In such
cases an expected output regular expression can be provided by creating
'<test_name>/configure-stdout.txt' and/or '<test_name>/configure-stderr.txt'
file. There are also more specialized versions of the file available:
- configure-${PACKAGING_TYPE}-${SUBTEST_SUFFIX}-std${o}.txt
- configure-${SUBTEST_SUFFIX}-std${o}.txt
- configure-${PACKAGING_TYPE}-std${o}.txt

build phase (optional and not available for source package tests)
-----------------------------------------------------------------

This phase only runs make command.

NOTE: By default all tests have CMAKE_BUILD_TYPE variable set to Debug.

CPack execution phase:
----------------------

Only executes CPack for content that was generated during CMake execution
phase.

NOTE: By default CPACK_PACKAGE_NAME variable is set to lower case test name.

Verification of generated files:
--------------------------------

Verification of generated files consists of two phases
- mandatory verification phase
- optional verification phase

Mandatory verification phase checks that expected files were generated and
contain expected files.
Mandatory verification phase also checks that no other unexpected package files
were generated (this is executed even if EXPECTED_FILES_COUNT contains 0 in
order to verify that no files were generated).
CMake script '<test_name>/ExpectedFiles.cmake' is required by
this step and must contain
- EXPECTED_FILES_COUNT variable that contains the number of expected files that
  will be generated (0 or more)

- EXPECTED_FILE_<file_number_starting_with_1> that contains globing expression
  that uniquely defines expected file name (will be used to find expected file)
  and should be present once for each expected file.
  NOTE: This variable should be used only as last resort as it sets generator
        specific globbing expression. Each generator can recreate file name from
        parts that are already populated by default but can always be
        overwritten if needed:
  - EXPECTED_FILE_<file_number_starting_with_1>_NAME is the name component of
    the file (by default it is set to package name that is same as test name
    in lowercase)
  - EXPECTED_FILE_<file_number_starting_with_1>_VERSION is the version of the
    package (by default it is set to '0.1.1')
  - EXPECTED_FILE_<file_number_starting_with_1>_REVISION is the revision of the
    package (by default it is set to '1')

- EXPECTED_FILE_CONTENT_<file_number_starting_with_1> that contains regular
  expression of files that should be present in generated file and should be
  present once for each expected file
  NOTE: This variable should be used only as last resort as it sets generator
        specific regular expression.
        EXPECTED_FILE_CONTENT_<file_number_starting_with_1>_LIST should be
        preferred as it requires a list of expected files and directories that
        is later changed automatically depending on the generator so expected
        package content can be written only once per test for all generators.

- EXPECTED_FILE_PACKAGING_PREFIX and
  EXPECTED_FILE_<file_number_starting_with_1>_PACKAGING_PREFIX variables can be
  set to explicitly specified CPACK_PACKAGING_PREFIX value. By default this
  variable does not need to be set as it is implicitly set to package generator
  specific prefix.

Optional verification phase is generator specific and is optionaly executed.
This phase is executed if '<test_name>/VerifyResult.cmake' script exists.

VerifyResult.cmake script also automatically prints out standard output and
standard error from CPack execution phase that is compared with
'<test_name>/<generator_name>-stdout.txt' regular expression and
'<test_name>/<generator_name>-stderr.txt' regular expresson respectively.
NOTE: For subtests generator name can also be suffixed with subtest name and/or
      packaging type (MONOLITHIC, COMPONENT, GROUP) and in such cases the
      preferences of which file will be used are as follows:
        - generator name + packaging type + subtest name
        - generator name + packaging type
        - generator name + subtest name
        - generator name
        - subtest name
        - default generator
      File name format: '<generator_name>-<packaging_type>-<subtest_name>-std<type>.txt'
                        where <type> can either be 'out' or 'err'.
      File name format: '<generator_name>-<packaging_type>-std<type>.txt'
                        where <type> can either be 'out' or 'err'.
      File name format: '<generator_name>-<subtest_name>-std<type>.txt' where
                        <type> can either be 'out' or 'err'.
NOTE: If none of the comparison files are present then the default generator
      file is used if present.

----------------------
Adding a new generator
----------------------

To add a new generator we must
- add new generator directory (e.g. RPM for RPM generator) to CPack test root
  directory that contains 'Helpers.cmake' script.
  - In this script some functions must exist:
    - getPackageContent: This function should list files that are contained in
                         a package.
      Function parameters:
      + FILE variable that will contain path to file for which the content
        should be listed
      + RESULT_VAR that will tell the function which variable in parent scope
        should contain the result (list of files inside package file)
    - getPackageNameGlobexpr: This function should generate package name
                              globbing expression.
      Function parameters:
      + NAME that will contain the expected package name
      + COMPONENT that will contain the expected package component
      + VERSION that will contain the expected package version
      + REVISION that will contain the expected package revision number
      + FILE_NO that will contain the file number for which the globbing
        expression is generated
      + RESULT_VAR that will tell the function which variable in parent scope
        should contain the result (file name globbing expression)
    - getPackageContentList: This function should return a list of files and
                             directories that are in the provided package.
      Function parameters:
      + FILE that will contain the package file for which the package content
        should be returned.
      + RESULT_VAR that will tell the function which variable in parent scope
        should contain the result (list of package content)
    - toExpectedContentList: This function should convert an expected package
                             content list into one that is expected for the
                             generator (e.g. rpm packages have install/relocate
                             path prefixes which aren't part of the package so
                             those paths have to be removed from the expected
                             content list).
      Function parameters:
      + FILE_NO that will contain the file number for which the conversion
        should be performed
      + CONTENT_VAR that will contain the input list and is also the variable
        in parent scope which should contain the result (converted content list)
- add 'Prerequirements.cmake' script to generator directory. In this script a
  function named 'get_test_prerequirements' must exist. This function should
  set a variable in parent scope (name of the variable is the first parameter)
  that tells if prerequirements for test execution are met (certain programs,
  OS specifics, ...) and create a config file (name of the variable which
  contains file name and path is provided with the second parameter) that
  should contain 'set' commands for variables that will later be used in tests
  (e.g. location of dpkg program for DEB packages)
- add tests the same way as described above
- add generator to 'add_RunCMake_test_group' function call that is located in
  RunCMake CMakeLists.txt file
- if needed add 'packaging_<packaging_type>_default.cmake' scripts that define
  default variables that will be set for each packaging type (MONOLITHIC,
  COMPONENT and GROUP)
- if needed add 'default_expected_std<type>.txt' files where <type> is either
  'out' or 'err' which will contain default expected output of package
  generation regular expression.
- add generator to list of other CPack generators in RunCMake/CMakeLists.txt