summaryrefslogtreecommitdiffstats
path: root/doc/cmake-vols-fetchcontent.md
diff options
context:
space:
mode:
authorjhendersonHDF <jhenderson@hdfgroup.org>2023-05-26 20:29:02 (GMT)
committerGitHub <noreply@github.com>2023-05-26 20:29:02 (GMT)
commit79bb60c3f6f67411e5d70b84743fc9f6b6143cbc (patch)
treedce0c97c567cb20eeb65de917e21347376979f18 /doc/cmake-vols-fetchcontent.md
parent77e64e0df46bb7498be014b1005efd49c4ad7518 (diff)
downloadhdf5-79bb60c3f6f67411e5d70b84743fc9f6b6143cbc.zip
hdf5-79bb60c3f6f67411e5d70b84743fc9f6b6143cbc.tar.gz
hdf5-79bb60c3f6f67411e5d70b84743fc9f6b6143cbc.tar.bz2
API test updates (#3018)
* Remove macros from api tests (#2929) * Remove macros and undefined callbacks (#2959) * Remove remaining macros from H5_api_tests_disabled.h (#2968) * Put some vol capability checks in testpar tests and remove remaining warnings (#2995) * API tests datatype generation cleanup * Clean up API tests' random datatype generation and fix bug with enum datatype generation * Init parallel API tests with MPI_THREAD_MULTIPLE * HDF5 API tests - Check VOL connector registration * Determine whether a VOL connector failed to load before running API tests * Cleanup some usages of H5VL_CAP_FLAG_CREATION_ORDER in API tests * Remove some now-unused macros from H5_api_tests_disabled.h * Enable HDF5 API tests by default * Implement CMake option to install HDF5 API tests * Check for invalid AAPL from H5Acreate * Enable building of VOL connectors alongside HDF5 in CMake * Prepend CMake VOL URL option indices with 0s so they come in order * Don't turn on API tests by default yet * Document VOL connector FetchContent functionality * Add release note for API test updates * Only install testing library if API tests are installed * Fix grammar
Diffstat (limited to 'doc/cmake-vols-fetchcontent.md')
-rw-r--r--doc/cmake-vols-fetchcontent.md195
1 files changed, 195 insertions, 0 deletions
diff --git a/doc/cmake-vols-fetchcontent.md b/doc/cmake-vols-fetchcontent.md
new file mode 100644
index 0000000..5f7b3d4
--- /dev/null
+++ b/doc/cmake-vols-fetchcontent.md
@@ -0,0 +1,195 @@
+# Building and testing HDF5 VOL connectors with CMake FetchContent
+
+This document details the process of using CMake options to build and test
+an HDF5 VOL connector alongside the HDF5 library when building HDF5 from
+source. There are several benefits that this may provide, but among them
+are the following:
+
+ * A VOL connector built this way can be tested at the same time that
+ HDF5 is, which eliminates the need to have a multi-step build process
+ where one builds HDF5, uses it to build the VOL connector and then
+ uses the external [HDF5 VOL tests](https://github.com/hdfGroup/vol-tests)
+ repository to test their connector.
+ * Building VOL connectors in this manner will usually install the built
+ connector library alongside the HDF5 library, allowing future opportunities
+ for HDF5 to set a default plugin path such that the HDF5_PLUGIN_PATH
+ environment variable doesn't need to be set.
+
+## Building
+
+To enable building of an HDF5 VOL connector using HDF5's CMake functionality,
+two CMake variables must first be set:
+
+ HDF5_VOL_ALLOW_EXTERNAL (Default: OFF)
+ This variable determines whether or not building of external HDF5 VOL connectors
+ is enabled.
+
+ HDF5_ALLOW_EXTERNAL_SUPPORT (Default: "NO")
+ This variable is a string that specifies the manner in which the source code for
+ an external VOL connector will be retrieved. Currently, this variable must be set
+ to "GIT" for building external VOL connectors.
+
+Once the `HDF5_VOL_ALLOW_EXTERNAL` option is set to ON and the `HDF5_ALLOW_EXTERNAL_SUPPORT`
+variable is set to "GIT", the CMake cache will be populated with a predefined
+(currently 10) amount of new variables, named:
+
+ HDF5_VOL_URL01
+ HDF5_VOL_URL02
+ HDF5_VOL_URL03
+ ...
+
+For each of these variables, a URL that points to an HDF5 VOL connector Git
+repository can be specified. These URLs should currently be HTTPS URLs. For
+example, to specify the HDF5 Asynchronous I/O VOL Connector developed by the
+ECP team, one can provide the following option to `cmake`:
+
+ -DHDF5_VOL_URL01=https://github.com/hpc-io/vol-async.git
+
+For each URL specified, HDF5's CMake code will attempt to use CMake's
+[FetchContent](https://cmake.org/cmake/help/latest/module/FetchContent.html)
+functionality to retrieve the source code for a VOL connector pointed to by
+that URL and will try to build that VOL connector as part of the HDF5 library
+build process. The VOL connector must be able to be built by CMake and currently
+must have a CMakeLists.txt file in the top level of the source tree in order to
+be buildable by this process. If the source code for a VOL connector is successfully
+retrieved, the HDF5 build's CMake cache will be populated with variables from
+the VOL connector's CMake code, as if one were building the connector by itself.
+This gives one the ability to customize the build of the connector as usual.
+
+The CMake cache will also be populated with a few new variables for each VOL
+connector that was successfully retrieved from a given URL. To generate these
+variables, the CMake code first creates an internal name for the VOL connector
+by stripping off the last part of the Git repository URL given for the connector,
+removing the ".git" suffix and any whitespace and then upper-casing the result.
+For example, the name of the VOL connector located at the URL
+https://github.com/hpc-io/vol-async.git would become "VOL-ASYNC". Then, the following
+new variables get created:
+
+ HDF5_VOL_<VOL name>_BRANCH (Default: "main")
+ This variable specifies the git branch name or tag to use when fetching
+ the source code for the VOL connector with the CMake-internal name
+ '<VOL name>'.
+
+ HDF5_VOL_<VOL name>_NAME (Default: "")
+ This variable specifies the string that should be used when setting the
+ HDF5_VOL_CONNECTOR environment variable for testing the VOL connector
+ with the CMake-internal name '<VOL name>'. The value for this variable
+ can be determined according to the canonical name given to the connector
+ by the connector's author(s), as well as any extra info that needs to be
+ passed to the connector for its configuration (see example below). This
+ variable must be set in order for the VOL connector to be testable with
+ HDF5's tests.
+
+ HDF5_VOL_<VOL name>_TEST_PARALLEL (Default: OFF)
+ This variable determines whether the VOL connector with the CMake-internal
+ name '<VOL name>' should be tested against HDF5's parallel tests.
+
+As an example, this would create the following variables for the
+previously-mentioned VOL connector:
+
+ HDF5_VOL_VOL-ASYNC_BRANCH
+ HDF5_VOL_VOL-ASYNC_NAME
+ HDF5_VOL_VOL-ASYNC_TEST_PARALLEL
+
+**NOTE**
+If a VOL connector requires extra information to be passed in its
+HDF5_VOL_<VOL name>_NAME variable and that information contains any semicolons,
+those semicolons should be escaped with a single backslash so that CMake
+doesn't parse the string as a list. If `cmake` is run from a shell, extra care
+may need to be taken when escaping the semicolons depending on how the
+shell interprets backslashes.
+
+### Example - Build and test HDF5 Asynchronous I/O VOL connector
+
+Assuming that the HDF5 source code has been checked out and a build directory
+has been created, running the following cmake command from that build directory
+will retrieve, build and test the HDF5 Asynchronous I/O VOL connector while
+building HDF5. Note that `[hdf5 options]` represents other build options that
+would typically be passed when building HDF5, such as `CMAKE_INSTALL_PREFIX`,
+`HDF5_BUILD_CPP_LIB`, etc.
+
+ cmake [hdf5 options]
+ -DHDF5_ENABLE_THREADSAFE=ON
+ -DHDF5_ENABLE_PARALLEL=ON
+ -DALLOW_UNSUPPORTED=ON
+ -DHDF5_TEST_API=ON
+ -DHDF5_VOL_ALLOW_EXTERNAL=ON
+ -DHDF5_ALLOW_EXTERNAL_SUPPORT="GIT"
+ -DHDF5_VOL_URL01=https://github.com/hpc-io/vol-async.git
+ -DHDF5_VOL_VOL-ASYNC_BRANCH=develop
+ -DHDF5_VOL_VOL-ASYNC_NAME="async under_vol=0\;under_info={}"
+ -DHDF5_VOL_VOL-ASYNC_TEST_PARALLEL=ON ..
+
+Here, we are specifying that:
+
+ * HDF5 should be built with thread-safety enabled (required by Async VOL connector)
+ * HDF5 should be built with parallel enabled (required by Async VOL connector)
+ * Allow unsupported HDF5 combinations (thread-safety and HL, which is on by default)
+ * Enable the API tests so that they can be tested with the Async VOL connector
+ * Build and use the HDF5 Asynchronous I/O VOL connector, located at
+ https://github.com/hpc-io/vol-async.git
+ * Clone the Asynchronous I/O VOL connector from the repository's 'develop' branch
+ * When testing the Asynchronous I/O VOL connector, the `HDF5_VOL_CONNECTOR` environment
+ variable should be set to "async under_vol=0\;under_info={}", which
+ specifies that the VOL connector with the canonical name "async" should
+ be loaded and it should be passed the string "under_vol=0;under_info={}"
+ for its configuration
+ * The Asynchronous I/O VOL connector should be tested against HDF5's parallel API tests
+
+Note that this also assumes that the Asynchronous I/O VOL connector's
+[other dependencies](https://hdf5-vol-async.readthedocs.io/en/latest/gettingstarted.html#preparation)
+are installed on the system in a way that CMake can find them. If that is not
+the case, the locations for these dependencies may need to be provided to CMake
+by passing extra options, such as:
+
+ -DABT_INCLUDE_DIR=/path/to/argobots/build/include
+ -DABT_LIBRARY=/path/to/argbots/build/lib/libabt.so
+
+which would help CMake find an argobots installation in a non-standard location.
+
+## Testing
+
+To facilitate testing of HDF5 VOL connectors when building HDF5, tests from
+the [HDF5 VOL tests](https://github.com/hdfGroup/vol-tests) repository were
+integrated back into the library and the following new CMake options were
+added to HDF5 builds for the 1.14.1 release:
+
+ HDF5_TEST_API (Default: OFF)
+ This variable determines whether the HDF5 API tests will be built and tested.
+
+ HDF5_TEST_API_INSTALL (Default: OFF)
+ This variable determines whether the HDF5 API test executables will be installed
+ on the system alongside the HDF5 library.
+
+ HDF5_TEST_API_ENABLE_ASYNC (Default: OFF)
+ This variable determines whether the HDF5 Asynchronous I/O API tests will be
+ built and tested. These tests will only run if a VOL connector reports that
+ it supports asynchronous I/O operations when queried via the H5Pget_vol_cap_flags
+ API routine.
+
+ HDF5_TEST_API_ENABLE_DRIVER (Default: OFF)
+ This variable determines whether the HDF5 API test driver program will be
+ built and used for testing. This driver program is useful when a VOL connector
+ uses a client/server model where the server program needs to be up and running
+ before the VOL connector can function. This option is currently not functional.
+
+When the `HDF5_TEST_API` option is set to ON, HDF5's CMake code builds and tests
+the new API tests using the native VOL connector. When one or more external VOL
+connectors are built successfully with the process described in this document,
+the CMake code will duplicate some of these API tests by adding separate
+versions of the tests (for each VOL connector that was built) that set the
+`HDF5_VOL_CONNECTOR` environment variable to the value specified for the
+HDF5_VOL_<VOL name>_NAME variable for each external VOL connector at build time.
+Running the `ctest` command will then run these new tests which load and run with
+each VOL connector that was built in turn. When run via the `ctest` command, the
+new tests typically follow the naming scheme:
+
+ HDF5_VOL_<VOL name lowercase>-h5_api_test_<test name>
+ HDF5_VOL_<VOL name lowercase>-h5_api_test_parallel_<test name>
+
+**NOTE**
+If dependencies of a built VOL connector are installed on the system in
+a non-standard location that would typically require one to set `LD_LIBRARY_PATH`
+or similar, one should ensure that those environment variables are set before
+running tests. Otherwise, the tests that run with that connector will likely
+fail due to being unable to load the necessary libraries for its dependencies.