HDF5 version 1.14.4-1 currently under development ================================================================================ INTRODUCTION ============ This document describes the differences between this release and the previous HDF5 release. It contains information on the platforms tested and known problems in this release. For more details check the HISTORY*.txt files in the HDF5 source. Note that documentation in the links below will be updated at the time of each final release. Links to HDF5 documentation can be found on The HDF5 web page: https://portal.hdfgroup.org/display/HDF5/HDF5 The official HDF5 releases can be obtained from: https://www.hdfgroup.org/downloads/hdf5/ Changes from release to release and new features in the HDF5-1.14.x release series can be found at: https://portal.hdfgroup.org/display/HDF5/Release+Specific+Information If you have any questions or comments, please send them to the HDF Help Desk: help@hdfgroup.org CONTENTS ======== - New Features - Support for new platforms and languages - Bug Fixes since HDF5-1.14.3 - Platforms Tested - Known Problems - CMake vs. Autotools installations New Features ============ Configuration: ------------- - Renamed HDF5_ENABLE_USING_MEMCHECKER to HDF5_USING_ANALYSIS_TOOL The HDF5_USING_ANALYSIS_TOOL is used to indicate to test macros that an analysis tool is being used and that the tests should not use the runTest.cmake macros and it's variations. The analysis tools, like valgrind, test the macro code instead of the program under test. HDF5_ENABLE_USING_MEMCHECKER is still used for controlling the HDF5 define, H5_USING_MEMCHECKER. - New option for building and naming tools in CMake The following option has been added: HDF5_BUILD_STATIC_TOOLS "Build Static Tools Not Shared Tools" OFF The default will build shared tools unless BUILD_SHARED_LIBS = OFF. Tools will no longer have "-shared" as only one set of tools will be created. - Incorporated HDF5 examples repository into HDF5 library. The HDF5Examples folder is equivalent to the repository hdf5-examples. As such it can build and test the examples during library build or after the library is installed. Previously, the hdf5-repository archives were downloaded for packaging with the library. Now the examples can be built and tested without a packaged install of the library. However to maintain the ability to use the HDF5Examples with an installed library, it is necessary to translate or synch the option names from those used by the library to those used by the examples. The typical pattern is: = HDF_BUILD_FORTRAN = ${HDF5_BUILD_FORTRAN} - Added new option for CMake to mark tests as SKIPPED. HDF5_DISABLE_TESTS_REGEX is a REGEX string that will be checked with test names and if there is a match then that test's property will be set to DISABLED. HDF5_DISABLE_TESTS_REGEX can be initialized on the command line: "-DHDF5_DISABLE_TESTS_REGEX:STRING=" See CMake documentation for regex-specification. - Added defaults to CMake for long double conversion checks HDF5 performs a couple of checks at build time to see if long double values can be converted correctly (IBM's Power architecture uses a special format for long doubles). These checks were performed using TRY_RUN, which is a problem when cross-compiling. These checks now use default values appropriate for most non-Power systems when cross-compiling. The cache values can be pre-set if necessary, which will preempt both the TRY_RUN and the default. Affected values: H5_LDOUBLE_TO_LONG_SPECIAL (default no) H5_LONG_TO_LDOUBLE_SPECIAL (default no) H5_LDOUBLE_TO_LLONG_ACCURATE (default yes) H5_LLONG_TO_LDOUBLE_CORRECT (default yes) H5_DISABLE_SOME_LDOUBLE_CONV (default no) Fixes GitHub #3585 Library: -------- - Parallel Library: ----------------- - Fortran Library: ---------------- - Add API support for Fortran MPI_F08 module definitions: Adds support for MPI's MPI_F08 module datatypes: type(MPI_COMM) and type(MPI_INFO) for HDF5 APIs: H5PSET_FAPL_MPIO_F, H5PGET_FAPL_MPIO_F, H5PSET_MPI_PARAMS_F, H5PGET_MPI_PARAMS_F Ref. #3951 - Added Fortran APIs: H5FGET_INTENT_F, H5SSEL_ITER_CREATE_F, H5SSEL_ITER_GET_SEQ_LIST_F, H5SSEL_ITER_CLOSE_F, H5S_mp_H5SSEL_ITER_RESET_F - Added Fortran Parameters: H5S_SEL_ITER_GET_SEQ_LIST_SORTED_F, H5S_SEL_ITER_SHARE_WITH_DATASPACE_F - Added Fortran Parameters: H5S_BLOCK_F and H5S_PLIST_F - The configuration definitions file, H5config_f.inc, is now installed and the HDF5 version number has been added to it. - Added Fortran APIs: h5fdelete_f - Added Fortran APIs: h5vlnative_addr_to_token_f and h5vlnative_token_to_address_f C++ Library: ------------ - Java Library: ------------- - Tools: ------ - High-Level APIs: ---------------- - C Packet Table API: ------------------- - Internal header file: --------------------- - Documentation: -------------- - Support for new platforms, languages and compilers ================================================== - Bug Fixes since HDF5-1.14.3 release =================================== Library ------- - Fixed a bug where H5Tset_fields does not account for any offset set for a floating-point datatype when determining if values set for spos, epos, esize, mpos and msize make sense for the datatype Previously, H5Tset_fields did not take datatype offsets into account when determining if the values set make sense for the datatype. This would cause the function to fail when the precision for a datatype is correctly set such that the offset bits are not included. This has now been fixed. - Fixed H5Fget_access_plist so that it returns the file locking settings for a file When H5Fget_access_plist (and the internal H5F_get_access_plist) is called on a file, the returned File Access Property List has the library's default file locking settings rather than any settings set for the file. This causes two problems: - Opening an HDF5 file through an external link using H5Gopen, H5Dopen, etc. with H5P_DEFAULT for the Dataset/Group/etc. Access Property List will cause the external file to be opened with the library's default file locking settings rather than inheriting them from the parent file. This can be surprising when a file is opened with file locking disabled, but its external files are opened with file locking enabled. - An application cannot make use of the H5Pset_elink_fapl function to match file locking settings between an external file and its parent file without knowing the correct setting ahead of time, as calling H5Fget_access_plist on the parent file will not return the correct settings. This has been fixed by copying a file's file locking settings into the newly-created File Access Property List in H5F_get_access_plist. This fix partially addresses GitHub issue #4011 - Memory usage growth issue Starting with the HDF5 1.12.1 release, an issue (GitHub issue #1256) was observed where running a simple program that has a loop of opening a file, reading from an object with a variable-length datatype and then closing the file would result in the process fairly quickly running out of memory. Upon further investigation, it was determined that this memory was being kept around in the library's datatype conversion pathway cache that is used to speed up datatype conversions which are repeatedly used within an HDF5 application's lifecycle. For conversions involving variable-length or reference datatypes, each of these cached pathway entries keeps a reference to its associated file for later use. Since the file was being closed and reopened on each loop iteration, and since the library compares for equality between instances of opened files (rather than equality of the actual files) when determining if it can reuse a cached conversion pathway, it was determining that no cached conversion pathways could be reused and was creating a new cache entry on each loop iteration during I/O. This would lead to constant growth of that cache and the memory it consumed, as well as constant growth of the memory consumed by each cached entry for the reference to its associated file. To fix this issue, the library now removes any cached datatype conversion path entries for variable-length or reference datatypes associated with a particular file when that file is closed. Fixes GitHub #1256 - Suppressed floating-point exceptions in H5T init code The floating-point datatype initialization code in H5Tinit_float.c could raise FE_INVALID exceptions while munging bits and performing comparisons that might involve NaN. This was not a problem when the initialization code was executed in H5detect at compile time (prior to 1.14.3), but now that the code is executed at library startup (1.14.3+), these exceptions can be caught by user code, as is the default in the NAG Fortran compiler. Starting in 1.14.4, we now suppress floating-point exceptions while initializing the floating-point types and clear FE_INVALID before restoring the original environment. Fixes GitHub #3831 - Fixed a file handle leak in the core VFD When opening a file with the core VFD and a file image, if the file already exists, the file check would leak the POSIX file handle. Fixes GitHub issue #635 - Dropped support for MPI-2 The MPI-2 supporting artifacts have been removed due to the cessation of MPI-2 maintenance and testing since version HDF5 1.12. Java Library ------------ - Configuration ------------- - Changed default of 'Error on HDF5 doxygen warnings' DOXYGEN_WARN_AS_ERROR option. The default setting of DOXYGEN_WARN_AS_ERROR to 'FAIL_ON_WARNINGS' has been changed to 'NO'. It was decided that the setting was too aggressive and should be a user choice. The github actions and scripts have been updated to reflect this. * HDF5_ENABLE_DOXY_WARNINGS: ON/OFF (Default: OFF) * --enable-doxygen-errors: enable/disable (Default: disable) - Removed an Autotools configure hack that causes problems on MacOS A sed line in configure.ac was added in the past to paper over some problems with older versions of the Autotools that would add incorrect linker flags. This hack is not needed with recent versions of the Autotools and the sed line errors on MacOS (though this was a silent error that didn't break the build) so the hack has been removed. Fixes GitHub issue #3843 - Fixed an issue where the h5tools_test_utils test program was being installed on the system for Autotools builds of HDF5 The h5tools_test_utils test program was mistakenly added to bin_PROGRAMS in its Makefile.am configuration file, causing the executable to be installed on the system. The executable is now added to noinst_PROGRAMS instead and will no longer be installed on the system for Autotools builds of HDF5. The CMake configuration code already avoids installing the executable on the system. Tools ----- - Renamed h5fuse.sh to h5fuse Addresses Discussion #3791 Performance ------------- - Fortran API ----------- - High-Level Library ------------------ - Fixed a memory leak in H5LTopen_file_image with H5LT_FILE_IMAGE_DONT_COPY flag When the H5LT_FILE_IMAGE_DONT_COPY flag is passed to H5LTopen_file_image, the internally-allocated udata structure gets leaked as the core file driver doesn't have a way to determine when or if it needs to call the "udata_free" callback. This has been fixed by freeing the udata structure when the "image_free" callback gets made during file close, where the file is holding the last reference to the udata structure. Fixes GitHub issue #827 Fortran High-Level APIs ----------------------- - Documentation ------------- - F90 APIs -------- - C++ APIs -------- - Testing ------- - Platforms Tested =================== Linux 5.19.0-1023-aws GNU gcc, gfortran, g++ #24-Ubuntu SMP x86_64 GNU/Linux (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 Ubuntu 22.04 Ubuntu clang version 14.0.0-1ubuntu1 Intel(R) oneAPI DPC++/C++ Compiler 2023.1.0 ifort (IFORT) 2021.9.0 20230302 (cmake and autotools) Linux 5.16.14-200.fc35 GNU gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9) #1 SMP x86_64 GNU/Linux GNU Fortran (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9) Fedora35 clang version 13.0.0 (Fedora 13.0.0-3.fc35) (cmake and autotools) Linux 5.14.21-cray_shasta_c cray-mpich/8.1.23 #1 SMP x86_64 GNU/Linux cce/15.0.0 (frontier) gcc/12.2.0 (cmake) Linux 5.11.0-34-generic GNU gcc (GCC) 9.4.0-1ubuntu1 #36-Ubuntu SMP x86_64 GNU/Linux GNU Fortran (GCC) 9.4.0-1ubuntu1 Ubuntu 20.04 Ubuntu clang version 10.0.0-4ubuntu1 Intel(R) oneAPI DPC++/C++ Compiler 2023.1.0 ifort (IFORT) 2021.9.0 20230302 (cmake and autotools) Linux 4.14.0-115.35.1.1chaos aue/openmpi/4.1.4-arm-22.1.0.12 #1 SMP aarch64 GNU/Linux Arm C/C++/Fortran Compiler version 22.1 (stria) (based on LLVM 13.0.1) (cmake) Linux 4.14.0-115.35.1.3chaos spectrum-mpi/rolling-release #1 SMP ppc64le GNU/Linux clang 12.0.1 (vortex) GCC 8.3.1 XL 2021.09.22 (cmake) Linux-4.14.0-115.21.2 spectrum-mpi/rolling-release #1 SMP ppc64le GNU/Linux clang 12.0.1, 14.0.5 (lassen) GCC 8.3.1 XL 16.1.1.2, 2021.09.22, 2022.08.05 (cmake) Linux-4.12.14-197.99-default cray-mpich/7.7.14 #1 SMP x86_64 GNU/Linux cce 12.0.3 (theta) GCC 11.2.0 llvm 9.0 Intel 19.1.2 Linux 3.10.0-1160.36.2.el7.ppc64 gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) #1 SMP ppc64be GNU/Linux g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) Power8 (echidna) GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39) Linux 3.10.0-1160.24.1.el7 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: Centos7 Version 4.8.5 20150623 (Red Hat 4.8.5-4) (jelly/kituo/moohan) Version 4.9.3, Version 7.2.0, Version 8.3.0, Version 9.1.0, Version 10.2.0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 17.0.0.098 Build 20160721 GNU C (gcc) and C++ (g++) 4.8.5 compilers with NAG Fortran Compiler Release 7.1(Hanzomon) Intel(R) C (icc) and C++ (icpc) 17.0.0.098 compilers with NAG Fortran Compiler Release 7.1(Hanzomon) MPICH 3.1.4 compiled with GCC 4.9.3 MPICH 3.3 compiled with GCC 7.2.0 OpenMPI 3.1.3 compiled with GCC 7.2.0 and 4.1.2 compiled with GCC 9.1.0 PGI C, Fortran, C++ for 64-bit target on x86_64; Versions 18.4.0 and 19.10-0 NVIDIA nvc, nvfortran and nvc++ version 22.5-0 (autotools and cmake) Linux-3.10.0-1160.0.0.1chaos openmpi-4.1.2 #1 SMP x86_64 GNU/Linux clang 6.0.0, 11.0.1 (quartz) GCC 7.3.0, 8.1.0 Intel 19.0.4, 2022.2, oneapi.2022.2 Linux-3.10.0-1160.90.1.1chaos openmpi/4.1 #1 SMP x86_64 GNU/Linux GCC 7.2.0 (skybridge) Intel/19.1 (cmake) Linux-3.10.0-1160.90.1.1chaos openmpi/4.1 #1 SMP x86_64 GNU/Linux GCC 7.2.0 (attaway) Intel/19.1 (cmake) Linux-3.10.0-1160.90.1.1chaos openmpi-intel/4.1 #1 SMP x86_64 GNU/Linux Intel/19.1.2, 21.3.0 and 22.2.0 (chama) (cmake) macOS Apple M1 11.6 Apple clang version 12.0.5 (clang-1205.0.22.11) Darwin 20.6.0 arm64 gfortran GNU Fortran (Homebrew GCC 11.2.0) 11.1.0 (macmini-m1) Intel icc/icpc/ifort version 2021.3.0 202106092021.3.0 20210609 macOS Big Sur 11.3.1 Apple clang version 12.0.5 (clang-1205.0.22.9) Darwin 20.4.0 x86_64 gfortran GNU Fortran (Homebrew GCC 10.2.0_3) 10.2.0 (bigsur-1) Intel icc/icpc/ifort version 2021.2.0 20210228 Mac OS X El Capitan 10.11.6 Apple clang version 7.3.0 from Xcode 7.3 64-bit gfortran GNU Fortran (GCC) 5.2.0 (osx1011test) Intel icc/icpc/ifort version 16.0.2 Linux 2.6.32-573.22.1.el6 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: Centos6 Version 4.4.7 20120313 (platypus) Version 4.9.3, 5.3.0, 6.2.0 MPICH 3.1.4 compiled with GCC 4.9.3 PGI C, Fortran, C++ for 64-bit target on x86_64; Version 19.10-0 Windows 10 x64 Visual Studio 2019 w/ clang 12.0.0 with MSVC-like command-line (C/C++ only - cmake) Visual Studio 2019 w/ Intel (C/C++ only - cmake) Visual Studio 2022 w/ clang 15.0.1 with MSVC-like command-line (C/C++ only - cmake) Visual Studio 2022 w/ Intel C/C++/Fortran oneAPI 2023 (cmake) Visual Studio 2019 w/ MSMPI 10.1 (C only - cmake) Known Problems ============== When HDF5 is compiled with NVHPC versions 23.5 - 23.9 (additional versions may also be applicable) and with -O2 (or higher) and -DNDEBUG, test failures occur in the following tests: H5PLUGIN-filter_plugin H5TEST-flush2 H5TEST-testhdf5-base MPI_TEST_t_filters_parallel Since these tests pass with an optimization level of -O1 (and -O0) and it is currently unclear whether the test failures are due to issues in HDF5 or issues in the 'nvc' compiler, the maximum optimization level for NVHPC has been set to -O1 until the test failures can be resolved. Note that even at -O1 optimization level, there still appears to be a sporadic test failure in the Java JUnit tests that has occasionally been seen in JUnit-TestH5Pfapl and JUnit-TestH5D. It is also unclear whether this is an issue in HDF5 or with the 'nvc' compiler. Finally, note that NVHPC 23.9 will fail to compile the test/tselect.c test file with a compiler error of 'use of undefined value' when the optimization level is -O2 or higher. Nvidia is aware of this issue and has suggested lowering the optimization level to -O1 for the time being: https://forums.developer.nvidia.com/t/hdf5-no-longer-compiles-with-nv-23-9/269045. CMake files do not behave correctly with paths containing spaces. Do not use spaces in paths because the required escaping for handling spaces results in very complex and fragile build files. At present, metadata cache images may not be generated by parallel applications. Parallel applications can read files with metadata cache images, but since this is a collective operation, a deadlock is possible if one or more processes do not participate. CPP ptable test fails on both VS2017 and VS2019 with Intel compiler, JIRA issue: HDFFV-10628. This test will pass with VS2015 with Intel compiler. The subsetting option in ph5diff currently will fail and should be avoided. The subsetting option works correctly in serial h5diff. Several tests currently fail on certain platforms: MPI_TEST-t_bigio fails with spectrum-mpi on ppc64le platforms. MPI_TEST-t_subfiling_vfd and MPI_TEST_EXAMPLES-ph5_subfiling fail with cray-mpich on theta and with XL compilers on ppc64le platforms. MPI_TEST_testphdf5_tldsc fails with cray-mpich 7.7 on cori and theta. Known problems in previous releases can be found in the HISTORY*.txt files in the HDF5 source. Please report any new problems found to help@hdfgroup.org. CMake vs. Autotools installations ================================= While both build systems produce similar results, there are differences. Each system produces the same set of folders on linux (only CMake works on standard Windows); bin, include, lib and share. Autotools places the COPYING and RELEASE.txt file in the root folder, CMake places them in the share folder. The bin folder contains the tools and the build scripts. Additionally, CMake creates dynamic versions of the tools with the suffix "-shared". Autotools installs one set of tools depending on the "--enable-shared" configuration option. build scripts ------------- Autotools: h5c++, h5cc, h5fc CMake: h5c++, h5cc, h5hlc++, h5hlcc The include folder holds the header files and the fortran mod files. CMake places the fortran mod files into separate shared and static subfolders, while Autotools places one set of mod files into the include folder. Because CMake produces a tools library, the header files for tools will appear in the include folder. The lib folder contains the library files, and CMake adds the pkgconfig subfolder with the hdf5*.pc files used by the bin/build scripts created by the CMake build. CMake separates the C interface code from the fortran code by creating C-stub libraries for each Fortran library. In addition, only CMake installs the tools library. The names of the szip libraries are different between the build systems. The share folder will have the most differences because CMake builds include a number of CMake specific files for support of CMake's find_package and support for the HDF5 Examples CMake project. The issues with the gif tool are: HDFFV-10592 CVE-2018-17433 HDFFV-10593 CVE-2018-17436 HDFFV-11048 CVE-2020-10809 These CVE issues have not yet been addressed and are avoided by not building the gif tool by default. Enable building the High-Level tools with these options: autotools: --enable-hlgiftools cmake: HDF5_BUILD_HL_GIF_TOOLS=ON