diff options
Diffstat (limited to 'release_docs/RELEASE.txt')
| -rw-r--r-- | release_docs/RELEASE.txt | 1786 |
1 files changed, 803 insertions, 983 deletions
diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt index 3d662c7..20d561c 100644 --- a/release_docs/RELEASE.txt +++ b/release_docs/RELEASE.txt @@ -3,6 +3,7 @@ HDF5 version 1.13.0 currently under development INTRODUCTION +============ This document describes the differences between this release and the previous HDF5 release. It contains information on the platforms tested and known @@ -31,11 +32,11 @@ If you have any questions or comments, please send them to the HDF Help Desk: CONTENTS +======== - New Features - Support for new platforms and languages -- Bug Fixes since HDF5-1.10.3 -- Bug Fixes since HDF5-1.10.2 +- Bug Fixes since HDF5-1.12.0 - Supported Platforms - Tested Configuration Features Summary - More Tested Platforms @@ -48,1349 +49,1156 @@ New Features Configuration: ------------- - - CMake option to link the generated Fortran MOD files into the include - directory. + - A C++11-compliant compiler is now required to build the C++ wrappers - The Fortran generation of MOD files by a Fortran compile can produce - different binary files between SHARED and STATIC compiles with different - compilers and/or different platforms. Note that it has been found that - different versions of Fortran compilers will produce incompatible MOD - files. Currently, CMake will locate these MOD files in subfolders of - the include directory and add that path to the Fortran library target - in the CMake config file, which can be used by the CMake find library - process. For other build systems using the binary from a CMake install, - a new CMake configuration can be used to copy the pre-chosen version - of the Fortran MOD files into the install include directory. - - The default will depend on the configuration of - BUILD_STATIC_LIBS and BUILD_SHARED_LIBS: - YES YES Default to SHARED - YES NO Default to STATIC - NO YES Default to SHARED - NO NO Default to SHARED - The defaults can be overriden by setting the config option - HDF5_INSTALL_MOD_FORTRAN to one of NO, SHARED, or STATIC + CMAKE_CXX_STANDARD is now set to 11 when building with CMake and + -std=c++11 is added when building with clang/gcc via the Autotools. - (ADB - 2020/07/9, HDFFV-11116) + (DER - 2021/05/27) - - CMake option to use AEC (open source SZip) library instead of SZip - The open source AEC library is a replacement library for SZip. In - order to use it for hdf5 the libaec CMake source was changed to add - "-fPIC" and exclude test files. Autotools does not build the - compression libraries within hdf5 builds. New option USE_LIBAEC is - required to compensate for the different files produced by AEC build. + - CMake will now run the shell script tests in test/ by default - (ADB - 2020/04/22, OESS-65) + The test directory includes several shell script tests that previously + were not run by CMake. These are now run by default. TEST_SHELL_SCRIPTS + has been set to ON and SH_PROGRAM has been set to bash (some test + scripts use bash-isms). Platforms without bash (e.g., Windows) will + ignore the script tests. - - CMake ConfigureChecks.cmake file now uses CHECK_STRUCT_HAS_MEMBER + (DER - 2021/05/23) - Some handcrafted tests in HDFTests.c has been removed and the CMake - CHECK_STRUCT_HAS_MEMBER module has been used. + - Removed unused HDF5_ENABLE_HSIZET option from CMake - (ADB - 2020/03/24, TRILAB-24) + This has been unused for some time and has no effect. - - Both build systems use same set of warnings flags + (DER - 2021/05/23) - GNU C, C++ and gfortran warnings flags were moved to files in a config - sub-folder named gnu-warnings. Flags that only are available for a specific - version of the compiler are in files named with that version. - Clang C warnings flags were moved to files in a config sub-folder - named clang-warnings. - Intel C, Fortran warnings flags were moved to files in a config sub-folder - named intel-warnings. + - CMake no longer builds the C++ library by default - There are flags in named "error-xxx" files with warnings that may - be promoted to errors. Some source files may still need fixes. + HDF5_BUILD_CPP_LIB now defaults to OFF, which is in line with the + Autotools build defaults. - There are also pairs of files named "developer-xxx" and "no-developer-xxx" - that are chosen by the CMake option:HDF5_ENABLE_DEV_WARNINGS or the - configure option:--enable-developer-warnings. + (DER - 2021/04/20) - In addition, CMake no longer applies these warnings for examples. + - Removal of pre-VS2015 work-arounds - (ADB - 2020/03/24, TRILAB-192) + HDF5 now requires Visual Studio 2015 or greater, so old work-around + code and definitions have been removed, including: - - Added test script for file size compare + * <inttypes.h> + * snprintf and vsnprintf + * llround, llroundf, lround, lroundf, round, roundf + * strtoll and strtoull + * va_copy + * struct timespec - if CMake minimum version is at least 3.14, the fileCompareTest.cmake - script will compare file sizes. + (DER - 2021/03/22) - (ADB - 2020/02/24, HDFFV-11036) + - Add CMake variable HDF5_LIB_INFIX - - Update CMake minimum version to 3.12 + This infix is added to all library names after 'hdf5'. + e.g. the infix '_openmpi' results in the library name 'libhdf5_openmpi.so' + This name is used in packages on debian based systems. + (see https://packages.debian.org/jessie/amd64/libhdf5-openmpi-8/filelist) - Updated CMake minimum version to 3.12 and added version checks - for Windows features. + (barcode - 2021/03/22) - (ADB - 2020/02/05, TRILABS-142) + - On macOS, Universal Binaries can now be built, allowing native execution on + both Intel and Apple Silicon (ARM) based Macs. - - Fixed CMake include properties for Fortran libraries + To do so, set CMAKE_OSX_ARCHITECTURES="x86_64;arm64" - Corrected the library properties for Fortran to use the - correct path for the Fortran module files. + (SAM - 2021/02/07, https://github.com/HDFGroup/hdf5/issues/311) - (ADB - 2020/02/04, HDFFV-11012) + - Added a configure-time option to control certain compiler warnings + diagnostics - - Added common warnings files for gnu and intel + A new configure-time option was added that allows some compiler warnings + diagnostics to have the default operation. This is mainly intended for + library developers and currently only works for gcc 10 and above. The + diagnostics flags apply to C, C++ and Fortran compilers and will appear + in "H5 C Flags", H5 C++ Flags" and H5 Fortran Flags, respectively. They + will NOT be exported to h5cc, etc. - Added warnings files to use one common set of flags - during configure for both autotools and CMake build - systems. The initial implementation only affects a - general set of flags for gnu and intel compilers. + The default is OFF, which will disable the warnings URL and color attributes + for the warnings output. ON will not add the flags and allow default behavior. - (ADB - 2020/01/17) + Autotools: --enable-diags - - Added new options to CMake for control of testing + CMake: HDF5_ENABLE_BUILD_DIAGS - Added CMake options (default ON); - HDF5_TEST_SERIAL AND/OR HDF5_TEST_PARALLEL - combined with: - HDF5_TEST_TOOLS - HDF5_TEST_EXAMPLES - HDF5_TEST_SWMR - HDF5_TEST_FORTRAN - HDF5_TEST_CPP - HDF5_TEST_JAVA + (ADB - 2021/02/05, HDFFV-11213) - (ADB - 2020/01/15, HDFFV-11001) + - CMake option to build the HDF filter plugins project as an external project - - Added Clang sanitizers to CMake for analyzer support if compiler is clang. + The HDF filter plugins project is a collection of registered compression + filters that can be dynamically loaded when needed to access data stored + in a hdf5 file. This CMake-only option allows the plugins to be built and + distributed with the hdf5 library and tools. Like the options for szip and + zlib, either a tgz file or a git repository can be specified for the source. - Added CMake code and files to execute the Clang sanitizers if - HDF5_ENABLE_SANITIZERS is enabled and the USE_SANITIZER option - is set to one of the following: - Address - Memory - MemoryWithOrigins - Undefined - Thread - Leak - 'Address;Undefined' + The option was refactored to use the CMake FetchContent process. This allows + more control over the filter targets, but required external project command + options to be moved to a CMake include file, HDF5PluginCache.cmake. Also + enabled the filter examples to be used as tests for operation of the + filter plugins. - (ADB - 2019/12/12, TRILAB-135) + (ADB - 2020/12/10, OESS-98) - - Update CMake for VS2019 support + - FreeBSD Autotools configuration now defaults to 'cc' and 'c++' compilers - CMake added support for VS2019 in version 3.15. Changes to the CMake - generator setting required changes to scripts. Also updated version - references in CMake files as necessary. + On FreeBSD, the autotools defaulted to 'gcc' as the C compiler and did + not process C++ options. Since FreeBSD 10, the default compiler has + been clang (via 'cc'). - (ADB - 2019/11/18, HDFFV-10962) + The default compilers have been set to 'cc' for C and 'c++' for C++, + which will pick up clang and clang++ respectively on FreeBSD 10+. + Additionally, clang options are now set correctly for both C and C++ + and g++ options will now be set if that compiler is being used (an + omission from the former functionality). - - Update CMake options to match new autotools options + (DER - 2020/11/28, HDFFV-11193) - Add configure options (autotools - CMake): - enable-asserts HDF5_ENABLE_ASSERTS - enable-symbols HDF5_ENABLE_SYMBOLS - enable-profiling HDF5_ENABLE_PROFILING - enable-optimization HDF5_ENABLE_OPTIMIZATION - In addition NDEBUG is no longer forced defined and relies on the CMake - process. - - (ADB - 2019/10/07, HDFFV-100901, HDFFV-10637, TRILAB-97) - - - Update CMake tests to use FIXTURES - - CMake test fixtures allow setup/cleanup tests and other dependency - requirements as properties for tests. This is more flexible for - modern CMake code. - - (ADB - 2019/07/23, HDFFV-10529) - - - Windows PDB files are always installed - - There are build configuration or flag settings for Windows that may not - generate PDB files. If those files are not generated then the install - utility will fail because those PDB files are not found. An optional - variable, DISABLE_PDB_FILES, was added to not install PDB files. - - (ADB - 2019/07/17, HDFFV-10424) - - - Add mingw CMake support with a toolchain file + - Fixed POSIX problems when building w/ gcc on Solaris - There has been a number of mingw issues that has been linked under - HDFFV-10845. It has been decided to implement the CMake cross-compiling - technique of toolchain files. We will use a linux platform with the mingw - compiler stack for testing. Only the C language is fully supported, and - the error tests are skipped. The C++ language works for static but shared - builds has a shared library issue with the mingw Standard Exception Handling - library, which is not available on Windows. Fortran has a common cross-compile - problem with the fortran configure tests. + When building on Solaris using gcc, the POSIX symbols were not + being set correctly, which could lead to issues like clock_gettime() + not being found. - (ADB - 2019/07/12, HDFFV-10845, HDFFV-10595) + The standard is now set to gnu99 when building with gcc on Solaris, + which allows POSIX things to be #defined and linked correctly. This + differs slightly from the gcc norm, where we set the standard to c99 + and manually set POSIX #define symbols. - - Windows PDB files are installed incorrectly + (DER - 2020/11/25, HDFFV-11191) - For static builds, the PDB files for windows should be installed next - to the static libraries in the lib folder. Also the debug versions of - libraries and PDB files are now correctly built using the default - CMAKE_DEBUG_POSTFIX setting. + - Added a configure-time option to consider certain compiler warnings + as errors - (ADB - 2019/07/09, HDFFV-10581) + A new configure-time option was added that converts some compiler warnings + to errors. This is mainly intended for library developers and currently + only works for gcc and clang. The warnings that are considered errors + will appear in the generated libhdf5.settings file. These warnings apply + to C and C++ code and will appear in "H5 C Flags" and H5 C++ Flags", + respectively. They will NOT be exported to h5cc, etc. - - Add option to build only shared libs + The default is OFF. Building with this option may fail when compiling + on operating systems and with compiler versions not commonly used by + the library developers. Compilation may also fail when headers not + under the control of the library developers (e.g., mpi.h, hdfs.h) raise + warnings. - A request was made to prevent building static libraries and only build - shared. A new option was added to CMake, ONLY_SHARED_LIBS, which will - skip building static libraries. Certain utility functions will build with - static libs but are not published. Tests are adjusted to use the correct - libraries depending on SHARED/STATIC settings. + Autotools: --enable-warnings-as-errors - (ADB - 2019/06/12, HDFFV-10805) + CMake: HDF5_ENABLE_WARNINGS_AS_ERRORS - - Add options to enable or disable building tools and tests + (DER - 2020/11/23, HDFFV-11189) - Configure options --enable-tests and --enable-tools were added for - autotools configure. These options are enabled by default, and can be - disabled with either --disable-tests (or tools) or --enable-tests=no - (or --enable-tools=no). Build time is reduced ~20% when tools are - disabled, 35% when tests are disabled, 45% when both are disabled. - Reenabling them after the initial build requires running configure - again with the option(s) enabled. + - Autotools and CMake target added to produce doxygen generated documentation - (LRK - 2019/06/12, HDFFV-9976) + The default is OFF or disabled. + Autoconf option is '--enable-doxygen' + autotools make target is 'doxygen' and will build all doxygen targets + CMake configure option is 'HDF5_BUILD_DOC'. + CMake target is 'doxygen' for all available doxygen targets + CMake target is 'hdf5lib_doc' for the src subdirectory - - Change tools test that test the error stack + (ADB - 2020/11/03) - There are some use cases which can cause the error stack of tools to be - different then the expected output. These tests now use grepTest.cmake, - this was changed to allow the error file to be searched for an expected string. + - CMake option to use MSVC naming conventions with MinGW - (ADB - 2019/04/15, HDFFV-10741) + HDF5_MSVC_NAMING_CONVENTION option enable to use MSVC naming conventions + when using a MinGW toolchain - - Keep stderr and stdout separate in tests + (xan - 2020/10/30) - Changed test handling of output capture. Tests now keep the stderr - output separate from the stdout output. It is up to the test to decide - which output to check against a reference. Also added the option - to grep for a string in either output. + - CMake option to statically link gcc libs with MinGW - (ADB - 2018/12/12, HDFFV-10632) + HDF5_MINGW_STATIC_GCC_LIBS allows to statically link libg/libstdc++ + with the MinGW toolchain - - Add toolchain and cross-compile support + (xan - 2020/10/30) - Added info on using a toolchain file to INSTALL_CMAKE.txt. A - toolchain file is also used in cross-compiling, which requires - CMAKE_CROSSCOMPILING_EMULATOR to be set. To help with cross-compiling - the fortran configure process, the HDF5UseFortran.cmake file macros - were improved. Fixed a Fortran configure file issue that incorrectly - used #cmakedefine instead of #define. + - CMake option to build the HDF filter plugins project as an external project - (ADB - 2018/10/04, HDFFV-10594) + The HDF filter plugins project is a collection of registered compression + filters that can be dynamically loaded when needed to access data stored + in a hdf5 file. This CMake-only option allows the plugins to be built and + distributed with the hdf5 library and tools. Like the options for szip and + zlib, either a tgz file or a git repository can be specified for the source. - - Add warning flags for Intel compilers + The necessary options are (see the INSTALL_CMake.txt file): + HDF5_ENABLE_PLUGIN_SUPPORT + PLUGIN_TGZ_NAME or PLUGIN_GIT_URL + There are more options necessary for various filters and the plugin project + documents should be referenced. - Identified Intel compiler specific warnings flags that should be used - instead of GNU flags. + (ADB - 2020/09/27, OESS-98) - (ADB - 2018/10/04, TRILABS-21) + - Added CMake option to format source files - - Add default rpath to targets + HDF5_ENABLE_FORMATTERS option will enable creation of targets using the + pattern - HDF5_*_SRC_FORMAT - where * corresponds to the source folder + or tool folder. All sources can be formatted by executing the format target; + make format - Default rpaths should be set in shared executables and - libraries to allow the use of loading dependent libraries - without requiring LD_LIBRARY_PATH to be set. The default - path should be relative using @rpath on osx and $ORIGIN - on linux. Windows is not affected. + (ADB - 2020/08/24) - (ADB - 2018/09/26, HDFFV-10594) + - Add file locking configure and CMake options - - Add missing USE_110_API_DEFAULT option. + HDF5 1.10.0 introduced a file locking scheme, primarily to help + enforce SWMR setup. Formerly, the only user-level control of the scheme + was via the HDF5_USE_FILE_LOCKING environment variable. - Option USE_110_API_DEFAULT sets the default version of - versioned APIs. The bin/makevers perl script did not set - the maxidx variable correctly when the 1.10 branch was - created. This caused the versioning process to always use - the latest version of any API. + This change introduces configure-time options that control whether + or not file locking will be used and whether or not the library + ignores errors when locking has been disabled on the file system + (useful on some HPC Lustre installations). - (ADB - 2018/08/17, HDFFV-10552) + In both the Autotools and CMake, the settings have the effect of changing + the default property list settings (see the H5Pset/get_file_locking() + entry, below). - - Added configuration checks for the following MPI functions: + The yes/no/best-effort file locking configure setting has also been + added to the libhdf5.settings file. - MPI_Mprobe - Used for the Parallel Compression feature - MPI_Imrecv - Used for the Parallel Compression feature + Autotools: - MPI_Get_elements_x - Used for the "big Parallel I/O" feature - MPI_Type_size_x - Used for the "big Parallel I/O" feature + An --enable-file-locking=(yes|no|best-effort) option has been added. - (JTH - 2018/08/02, HDFFV-10512) + yes: Use file locking. + no: Do not use file locking. + best-effort: Use file locking and ignore "disabled" errors. - - Added section to the libhdf5.settings file to indicate - the status of the Parallel Compression and "big Parallel I/O" - features. + CMake: - (JTH - 2018/08/02, HDFFV-10512) + Two self-explanatory options have been added: - - Add option to execute swmr shell scripts from CMake. + HDF5_USE_FILE_LOCKING + HDF5_IGNORE_DISABLED_FILE_LOCKS - Option TEST_SHELL_SCRIPTS redirects processing into a - separate ShellTests.cmake file for UNIX types. The tests - execute the shell scripts if a SH program is found. + Setting both of these to ON is the equivalent to the Autotools' + best-effort setting. - (ADB - 2018/07/16) + NOTE: + The precedence order of the various file locking control mechanisms is: + 1) HDF5_USE_FILE_LOCKING environment variable (highest) - Library: - -------- - - Add Mirror VFD + 2) H5Pset_file_locking() - Use TCP/IP sockets to perform write-only (W/O) file I/O on a remote - machine. Must be used in conjunction with the Splitter VFD. + 3) configure/CMake options (which set the property list defaults) - (JOS - 2020/03/13, TBD) + 4) library defaults (currently best-effort) - - Add Splitter VFD + (DER - 2020/07/30, HDFFV-11092) - Maintain separate R/W and W/O channels for "concurrent" file writes - to two files using a single HDF5 file handle. + - CMake option to link the generated Fortran MOD files into the include + directory. - (JOS - 2020/03/13, TBD) + The Fortran generation of MOD files by a Fortran compile can produce + different binary files between SHARED and STATIC compiles with different + compilers and/or different platforms. Note that it has been found that + different versions of Fortran compilers will produce incompatible MOD + files. Currently, CMake will locate these MOD files in subfolders of + the include directory and add that path to the Fortran library target + in the CMake config file, which can be used by the CMake find library + process. For other build systems using the binary from a CMake install, + a new CMake configuration can be used to copy the pre-chosen version + of the Fortran MOD files into the install include directory. + + The default will depend on the configuration of + BUILD_STATIC_LIBS and BUILD_SHARED_LIBS: + YES YES Default to SHARED + YES NO Default to STATIC + NO YES Default to SHARED + NO NO Default to SHARED + The defaults can be overridden by setting the config option + HDF5_INSTALL_MOD_FORTRAN to one of NO, SHARED, or STATIC - - Refactored public exposure of haddr_t type in favor of "object tokens" + (ADB - 2020/07/09, HDFFV-11116) - To better accommodate HDF5 VOL connectors where "object addresses in a file" - may not make much sense, the following changes were made to the library: - - * Introduced new H5O_token_t "object token" type, which represents a - unique and permanent identifier for referencing an HDF5 object within - a container; these "object tokens" are meant to replace object addresses. - Along with the new type, a new H5Oopen_by_token API call was introduced - to open an object by a token, similar to how object addresses were - previously used with H5Oopen_by_addr. - - * Introduced new H5Lget_info2, H5Lget_info_by_idx2, H5Literate2, H5Literate_by_name2, - H5Lvisit2 and H5Lvisit_by_name2 API calls, along with their associated H5L_info2_t - struct and H5L_iterate2_t callback function, which work with the newly-introduced - object tokens, instead of object addresses. The original functions have been - renamed to version 1 functions and are deprecated in favor of the new version 2 - functions. The H5L_info_t and H5L_iterate_t types have been renamed to version 1 - types and are now deprecated in favor of their version 2 counterparts. For each of - the functions and types, compatibility macros take place of the original symbols. - - * Introduced new H5Oget_info3, H5Oget_info_by_name3, H5Oget_info_by_idx3, - H5Ovisit3 and H5Ovisit_by_name3 API calls, along with their associated H5O_info2_t - struct and H5O_iterate2_t callback function, which work with the newly-introduced - object tokens, instead of object addresses. The version 2 functions are now - deprecated in favor of the version 3 functions. The H5O_info_t and H5O_iterate_t - types have been renamed to version 1 types and are now deprecated in favor of their - version 2 counterparts. For each, compatibility macros take place of the original - symbols. - - * Introduced new H5Oget_native_info, H5Oget_native_info_by_name and - H5Oget_native_info_by_idx API calls, along with their associated H5O_native_info_t - struct, which are used to retrieve the native HDF5 file format-specific information - about an object. This information (such as object header info and B-tree/heap info) - has been removed from the new H5O_info2_t struct so that the more generic - H5Oget_info(_by_name/_by_idx)3 routines will not try to retrieve it for non-native - VOL connectors. - - * Added new H5Otoken_cmp, H5Otoken_to_str and H5Otoken_from_str routines to compare - two object tokens, convert an object token into a nicely-readable string format and - to convert an object token string back into a real object token, respectively. - - (DER, QAK, JTH - 2020/01/16) + - CMake option to use AEC (open source SZip) library instead of SZip - - Add new public function H5Sselect_adjust. + The open source AEC library is a replacement library for SZip. In + order to use it for hdf5 the libaec CMake source was changed to add + "-fPIC" and exclude test files. Autotools does not build the + compression libraries within hdf5 builds. New option USE_LIBAEC is + required to compensate for the different files produced by AEC build. - This function shifts a dataspace selection by a specified logical offset - within the dataspace extent. This can be useful for VOL developers to - implement chunked datasets. + (ADB - 2020/04/22, OESS-65) - (NAF - 2019/11/18) + - CMake ConfigureChecks.cmake file now uses CHECK_STRUCT_HAS_MEMBER - - Add new public function H5Sselect_project_intersection. + Some handcrafted tests in HDFTests.c has been removed and the CMake + CHECK_STRUCT_HAS_MEMBER module has been used. - This function computes the intersection between two dataspace selections - and projects that intersection into a third selection. This can be useful - for VOL developers to implement chunked or virtual datasets. + (ADB - 2020/03/24, TRILAB-24) - (NAF - 2019/11/13, ID-148) + - Both build systems use same set of warnings flags - - Add new public function H5VLget_file_type. + GNU C, C++ and gfortran warnings flags were moved to files in a config + sub-folder named gnu-warnings. Flags that only are available for a specific + version of the compiler are in files named with that version. + Clang C warnings flags were moved to files in a config sub-folder + named clang-warnings. + Intel C, Fortran warnings flags were moved to files in a config sub-folder + named intel-warnings. - This function returns a datatype equivalent to the supplied datatype but - with the location set to be in the file. This datatype can then be used - with H5Tconvert to convert data between file and in-memory representation. - This funcition is intended for use only by VOL connector developers. + There are flags in named "error-xxx" files with warnings that may + be promoted to errors. Some source files may still need fixes. - (NAF - 2019/11/08, ID-127) + There are also pairs of files named "developer-xxx" and "no-developer-xxx" + that are chosen by the CMake option:HDF5_ENABLE_DEV_WARNINGS or the + configure option:--enable-developer-warnings. - - Add S3 and HDFS VFDs to HDF5 maintenance + In addition, CMake no longer applies these warnings for examples. - Fix windows requirements and java tests. Windows requires CMake 3.13. - Install openssl library (with dev files); - from "Shining Light Productions". msi package preferred. + (ADB - 2020/03/24, TRILAB-192) - PATH should have been updated with the installation dir. - set ENV variable OPENSSL_ROOT_DIR to the installation dir. - set ENV variable OPENSSL_CONF to the cfg file, likely %OPENSSL_ROOT_DIR%\bin\openssl.cfg - Install libcurl library (with dev files); - download the latest released version using git: https://github.com/curl/curl.git + - Added test script for file size compare - Open a Visual Studio Command prompt - change to the libcurl root folder - run the "buildconf.bat" batch file - change to the winbuild directory - nmake /f Makefile.vc mode=dll MACHINE=x64 - copy libcurl-vc-x64-release-dll-ipv6-sspi-winssl dir to C:\curl (installation dir) - set ENV variable CURL_ROOT to C:\curl (installation dir) - update PATH ENV variable to %CURL_ROOT%\bin (installation bin dir). - the aws credentials file should be in %USERPROFILE%\.aws folder - set the ENV variable "HDF5_ROS3_TEST_BUCKET_URL=https://s3.us-east-2.amazonaws.com/hdf5ros3" + If CMake minimum version is at least 3.14, the fileCompareTest.cmake + script will compare file sizes. - (ADB - 2019/09/12, HDFFV-10854) + (ADB - 2020/02/24, HDFFV-11036) - - Added new chunk query functions + - Update CMake minimum version to 3.12 - The following public functions were added to discover information about - the chunks in an HDF5 file. - herr_t H5Dget_num_chunks(dset_id, fspace_id, *nchunks) - herr_t H5Dget_chunk_info_by_coord(dset_id, *coord, *filter_mask, *addr, *size) - herr_t H5Dget_chunk_info(dset_id, fspace_id, index, *coord, *filter_mask, *addr, *size) + Updated CMake minimum version to 3.12 and added version checks + for Windows features. - (BMR - 2019/06/11, HDFFV-10677) + (ADB - 2020/02/05, TRILABS-142) - - Improved the performance of virtual dataset I/O + - Fixed CMake include properties for Fortran libraries - Refactored the internal dataspace routines used by the virtual dataset - code to improve performance, especially when one of the selections - involved is very long and non-contiguous. + Corrected the library properties for Fortran to use the + correct path for the Fortran module files. - (NAF - 2019/05/31, HDFFV-10693) + (ADB - 2020/02/04, HDFFV-11012) - - Added the ability to open files with UTF-8 file names on Windows. + - Added common warnings files for gnu and intel - The POSIX open(2) API call on Windows is limited to ASCII - file names. The library has been updated to convert incoming file - names to UTF-16 (via MultiByteToWideChar(CP_UTF8, ...) and use - _wopen() instead. + Added warnings files to use one common set of flags + during configure for both autotools and CMake build + systems. The initial implementation only affects a + general set of flags for gnu and intel compilers. - (DER - 2019/03/15, HDFFV-2714, HDFFV-3914, HDFFV-3895, HDFFV-8237, HDFFV-10413, HDFFV-10691) + (ADB - 2020/01/17) - - Add new API H5M for map objects. Currently not supported by native - library, can be supported by VOL connectors. + - Added new options to CMake for control of testing - (NAF - 2019/03/01) + Added CMake options (default ON); + HDF5_TEST_SERIAL AND/OR HDF5_TEST_PARALLEL + combined with: + HDF5_TEST_TOOLS + HDF5_TEST_EXAMPLES + HDF5_TEST_SWMR + HDF5_TEST_FORTRAN + HDF5_TEST_CPP + HDF5_TEST_JAVA - - Add new H5R_ref_t type for object, dataset region and _attribute_ - references. This new type will deprecate the current hobj_ref_t - and hdset_reg_ref_t types for references. Added H5T_REF datatype - to read and write new reference types. As opposed to previous - reference types, reference creation no longer modifies existing - files. New reference types also now support references to external - files. + (ADB - 2020/01/15, HDFFV-11001) - (JS - 2019/10/08) + - Added Clang sanitizers to CMake for analyzer support if compiler is clang. - - Remove H5I_REFERENCE from the library + Added CMake code and files to execute the Clang sanitizers if + HDF5_ENABLE_SANITIZERS is enabled and the USE_SANITIZER option + is set to one of the following: + Address + Memory + MemoryWithOrigins + Undefined + Thread + Leak + 'Address;Undefined' - This ID class was never used by the library and has been removed. + (ADB - 2019/12/12, TRILAB-135) - (DER - 2018/12/08, HDFFV-10252) + - Update CMake for VS2019 support - - Allow pre-generated H5Tinit.c and H5make_libsettings.c to be used. + CMake added support for VS2019 in version 3.15. Changes to the CMake + generator setting required changes to scripts. Also updated version + references in CMake files as necessary. - Rather than always running H5detect and generating H5Tinit.c and - H5make_libsettings.c, supply a location for those files. + (ADB - 2019/11/18, HDFFV-10962) - (ADB - 2018/09/18, HDFFV-10332) + - Update CMake options to match new autotools options - - Fix shutdown failure when using H5VLregister_connector_by_name/value + Add configure options (autotools - CMake): + enable-asserts HDF5_ENABLE_ASSERTS + enable-symbols HDF5_ENABLE_SYMBOLS + enable-profiling HDF5_ENABLE_PROFILING + enable-optimization HDF5_ENABLE_OPTIMIZATION + In addition NDEBUG is no longer forced defined and relies on the CMake + process. - When using H5VLregister_connector_by_name/value to dynamically load a - VOL connector plugin, the library can experience segmentation faults - when the library is closed. This is due to the library unloading - the plugin interface before the virtual object layer. Then, when the - VOL shutdown occurs, it will attempt to close the VOL connector, - however this will fail since the plugin will already have been unloaded. + (ADB - 2019/10/07, HDFFV-100901, HDFFV-10637, TRILAB-97) - (DER - 2020/03/18, HDFFV-11057) + Library: + -------- + - H5Gcreate1() now rejects size_hint parameters larger than UINT32_MAX - Parallel Library: - ----------------- - - Changed the default behavior in parallel when reading the same dataset in its entirely - (i.e. H5S_ALL dataset selection) which is being read by all the processes collectively. - The dataset mush be contiguous, less than 2GB, and of an atomic datatype. - The new behavior is the HDF5 library will use an MPI_Bcast to pass the data read from - the disk by the root process to the remain processes in the MPI communicator associated - with the HDF5 file. + The size_hint value is ultimately stored in a uint32_t struct field, + so specifying a value larger than this on a 64-bit machine can cause + undefined behavior including crashing the system. - (MSB - 2019/01/02, HDFFV-10652) + The documentation for this API call was also incorrect, stating that + passing a negative value would cause the library to use a default + value. Instead, passing a "negative" value actually passes a very large + value, which is probably not what the user intends and can cause + crashes on 64-bit systems. - Fortran Library: - ---------------- - - Added new Fortran derived type, c_h5o_info_t, which is interoperable with - C's h5o_info_t. This is needed for callback functions which - pass C's h5o_info_t data type definition. + The Doxygen documentation has been updated and passing values larger + than UINT32_MAX for size_hint will now produce a normal HDF5 error. - (MSB, 2019/01/08, HDFFV-10443) + (DER - 2021/04/29, HDFFV-11241) - - Added new Fortran API, H5gmtime, which converts (C) 'time_t' structure - to Fortran DATE AND TIME storage format. - (MSB, 2019/01/08, HDFFV-10443) + - H5Pset_fapl_log() no longer crashes when passed an invalid fapl ID - - Added new Fortran 'fields' optional parameter to: h5ovisit_f, h5oget_info_by_name_f, - h5oget_info, h5oget_info_by_idx and h5ovisit_by_name_f. + When passed an invalid fapl ID, H5Pset_fapl_log() would usually + segfault when attempting to free an uninitialized pointer in the error + handling code. This behavior is more common in release builds or + when the memory sanitization checks were not selected as a build + option. - (MSB, 2019/01/08, HDFFV-10443) + The pointer is now correctly initialized and the API call now + produces a normal HDF5 error when fed an invalid fapl ID. - C++ Library: - ------------ - - Added new wrappers for H5Pset/get_create_intermediate_group() - LinkCreatPropList::setCreateIntermediateGroup() - LinkCreatPropList::getCreateIntermediateGroup() + (DER - 2021/04/28, HDFFV-11240) - (BMR - 2019/04/22, HDFFV-10622) + - Fixes a segfault when H5Pset_mdc_log_options() is called multiple times - - Added new wrapper for H5Ovisit2() - H5Object::visit() + The call incorrectly attempts to free an internal copy of the previous + log location string, which causes a segfault. This only happens + when the call is invoked multiple times on the same property list. + On the first call to a given fapl, the log location is set to NULL so + the segfault does not occur. - (BMR - 2019/02/14, HDFFV-10532) + The string is now handled properly and the segfault no longer occurs. + (DER - 2021/04/27, HDFFV-11239) - Java Library: - ---------------- - - Added ability to test java library with VOLs. + - HSYS_GOTO_ERROR now emits the results of GetLastError() on Windows - Created new CMake script that combines the java and vol test scripts. + HSYS_GOTO_ERROR is an internal macro that is used to produce error + messages when system calls fail. These strings include errno and the + the associated strerror() value, which are not particularly useful + when a Win32 API call fails. - (ADB - 2020/02/03, HDFFV-10996) + On Windows, this macro has been updated to include the result of + GetLastError(). When a system call fails on Windows, usually only + one of errno and GetLastError() will be useful, however we emit both + for the user to parse. The Windows error message is not emitted as + it would be awkward to free the FormatMessage() buffer given the + existing HDF5 error framework. Users will have to look up the error + codes in MSDN. - - Tests fail for non-English locale. + The format string on Windows has been changed from: - In the JUnit tests with a non-English locale, only the part before - the decimal comma is replaced by XXXX and this leads to a comparison - error. Changed the regex for the Time substitution. + "%s, errno = %d, error message = '%s'" - (ADB - 2020/01/09, HDFFV-10995) + to: - - Fix a failure in JUnit-TestH5P on 32-bit architectures + "%s, errno = %d, error message = '%s', Win32 GetLastError() = %"PRIu32"" - (JTH - 2019/04/30) + for those inclined to parse it for error values. - - Duplicate the data read/write functions of Datasets for Attributes. + (DER - 2021/03/21) - Region references could not be displayed for attributes as they could - for datasets. Datasets had overloaded read and write functions for different - datatypes that were not available for attributes. After adding similar - functions, attribute region references work normally. + - File locking now works on Windows - (ADB - 2018/12/12, HDFVIEW-4) + Since version 1.10.0, the HDF5 library has used a file locking scheme + to help enforce one reader at a time accessing an HDF5 file, which can + be helpful when setting up readers and writers to use the single- + writer/multiple-readers (SWMR) access pattern. - - Removed H5I_REFERENCE from the Java wrappers + In the past, this was only functional on POSIX systems where flock() or + fcntl() were present. Windows used a no-op stub that always succeeded. - This ID class was never used by the library and has been removed - from the Java wrappers. + HDF5 now uses LockFileEx() and UnlockFileEx() to lock the file using the + same scheme as POSIX systems. We lock the entire file when we set up the + locks (by passing DWORDMAX as both size parameters to LockFileEx()). - (DER - 2018/12/08, HDFFV-10252) + (DER - 2021/03/19, HDFFV-10191) + - H5Epush_ret() now requires a trailing semicolon - Tools: - ------ - - h5repack was fixed to repack the reference attributes properly. - The code line that checks if the update of reference inside a compound - datatype is misplaced outside the code block loop that carries out the - check. In consequence, the next attribute that is not the reference - type was repacked again as the reference type and caused the failure of - repacking. The fix is to move the corresponding code line to the correct - code block. + H5Epush_ret() is a function-like macro that has been changed to + contain a `do {} while(0)` loop. Consequently, a trailing semicolon + is now required to end the `while` statement. Previously, a trailing + semi would work, but was not mandatory. This change was made to allow + clang-format to correctly format the source code. - (KY -2020/02/07, HDFFV-11014) + (SAM - 2021/03/03) - - h5diff was updated to use the new reference APIs. + - Improved performance of H5Sget_select_elem_pointlist - h5diff uses the new reference APIs to compare references. - Attribute references can also be compared. + Modified library to cache the point after the last block of points + retrieved by H5Sget_select_elem_pointlist, so a subsequent call to the + same function to retrieve the next block of points from the list can + proceed immediately without needing to iterate over the point list. - (ADB - 2019/12/19, HDFFV-10980) + (NAF - 2021/01/19) - - h5dump and h5ls were updated to use the new reference APIs. + - Replaced H5E_ATOM with H5E_ID in H5Epubgen.h - The tools library now use the new reference APIs to inspect a - file. Also the DDL spec was updated to reflect the format - changes produced with the new APIs. The export API and support - functions in the JNI were updated to match. + The term "atom" is archaic and not in line with current HDF5 library + terminology, which uses "ID" instead. "Atom" has mostly been purged + from the library internals and this change removes H5E_ATOM from + the H5Epubgen.h (exposed via H5Epublic.h) and replaces it with + H5E_ID. - (ADB - 2019/12/06, HDFFV-10876 and HDFFV-10877) + (DER - 2020/11/24, HDFFV-11190) - - h5repack was fixed to repack datasets with external storage - to other types of storage. + - Add a new public function H5Ssel_iter_reset - New test added to repack files and verify the correct data using h5diff. + This function resets a dataspace selection iterator back to an + initial state so that it may be used for iteration once more. + This can be useful when needing to iterate over a selection + multiple times without having to repeatedly create/destroy + a selection iterator for that dataspace selection. - (JS - 2019/09/25, HDFFV-10408) - (ADB - 2019/10/02, HDFFV-10918) + (JTH - 2020/09/18) - - h5dump was fixed for 128-bit floats, but was missing a test. + - Remove HDFS VFD stubs - New test greps for the first 15 numbers of the 128-bit value. + The original implementation of the HDFS VFD included non-functional + versions of the following public API calls when the HDFS VFD is + not built as a part of the HDF5 library: - (ADB - 2019/06/23, HDFFV-9407) + * H5FD_hdfs_init() + * H5Pget_fapl_hdfs() + * H5Pset_fapl_hdfs() + They will remain present in HDF5 1.10 and HDF5 1.12 releases + for binary compatibility purposes but have been removed as of 1.14.0. - High-Level APIs: - --------------- - - + Note that this has nothing to do with the real HDFS VFD API calls + that are fully functional when the HDFS VFD is configured and built. - C Packet Table API - ------------------ - - + We simply changed: - Internal header file - -------------------- - - + #ifdef LIBHDFS + <real API call> + #else + <useless stub> + #endif - Documentation - ------------- - - + to: -Support for new platforms, languages and compilers. -======================================= - - + #ifdef LIBHDFS + <real API call> + #endif -Bug Fixes since HDF5-1.10.3 release -================================== + Which is how the other optional VFDs are handled. - Library - ------- - - Fixed issues CVE-2018-13870 and CVE-2018-13869 + (DER - 2020/08/27) - When a buffer overflow occurred because a name length was corrupted - and became very large, h5dump crashed on memory access violation. + - Add Mirror VFD - A check for reading pass the end of the buffer was added to multiple - locations to prevent the crashes and h5dump now simply fails with an - error message when this error condition occurs. + Use TCP/IP sockets to perform write-only (W/O) file I/O on a remote + machine. Must be used in conjunction with the Splitter VFD. - (BMR - 2020/7/22, HDFFV-11120 and HDFFV-11121) + (JOS - 2020/03/13, TBD) - - Fixed the segmentation fault when reading attributes with multiple threads + - Add Splitter VFD - It was reported that the reading of attributes with variable length string - datatype will crash with segmentation fault particularly when the number of - threads is high (>16 threads). The problem was due to the file pointer that - was set in the variable length string datatype for the attribute. That file - pointer was already closed when the attribute was accessed. + Maintain separate R/W and W/O channels for "concurrent" file writes + to two files using a single HDF5 file handle. - The problem was fixed by setting the file pointer to the current opened file pointer - when the attribute was accessed. Similar patch up was done before when reading - dataset with variable length string datatype. + (JOS - 2020/03/13, TBD) - (VC - 2020/07/13, HDFFV-11080) + - Refactored public exposure of haddr_t type in favor of "object tokens" - - Fixed CVE-2020-10810 + To better accommodate HDF5 VOL connectors where "object addresses in a file" + may not make much sense, the following changes were made to the library: + + * Introduced new H5O_token_t "object token" type, which represents a + unique and permanent identifier for referencing an HDF5 object within + a container; these "object tokens" are meant to replace object addresses. + Along with the new type, a new H5Oopen_by_token API call was introduced + to open an object by a token, similar to how object addresses were + previously used with H5Oopen_by_addr. + + * Introduced new H5Lget_info2, H5Lget_info_by_idx2, H5Literate2, H5Literate_by_name2, + H5Lvisit2 and H5Lvisit_by_name2 API calls, along with their associated H5L_info2_t + struct and H5L_iterate2_t callback function, which work with the newly-introduced + object tokens, instead of object addresses. The original functions have been + renamed to version 1 functions and are deprecated in favor of the new version 2 + functions. The H5L_info_t and H5L_iterate_t types have been renamed to version 1 + types and are now deprecated in favor of their version 2 counterparts. For each of + the functions and types, compatibility macros take place of the original symbols. + + * Introduced new H5Oget_info3, H5Oget_info_by_name3, H5Oget_info_by_idx3, + H5Ovisit3 and H5Ovisit_by_name3 API calls, along with their associated H5O_info2_t + struct and H5O_iterate2_t callback function, which work with the newly-introduced + object tokens, instead of object addresses. The version 2 functions are now + deprecated in favor of the version 3 functions. The H5O_info_t and H5O_iterate_t + types have been renamed to version 1 types and are now deprecated in favor of their + version 2 counterparts. For each, compatibility macros take place of the original + symbols. + + * Introduced new H5Oget_native_info, H5Oget_native_info_by_name and + H5Oget_native_info_by_idx API calls, along with their associated H5O_native_info_t + struct, which are used to retrieve the native HDF5 file format-specific information + about an object. This information (such as object header info and B-tree/heap info) + has been removed from the new H5O_info2_t struct so that the more generic + H5Oget_info(_by_name/_by_idx)3 routines will not try to retrieve it for non-native + VOL connectors. + + * Added new H5Otoken_cmp, H5Otoken_to_str and H5Otoken_from_str routines to compare + two object tokens, convert an object token into a nicely-readable string format and + to convert an object token string back into a real object token, respectively. + + (DER, QAK, JTH - 2020/01/16) - The tool h5clear produced a segfault during an error recovery in - the superblock decoding. An internal pointer was reset to prevent - further accessing when it is not assigned with a value. + - Add new public function H5Sselect_adjust. - (BMR - 2020/6/29, HDFFV-11053) + This function shifts a dataspace selection by a specified logical offset + within the dataspace extent. This can be useful for VOL developers to + implement chunked datasets. - - Fixed CVE-2018-17435 + (NAF - 2019/11/18) - The tool h52gif produced a segfault when the size of an attribute - message was corrupted and caused a buffer overflow. + - Add new public function H5Sselect_project_intersection. - The problem was fixed by verifying the attribute message's size - against the buffer size before accessing the buffer. h52gif was - also fixed to display the failure instead of silently exiting - after the segfault was eliminated. + This function computes the intersection between two dataspace selections + and projects that intersection into a third selection. This can be useful + for VOL developers to implement chunked or virtual datasets. - (BMR - 2020/6/19, HDFFV-10591) + (NAF - 2019/11/13, ID-148) - - Improved peformance when creating a large number of small datasets by - retrieving default property values from the API context instead of doing - skip list searches. + - Add new public function H5VLget_file_type. - (CJH - 2019/12/10, HDFFV-10658) + This function returns a datatype equivalent to the supplied datatype but + with the location set to be in the file. This datatype can then be used + with H5Tconvert to convert data between file and in-memory representation. + This function is intended for use only by VOL connector developers. - - Fixed user-created data access properties not existing in the property list - returned by H5Dget_access_plist. Thanks to Steven Varga for submitting a - reproducer and a patch. + (NAF - 2019/11/08, ID-127) - (CJH - 2019/12/9, HDFFV-10934) - - Fixed an assertion failure in the parallel library when collectively - filling chunks. As it is required that chunks be written in - monotonically non-decreasing order of offset in the file, this assertion - was being triggered when the list of chunk file space allocations being - passed to the collective chunk filling routine was not sorted according - to this particular requirement. + Parallel Library: + ----------------- + - - The addition of a sort of the out of order chunks trades a bit of - performance for the elimination of this assertion and of any complaints - from MPI implementations about the file offsets used being out of order. - (JTH - 2019/10/07) + Fortran Library: + ---------------- + - Add wrappers for H5Pset/get_file_locking() API calls - - Fixed the iteration error in test_versionbounds() in test/dtypes.c + h5pget_file_locking_f() + h5pset_file_locking_f() - The test was supposed to loop through all valid combinations of - low and high bounds in the array versions[], but they were set to - H5F_LIBVER_EARLIEST always without changing. + See the configure option discussion for HDFFV-11092 (above) for more + information on the file locking feature and how it's controlled. - The problem was fixed by indexing low and high into the array versions[]. + (DER - 2020/07/30, HDFFV-11092) - (VC - 2019/09/30) + C++ Library: + ------------ + - Add wrappers for H5Pset/get_file_locking() API calls - - Fixed the slowness of regular hyperslab selection in a chunked dataset + FileAccPropList::setFileLocking() + FileAccPropList::getFileLocking() - It was reported that the selection of every 10th element from a 20G - chunked dataset was extremely slow and sometimes could hang the system. - The problem was due to the iteration and the building of the span tree - for all the selected elements in file space. + See the configure option discussion for HDFFV-11092 (above) for more + information on the file locking feature and how it's controlled. - As the selected elements are going to a 1-d contiguous single block - memory space, the problem was fixed by building regular hyperslab selections - in memory space for the selected elements in file space. + (DER - 2020/07/30, HDFFV-11092) - (VC - 2019/09/26, HDFFV-10585) - - Fixed a bug caused by bad tag value when condensing object header - messages + Java Library: + ------------- + - Replaced HDF5AtomException with HDF5IdException - There was an assertion failure when moving meessages from running a - user test program with library release hdf5.1.10.4. It was because - the tag value (object header's address) was not set up when entering - the library routine H5O__chunk_update_idx(), which will eventually - verifies the metadata tag value when protecting the object header. + Since H5E_ATOM changed to H5E_ID in the C library, the Java exception + that wraps the error category was also renamed. Its functionality + remains unchanged aside from the name. - The problem was fixed by replacing FUNC_ENTER_PACKAGE in H5O__chunk_update_idx() - with FUNC_ENTER_PACKAGE_TAG(oh->cache_info.addr) to set up the metadata tag. + (See also the HDFFV-11190 note in the C library section) - (VC - 2019/08/23, HDFFV-10873) + (DER - 2020/11/24, HDFFV-11190) - - Fixed the test failure from test_metadata_read_retry_info() in - test/swmr.c + - Added new H5S functions. - The test failure is due to the incorrect number of bins returned for - retry info (info.nbins). The # of bins expected for 101 read attempts - is 3 instead of 2. The routine H5F_set_retries() in src/H5Fint.c - calculates the # of bins by first obtaining the log10 value for - (read attempts - 1). For PGI/19, the log10 value for 100 read attempts - is 1.9999999999999998 instead of 2.00000. When casting the log10 value - to unsigned later on, the decimal part is chopped off causing the test - failure. + H5Sselect_copy, H5Sselect_shape_same, H5Sselect_adjust, + H5Sselect_intersect_block, H5Sselect_project_intersection, + H5Scombine_hyperslab, H5Smodify_select, H5Scombine_select + wrapper functions added. - This was fixed by obtaining the rounded integer value (HDceil) for the - log10 value of read attempts first before casting the result to unsigned. + (ADB - 2020/10/27, HDFFV-10868) - (VC - 2019/8/14, HDFFV-10813) + - Add wrappers for H5Pset/get_file_locking() API calls - - Fixed an issue where creating a file with non-default file space info - together with library high bound setting to H5F_LIBVER_V18. + H5Pset_file_locking() + H5Pget_use_file_locking() + H5Pget_ignore_disabled_file_locking() - When setting non-default file space info in fcpl via - H5Pset_file_space_strategy() and then creating a file with - both high and low library bounds set to - H5F_LIBVER_V18 in fapl, the library succeeds in creating the file. - File creation should fail because the feature of setting non-default - file space info does not exist in library release 1.8 or earlier. + Unlike the C++ and Fortran wrappers, there are separate getters for the + two file locking settings, each of which returns a boolean value. - This was fixed by setting and checking the proper version in the - file space info message based on the library low and high bounds - when creating and opening the HDF5 file. + See the configure option discussion for HDFFV-11092 (above) for more + information on the file locking feature and how it's controlled. - (VC - 2019/6/25, HDFFV-10808) + (DER - 2020/07/30, HDFFV-11092) - - When iterating over an old-style group (i.e., when not using the latest - file format) of size 0, a NULL pointer representing the empty links - table would be sent to qsort(3) for sorting, which is undefined behavior. + - Added ability to test java library with VOLs. - Iterating over an empty group is explicitly tested in the links test. - This has not caused any failures to date and was flagged by gcc's - -fsanitize=undefined. + Created a new CMake script that combines the java and vol test scripts. - The library no longer attempts to sort an empty array. + (ADB - 2020/02/03, HDFFV-10996) - (DER - 2019/06/18, HDFFV-10829) + - Tests fail for non-English locales. - - Fixed an issue where copying a version 1.8 dataset between files using - H5Ocopy fails due to an incompatible fill version + In the JUnit tests with a non-English locale, only the part before + the decimal comma is replaced by XXXX and this leads to a comparison + error. Changed the regex for the Time substitution. - When using the HDF5 1.10.x H5Ocopy() API call to copy a version 1.8 - dataset to a file created with both high and low library bounds set to - H5F_LIBVER_V18, the H5Ocopy() call will fail with the error stack indicating - that the fill value version is out of bounds. + (ADB - 2020/01/09, HDFFV-10995) - This was fixed by changing the fill value message version to H5O_FILL_VERSION_3 - (from H5O_FILL_VERSION_2) for H5F_LIBVER_V18. - (VC - 2019/6/14, HDFFV-10800) + Tools: + ------ + - h5repack added help text for user-defined filters. - - Some oversights in the index iterating area of the library caused - a callback function to continue iterating even though it's supposed - to stop. + Added help text line that states the valid values of the filter flag + for user-defined filters; + filter_flag: 1 is OPTIONAL or 0 is MANDATORY - Added the returned value check to the for loop's conditions in - H5EA_iterate(), H5FA_iterate(), and H5D__none_idx_iterate(). The - iteration now stops when it should. + (ADB - 2021/01/14, HDFFV-11099) - (BMR - 2019/06/11, HDFFV-10661) + - Added h5delete tool - - Fixed a bug that would cause an error or cause fill values to be - incorrectly read from a chunked dataset using the "single chunk" index if - the data was held in cache and there was no data on disk. + Deleting HDF5 storage when using the VOL can be tricky when the VOL + does not create files. The h5delete tool is a simple wrapper around + the H5Fdelete() API call that uses the VOL specified in the + HDF5_VOL_CONNECTOR environment variable to delete a "file". If + the call to H5Fdelete() fails, the tool will attempt to use + the POSIX remove(3) call to remove the file. - (NAF - 2019/03/06) + Note that the HDF5 library does currently have support for + H5Fdelete() in the native VOL connector. - - Fixed a bug that could cause an error or cause fill values to be - incorrectly read from a dataset that was written to using H5Dwrite_chunk - if the dataset was not closed after writing. + (DER - 2020/12/16) - (NAF - 2019/03/06, HDFFV-10716) + - h5repack added options to control how external links are handled. - - Fixed memory leak in scale offset filter + Currently h5repack preserves external links and cannot copy and merge + data from the external files. Two options, merge and prune, were added to + control how to merge data from an external link into the resulting file. + --merge Follow external soft link recursively and merge data. + --prune Do not follow external soft links and remove link. + --merge --prune Follow external link, merge data and remove dangling link. - In a special case where the MinBits is the same as the number of bits in - the datatype's precision, the filter's data buffer was not freed, causing - the memory usage to grow. In general the buffer was freed correctly. The - Minbits are the minimal number of bits to store the data values. Please - see the reference manual for H5Pset_scaleoffset for the detail. + (ADB - 2020/08/05, HDFFV-9984) - (RL - 2019/3/4, HDFFV-10705) + - h5repack was fixed to repack the reference attributes properly. + The code line that checks if the update of reference inside a compound + datatype is misplaced outside the code block loop that carries out the + check. In consequence, the next attribute that is not the reference + type was repacked again as the reference type and caused the failure of + repacking. The fix is to move the corresponding code line to the correct + code block. - - Fix hangs with collective metadata reads during chunked dataset I/O + (KY -2020/02/07, HDFFV-11014) - In the parallel library, it was discovered that when a particular - sequence of operations following a pattern of: + - h5diff was updated to use the new reference APIs. - "write to chunked dataset" -> "flush file" -> "read from dataset" + h5diff uses the new reference APIs to compare references. + Attribute references can also be compared. - occurred with collective metadata reads enabled, hangs could be - observed due to certain MPI ranks not participating in the collective - metadata reads. + (ADB - 2019/12/19, HDFFV-10980) - To fix the issue, collective metadata reads are now disabled during - chunked dataset raw data I/O. + - h5dump and h5ls were updated to use the new reference APIs. - (JTH - 2019/02/11, HDFFV-10563, HDFFV-10688) + The tools library now use the new reference APIs to inspect a + file. Also the DDL spec was updated to reflect the format + changes produced with the new APIs. The export API and support + functions in the JNI were updated to match. - - Performance issue when closing an object + (ADB - 2019/12/06, HDFFV-10876 and HDFFV-10877) - The slow down is due to the search of the "tag_list" to find - out the "corked" status of an object and "uncork" it if so. - Improve porformance by skipping the search of the "tag_list" - if there are no "corked" objects when closing an object. + High-Level APIs: + ---------------- + - - (VC - 2019/2/6) + C Packet Table API: + ------------------- + - - - Fixed a potential invalid memory access and failure that could occur when - decoding an unknown object header message (from a future version of the - library). + Internal header file: + --------------------- + - - (NAF - 2019/01/07) + Documentation: + -------------- + - - - Deleting attributes in dense storage +Support for new platforms, languages and compilers +================================================== + - - The library aborts with "infinite loop closing library" after - attributes in dense storage are created and then deleted. +Bug Fixes since HDF5-1.12.0 release +=================================== + Library + ------- + - Fixed CVE-2018-14460 - When deleting the attribute nodes from the name index v2 B-tree, - if an attribute is found in the intermediate B-tree nodes, - which may be merged/redistributed in the process, we need to - free the dynamically allocated spaces for the intermediate - decoded attribute. + The tool h5repack produced a segfault when the rank in dataspace + message was corrupted, causing invalid read while decoding the + dimension sizes. - (VC - 2018/12/26, HDFFV-10659) + The problem was fixed by ensuring that decoding the dimension sizes + and max values will not go beyong the end of the buffer. - - Allow H5detect and H5make_libsettings to take a file as an argument. + (BMR - 2021/05/12, HDFFV-11223) - Rather than only writing to stdout, add a command argument to name - the file that H5detect and H5make_libsettings will use for output. - Without an argument, stdout is still used, so backwards compatibility - is maintained. + - Fixed CVE-2018-11206 - (ADB - 2018/09/05, HDFFV-9059) + The tool h5dump produced a segfault when the size of a fill value + message was corrupted and caused a buffer overflow. - - A bug was discovered in the parallel library where an application - would hang if a collective read/write of a chunked dataset occurred - when collective metadata reads were enabled and some of the ranks - had no selection in the dataset's dataspace. The ranks which had no - selection in the dataset's dataspace called H5D__chunk_addrmap() to - retrieve the lowest chunk address in the dataset. This is because we - require reads/writes to be performed in strictly non-decreasing order - of chunk address in the file. + The problem was fixed by verifying the fill value's size + against the buffer size before attempting to access the buffer. - When the chunk index used was a version 1 or 2 B-tree, these - non-participating ranks would issue a collective MPI_Bcast() call - that the participating ranks would not issue, causing the hang. Since - the non-participating ranks are not actually reading/writing anything, - the H5D__chunk_addrmap() call can be safely removed and the address used - for the read/write can be set to an arbitrary number (0 was chosen). + (BMR - 2021/03/15, HDFFV-10480) - (JTH - 2018/08/25, HDFFV-10501) + - Fixed CVE-2018-14033 (same issue as CVE-2020-10811) - - fcntl(2)-based file locking incorrectly passed the lock argument struct - instead of a pointer to the struct, causing errors on systems where - flock(2) is not available. + The tool h5dump produced a segfault when the storage size message + was corrupted and caused a buffer overflow. - File locking is used when files are opened to enforce SWMR semantics. A - lock operation takes place on all file opens unless the - HDF5_USE_FILE_LOCKING environment variable is set to the string "FALSE". - flock(2) is preferentially used, with fcntl(2) locks as a backup if - flock(2) is unavailable on a system (if neither is available, the lock - operation fails). On these systems, the file lock will often fail, which - causes HDF5 to not open the file and report an error. + The problem was fixed by verifying the storage size against the + buffer size before attempting to access the buffer. - This bug only affects POSIX systems. Win32 builds on Windows use a no-op - locking call which always succeeds. Systems which exhibit this bug will - have H5_HAVE_FCNTL defined but not H5_HAVE_FLOCK in the configure output. + (BMR - 2021/03/15, HDFFV-11159/HDFFV-11049) - This bug affects HDF5 1.10.0 through 1.10.5. + - Remove underscores on header file guards - fcntl(2)-based file locking now correctly passes the struct pointer. + Header file guards used a variety of underscores at the beginning of the define. - (DER - 2019/08/27, HDFFV-10892) + Removed all leading (some trailing) underscores from header file guards. - - Torn pread/pwrite I/O would result in read and write corruption. + (ADB - 2021/03/03, #361) - In the sec2, log, and core (with backing store) virtual file drivers - (VFDs), the read and write calls incorrectly reset the offset parameter - on torn pread and pwrite operations (i.e., I/O operations which fail to - be written atomically by the OS). For this bug to occur, pread/pwrite - have to be configured (this is the default if they are present on the - system) and the pread/pwrite operation has to fail to transfer all - the bytes, resulting in a multiple pread/pwrite calls. + - Fixed issue with MPI communicator and info object not being + copied into new FAPL retrieved from H5F_get_access_plist - This feature was initially enabled in HDF5 1.10.5 so the bug is - limited to that version. + Added logic to copy the MPI communicator and info object into + the output FAPL. MPI communicator is retrieved from the VFD, while + the MPI info object is retrieved from the file's original FAPL. - (DER - 2019/12/09, HDFFV-10945) + (JTH - 2021/02/15, HDFFV-11109) - - H5Sset_extent_none() sets the dataspace class to H5S_NO_CLASS which - causes asserts/errors when passed to other dataspace API calls. + - Fixed problems with vlens and refs inside compound using + H5VLget_file_type() - H5S_NO_CLASS is an internal class value that should not have been - exposed via a public API call. + Modified library to properly ref count H5VL_object_t structs and only + consider file vlen and reference types to be equal if their files are + the same. - In debug builds of the library, this can cause asserts to trip. In - non-debug builds, it will produce normal library errors. + (NAF - 2021/01/22) - The new library behavior is for H5Sset_extent_none() to convert - the dataspace into one of type H5S_NULL, which is better handled - by the library and easier for developers to reason about. + - Creation of dataset with optional filter - (DER - 2020/07/27, HDFFV-11027) + When the combination of type, space, etc doesn't work for filter + and the filter is optional, it was supposed to be skipped but it was + not skipped and the creation failed. + Allowed the creation of the dataset in such a situation. - Java Library: - ---------------- - - JNI native library dependencies + (BMR - 2020/08/13, HDFFV-10933) - The build for the hdf5_java native library used the wrong - hdf5 target library for CMake builds. Correcting the hdf5_java - library to build with the shared hdf5 library required testing - paths to change also. + - Explicitly declared dlopen to use RTLD_LOCAL - (ADB - 2018/08/31, HDFFV-10568) - - Java iterator callbacks + dlopen documentation states that if neither RTLD_GLOBAL nor + RTLD_LOCAL are specified, then the default behavior is unspecified. + The default on linux is usually RTLD_LOCAL while macos will default + to RTLD_GLOBAL. - Change global callback object to a small stack structure in order - to fix a runtime crash. This crash was discovered when iterating - through a file with nested group members. The global variable - visit_callback is overwritten when recursion starts. When recursion - completes, visit_callback will be pointing to the wrong callback method. + (ADB - 2020/08/12, HDFFV-11127) - (ADB - 2018/08/15, HDFFV-10536) + - H5Sset_extent_none() sets the dataspace class to H5S_NO_CLASS which + causes asserts/errors when passed to other dataspace API calls. - - Java HDFLibraryException class + H5S_NO_CLASS is an internal class value that should not have been + exposed via a public API call. - Change parent class from Exception to RuntimeException. + In debug builds of the library, this can cause assert() function to + trip. In non-debug builds, it will produce normal library errors. - (ADB - 2018/07/30, HDFFV-10534) + The new library behavior is for H5Sset_extent_none() to convert + the dataspace into one of type H5S_NULL, which is better handled + by the library and easier for developers to reason about. - - JNI Read and Write + (DER - 2020/07/27, HDFFV-11027) - Refactored variable-length functions, H5DreadVL and H5AreadVL, - to correct dataset and attribute reads. New write functions, - H5DwriteVL and H5AwriteVL, are under construction. + - Fixed issues CVE-2018-13870 and CVE-2018-13869 - (ADB - 2018/06/02, HDFFV-10519) + When a buffer overflow occurred because a name length was corrupted + and became very large, h5dump crashed on memory access violation. - Configuration - ------------- - - Correct option for default API version + A check for reading pass the end of the buffer was added to multiple + locations to prevent the crashes and h5dump now simply fails with an + error message when this error condition occurs. - CMake options for default API version are not mutually exclusive. - Change the multiple BOOL options to a single STRING option with the - strings; v16, v18, v110, v112. + (BMR - 2020/07/22, HDFFV-11120 and HDFFV-11121) - (ADB - 2019/08/12, HDFFV-10879) + - Fixed the segmentation fault when reading attributes with multiple threads - Performance - ------------- - - + It was reported that the reading of attributes with variable length string + datatype will crash with segmentation fault particularly when the number of + threads is high (>16 threads). The problem was due to the file pointer that + was set in the variable length string datatype for the attribute. That file + pointer was already closed when the attribute was accessed. - Fortran - -------- - - Added symbolic links libhdf5_hl_fortran.so to libhdf5hl_fortran.so and - libhdf5_hl_fortran.a to libhdf5hl_fortran.a in hdf5/lib directory for - autotools installs. These were added to match the name of the files - installed by cmake and the general pattern of hl lib files. We will - change the names of the installed lib files to the matching name in - the next major release. + The problem was fixed by setting the file pointer to the current opened file pointer + when the attribute was accessed. Similar patch up was done before when reading + dataset with variable length string datatype. - (LRK - 2019/01/04, HDFFV-10596) + (VC - 2020/07/13, HDFFV-11080) - - Made Fortran specific subroutines PRIVATE in generic procedures. + - Fixed CVE-2020-10810 - Effected generic procedures were functions in H5A, H5D, H5P, H5R and H5T. + The tool h5clear produced a segfault during an error recovery in + the superblock decoding. An internal pointer was reset to prevent + further accessing when it is not assigned with a value. - (MSB, 2018/12/04, HDFFV-10511) + (BMR - 2020/06/29, HDFFV-11053) - - Fixed issue with Fortran not returning h5o_info_t field values - meta_size%attr%index_size and meta_size%attr%heap_size. + - Fixed CVE-2018-17435 - (MSB, 2018/1/8, HDFFV-10443) + The tool h52gif produced a segfault when the size of an attribute + message was corrupted and caused a buffer overflow. - - Corrected INTERFACE INTENT(IN) to INTENT(OUT) for buf_size in h5fget_file_image_f. + The problem was fixed by verifying the attribute message's size + against the buffer size before accessing the buffer. h52gif was + also fixed to display the failure instead of silently exiting + after the segfault was eliminated. - (MSB - 2020/2/18, HDFFV-11029) + (BMR - 2020/06/19, HDFFV-10591) - Tools - ----- - - The tools library was updated by standardizing the error stack process. - General sequence is: - h5tools_setprogname(PROGRAMNAME); - h5tools_setstatus(EXIT_SUCCESS); - h5tools_init(); - ... process the command-line (check for error-stack enable) ... - h5tools_error_report(); - ... (do work) ... - h5diff_exit(ret); + Java Library + ------------ + - JNI utility function does not handle new references. - (ADB - 2020/07/20, HDFFV-11066) + The JNI utility function for converting reference data to string did + not use the new APIs. In addition to fixing that function, added new + java tests for using the new APIs. - - h5diff fixed a command line parsing error. + (ADB - 2021/02/16, HDFFV-11212) - h5diff would ignore the argument to -d (delta) if it is smaller than DBL_EPSILON. - The macro H5_DBL_ABS_EQUAL was removed and a direct value comparision was used. + - The H5FArray.java class, in which virtually the entire execution time + is spent using the HDFNativeData method that converts from an array + of bytes to an array of the destination Java type. - (ADB - 2020/07/20, HDFFV-10897) + 1. Convert the entire byte array into a 1-d array of the desired type, + rather than performing 1 conversion per row; + 2. Use the Java Arrays method copyOfRange to grab the section of the + array from (1) that is desired to be inserted into the destination array. - - h5diff added a command line option to ignore attributes. + (PGT,ADB - 2020/12/13, HDFFV-10865) - h5diff would ignore all objects with a supplied path if the exclude-path argument is used. - Adding the exclude-attribute argument will only eclude attributes, with the supplied path, - from comparision. + - Added ability to test java library with VOLs. - (ADB - 2020/07/20, HDFFV-5935) + Created a new CMake script that combines the java and vol test scripts. - - h5diff added another level to the verbose argument to print filenames. + (ADB - 2020/02/03, HDFFV-10996) - Added verbose level 3 that is level 2 plus the filenames. The levels are: - 0 : Identical to '-v' or '--verbose' - 1 : All level 0 information plus one-line attribute status summary - 2 : All level 1 information plus extended attribute status report - 3 : All level 2 information plus file names + - Tests fail for non-English locales. - (ADB - 2020/07/20, HDFFV-10005) + In the JUnit tests with a non-English locale, only the part before + the decimal comma is replaced by XXXX and this leads to a comparison + error. Changed the regex for the Time substitution. - High-Level APIs: - ------ - - + (ADB - 2020/01/09, HDFFV-10995) - Fortran High-Level APIs: - ------ - - - Documentation + Configuration ------------- - - - - F90 APIs - -------- - - - - C++ APIs - -------- - - - - Testing - ------- - - Stopped java/test/junit.sh.in installing libs for testing under ${prefix} - - Lib files needed are now copied to a subdirectory in the java/test - directory, and on Macs the loader path for libhdf5.xxxs.so is changed - in the temporary copy of libhdf5_java.dylib. - - (LRK, 2020/7/2, HDFFV-11063) - - - Fixed a test failure in testpar/t_dset.c caused by - the test trying to use the parallel filters feature - on MPI-2 implementations. - - (JTH, 2019/2/7) - -Bug Fixes since HDF5-1.10.2 release -================================== + - Remove arbitrary warning flag groups from CMake builds - Library - ------- - - Java HDF5LibraryException class - - The error minor and major values would be lost after the - constructor executed. - - Created two local class variables to hold the values obtained during - execution of the constructor. Refactored the class functions to retrieve - the class values rather then calling the native functions. - The native functions were renamed and called only during execution - of the constructor. - Added error checking to calling class constructors in JNI classes. - - (ADB - 2018/08/06, HDFFV-10544) + The arbitrary groups were created to reduce the quantity of warnings being + reported that overwhelmed testing report systems. Considerable work has + been accomplished to reduce the warning count and these arbitrary groups + are no longer needed. + Also the default for all warnings, HDF5_ENABLE_ALL_WARNINGS, is now ON. - - Added checks of the defined MPI_VERSION to guard against usage of - MPI-3 functions in the Parallel Compression and "big Parallel I/O" - features when HDF5 is built with MPI-2. Previously, the configure - step would pass but the build itself would fail when it could not - locate the MPI-3 functions used. + Visual Studio warnings C4100, C4706, and C4127 have been moved to + developer warnings, HDF5_ENABLE_DEV_WARNINGS, and are disabled for normal builds. - As a result of these new checks, HDF5 can again be built with MPI-2, - but the Parallel Compression feature will be disabled as it relies - on the MPI-3 functions used. + (ADB - 2021/03/22, HDFFV-11228) - (JTH - 2018/08/02, HDFFV-10512) + - Reclassify CMake messages, to allow new modes and --log-level option - - User's patches: CVEs + CMake message commands have a mode argument. By default, STATUS mode + was chosen for any non-error message. CMake version 3.15 added additional + modes, NOTICE, VERBOSE, DEBUG and TRACE. All message commands with a mode + of STATUS were reviewed and most were reclassified as VERBOSE. The new + mode was protected by a check for a CMake version of at least 3.15. If CMake + version 3.17 or above is used, the user can use the command line option + of "--log-level" to further restrict which message commands are displayed. - The following patches have been applied: + (ADB - 2021/01/11, HDFFV-11144) - CVE-2018-11202 - NULL pointer dereference was discovered in - H5S_hyper_make_spans in H5Shyper.c (HDFFV-10476) - https://security-tracker.debian.org/tracker/CVE-2018-11202 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=3DCVE-2018-11202 + - Fixes Autotools determination of the stat struct having an st_blocks field - CVE-2018-11203 - A division by zero was discovered in - H5D__btree_decode_key in H5Dbtree.c (HDFFV-10477) - https://security-tracker.debian.org/tracker/CVE-2018-11203 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=3DCVE-2018-11203 + A missing parenthesis in an autoconf macro prevented building the test + code used to determine if the stat struct contains the st_blocks field. + Now that the test functions correctly, the H5_HAVE_STAT_ST_BLOCKS #define + found in H5pubconf.h will be defined correctly on both the Autotools and + CMake. This #define is only used in the tests and does not affect the + HDF5 C library. - CVE-2018-11204 - A NULL pointer dereference was discovered in - H5O__chunk_deserialize in H5Ocache.c (HDFFV-10478) - https://security-tracker.debian.org/tracker/CVE-2018-11204 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=3DCVE-2018-11204 + (DER - 2021/01/07, HDFFV-11201) - CVE-2018-11206 - An out of bound read was discovered in - H5O_fill_new_decode and H5O_fill_old_decode in H5Ofill.c - (HDFFV-10480) - https://security-tracker.debian.org/tracker/CVE-2018-11206 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=3DCVE-2018-11206 - CVE-2018-11207 - A division by zero was discovered in - H5D__chunk_init in H5Dchunk.c (HDFFV-10481) - https://security-tracker.debian.org/tracker/CVE-2018-11207 - https://cve.mitre.org/cgi-bin/cvename.cgi?name=3DCVE-2018-11207 - - (BMR - 2018/7/22, PR#s: 1134 and 1139, - HDFFV-10476, HDFFV-10477, HDFFV-10478, HDFFV-10480, HDFFV-10481) - - - H5Adelete - - H5Adelete failed when deleting the last "large" attribute that - is stored densely via fractal heap/v2 b-tree. + Tools + ----- + - Changed how h5dump and h5ls identify long double. - After removing the attribute, update the ainfo message. If the - number of attributes goes to zero, remove the message. + Long double support is not consistent across platforms. Tools will always + identify long double as 128-bit [little/big]-endian float nn-bit precision. + New test file created for datasets with attributes for float, double and + long double. In addition any unknown integer or float datatype will now + also show the number of bits for precision. + These files are also used in the java tests. - (VC - 2018/07/20, HDFFV-9277) + (ADB - 2021/03/24, HDFFV-11229) - - A bug was discovered in the parallel library which caused partial - parallel reads of filtered datasets to return incorrect data. The - library used the incorrect dataspace for each chunk read, causing - the selection used in each chunk to be wrong. + - Fixed tools argument parsing. - The bug was not caught during testing because all of the current - tests which do parallel reads of filtered data read all of the data - using an H5S_ALL selection. Several tests were added which exercise - partial parallel reads. + Tools parsing used the length of the option from the long array to match + the option from the command line. This incorrectly matched a shorter long + name option that happened to be a subset of another long option. + Changed to match whole names. - (JTH - 2018/07/16, HDFFV-10467) + (ADB - 2021/01/19, HDFFV-11106) - - A bug was discovered in the parallel library which caused parallel - writes of filtered datasets to trigger an assertion failure in the - file free space manager. + - The tools library was updated by standardizing the error stack process. - This occurred when the filter used caused chunks to repeatedly shrink - and grow over the course of several dataset writes. The previous chunk - information, such as the size of the chunk and the offset in the file, - was being cached and not updated after each write, causing the next write - to the chunk to retrieve the incorrect cached information and run into - issues when reallocating space in the file for the chunk. + General sequence is: + h5tools_setprogname(PROGRAMNAME); + h5tools_setstatus(EXIT_SUCCESS); + h5tools_init(); + ... process the command-line (check for error-stack enable) ... + h5tools_error_report(); + ... (do work) ... + h5diff_exit(ret); - (JTH - 2018/07/16, HDFFV-10509) + (ADB - 2020/07/20, HDFFV-11066) - - A bug was discovered in the parallel library which caused the - H5D__mpio_array_gatherv() function to allocate too much memory. + - h5diff fixed a command line parsing error. - When the function is called with the 'allgather' parameter set - to a non-true value, the function will receive data from all MPI - ranks and gather it to the single rank specied by the 'root' - parameter. However, the bug in the function caused memory for - the received data to be allocated on all MPI ranks, not just the - singular rank specified as the receiver. In some circumstances, - this would cause an application to fail due to the large amounts - of memory being allocated. + h5diff would ignore the argument to -d (delta) if it is smaller than DBL_EPSILON. + The macro H5_DBL_ABS_EQUAL was removed and a direct value comparision was used. - (JTH - 2018/07/16, HDFFV-10467) + (ADB - 2020/07/20, HDFFV-10897) - - Error checks in h5stat and when decoding messages + - h5diff added a command line option to ignore attributes. - h5stat exited with seg fault/core dumped when - errors are encountered in the internal library. + h5diff would ignore all objects with a supplied path if the exclude-path argument is used. + Adding the exclude-attribute argument will only exclude attributes, with the supplied path, + from comparison. - Add error checks and --enable-error-stack option to h5stat. - Add range checks when decoding messages: old fill value, old - layout and refcount. + (ADB - 2020/07/20, HDFFV-5935) - (VC - 2018/07/11, HDFFV-10333) + - h5diff added another level to the verbose argument to print filenames. - - If an HDF5 file contains a malformed compound datatype with a - suitably large offset, the type conversion code can run off - the end of the type conversion buffer, causing a segmentation - fault. + Added verbose level 3 that is level 2 plus the filenames. The levels are: + 0 : Identical to '-v' or '--verbose' + 1 : All level 0 information plus one-line attribute status summary + 2 : All level 1 information plus extended attribute status report + 3 : All level 2 information plus file names - This issue was reported to The HDF Group as issue #CVE-2017-17507. + (ADB - 2020/07/20, HDFFV-10005) - NOTE: The HDF5 C library cannot produce such a file. This condition - should only occur in a corrupt (or deliberately altered) file - or a file created by third-party software. + - h5repack was fixed to repack the reference attributes properly. - THE HDF GROUP WILL NOT FIX THIS BUG AT THIS TIME + The code line that checks if the update of reference inside a compound + datatype is misplaced outside the code block loop that carries out the + check. In consequence, the next attribute that is not the reference + type was repacked again as the reference type and caused the failure of + repacking. The fix is to move the corresponding code line to the correct + code block. - Fixing this problem would involve updating the publicly visible - H5T_conv_t function pointer typedef and versioning the API calls - which use it. We normally only modify the public API during - major releases, so this bug will not be fixed at this time. + (KY -2020/02/10, HDFFV-11014) - (DER - 2018/02/26, HDFFV-10356) + - h5diff was updated to use the new reference APIs. - - Inappropriate linking with deprecated MPI C++ libraries + h5diff uses the new reference APIs to compare references. + Attribute references can also be compared. - HDF5 does not define *_SKIP_MPICXX in the public headers, so applications - can inadvertently wind up linking to the deprecated MPI C++ wrappers. + (ADB - 2019/12/19, HDFFV-10980) - MPICH_SKIP_MPICXX and OMPI_SKIP_MPICXX have both been defined in H5public.h - so this should no longer be an issue. HDF5 makes no use of the deprecated - MPI C++ wrappers. + - h5dump and h5ls were updated to use the new reference APIs. - (DER - 2019/09/17, HDFFV-10893) + The tools library now use the new reference APIs to inspect a + file. Also the DDL spec was updated to reflect the format + changes produced with the new APIs. The export API and support + functions in the JNI were updated to match. + (ADB - 2019/12/06, HDFFV-10876 and HDFFV-10877) - Configuration + Performance ------------- - - Applied patches to address Cywin build issues - - There were three issues for Cygwin builds: - - Shared libs were not built. - - The -std=c99 flag caused a SIG_SETMASK undeclared error. - - Undefined errors when buildbing test shared libraries. - - Patches to address these issues were received and incorporated in this version. - - (LRK - 2018/07/18, HDFFV-10475) - - - Moved the location of gcc attribute. - - The gcc attribute(no_sanitize), named as the macro HDF_NO_UBSAN, - was located after the function name. Builds with GCC 7 did not - indicate any problem, but GCC 8 issued errors. Moved the - attribute before the function name, as required. - - (ADB - 2018/05/22, HDFFV-10473) - - - Reworked java test suite into individual JUnit tests. - - Testing the whole suite of java unit tests in a single JUnit run - made it difficult to determine actual failures when tests would fail. - Running each file set of tests individually, allows individual failures - to be diagnosed easier. A side benefit is that tests for optional components - of the library can be disabled if not configured. - - (ADB - 2018/05/16, HDFFV-9739) + - - - Converted CMake global commands ADD_DEFINITIONS and INCLUDE_DIRECTORIES - to use target_* type commands. This change modernizes the CMake usage - in the HDF5 library. - In addition, there is the intention to convert to generator expressions, - where possible. The exception is Fortran FLAGS on Windows Visual Studio. - The HDF macros TARGET_C_PROPERTIES and TARGET_FORTRAN_PROPERTIES have - been removed with this change in usage. + Fortran API + ----------- + - Corrected INTERFACE INTENT(IN) to INTENT(OUT) for buf_size in h5fget_file_image_f. - The additional language (C++ and Fortran) checks have also been localized - to only be checked when that language is enabled. + (MSB - 2020/02/18, HDFFV-11029) - (ADB - 2018/05/08) - Performance - ------------- + High-Level Library + ------------------ - - Fortran - -------- - - - Tools - ----- + Fortran High-Level APIs + ----------------------- - - High-Level APIs: - ------ - - - - Fortran High-Level APIs: - ------ - - Documentation ------------- - + F90 APIs -------- - + C++ APIs -------- - - Adding default arguments to existing functions - - Added the following items: - + Two more property list arguments are added to H5Location::createDataSet: - const DSetAccPropList& dapl = DSetAccPropList::DEFAULT - const LinkCreatPropList& lcpl = LinkCreatPropList::DEFAULT - - + One more property list argument is added to H5Location::openDataSet: - const DSetAccPropList& dapl = DSetAccPropList::DEFAULT + - Added DataSet::operator= - (BMR - 2018/07/21, PR# 1146) + Some compilers complain if the copy constructor is given explicitly + but the assignment operator is implicitly set to default. - - Improvement C++ documentation + (2021/05/19) - Replaced the table in main page of the C++ documentation from mht to htm format - for portability. - - (BMR - 2018/07/17, PR# 1141) Testing ------- - - The dt_arith test failed on IBM Power8 and Power9 machines when testing - conversions from or to long double types, especially when special values - such as infinity or NAN were involved. In some cases the results differed - by extremely small amounts from those on other machines, while some other - tests resulted in segmentation faults. These conversion tests with long - double types have been disabled for ppc64 machines until the problems are - better understood and can be properly addressed. + - Stopped java/test/junit.sh.in installing libs for testing under ${prefix} + + Lib files needed are now copied to a subdirectory in the java/test + directory, and on Macs the loader path for libhdf5.xxxs.so is changed + in the temporary copy of libhdf5_java.dylib. + + (LRK, 2020/07/02, HDFFV-11063) - (SRL - 2019/01/07, TRILAB-98) Supported Platforms =================== @@ -1414,12 +1222,10 @@ Supported Platforms (emu) Sun Fortran 95 8.6 SunOS_sparc Sun C++ 5.12 SunOS_sparc - Windows 7 x64 Visual Studio 2015 w/ Intel C, Fortran 2018 (cmake) - Visual Studio 2015 w/ MSMPI 10 (cmake) - Windows 10 x64 Visual Studio 2015 w/ Intel Fortran 18 (cmake) Visual Studio 2017 w/ Intel Fortran 19 (cmake) Visual Studio 2019 w/ Intel Fortran 19 (cmake) + Visual Studio 2019 w/ MSMPI 10.1 (cmake) Mac OS X Yosemite 10.10.5 Apple clang/clang++ version 6.1 from Xcode 7.0 64-bit gfortran GNU Fortran (GCC) 4.9.2 @@ -1451,10 +1257,6 @@ Platform C F90/ F90 C++ zlib SZIP parallel F2003 parallel Solaris2.11 32-bit n y/y n y y y Solaris2.11 64-bit n y/n n y y y -Windows 7 y y/y n y y y -Windows 7 x64 y y/y y y y y -Windows 7 Cygwin n y/n n y y y -Windows 7 x64 Cygwin n y/n n y y y Windows 10 y y/y n y y y Windows 10 x64 y y/y n y y y Mac OS X Mountain Lion 10.8.5 64-bit n y/y n y y y @@ -1473,10 +1275,6 @@ Platform Shared Shared Shared Thread- C libs F90 libs C++ libs safe Solaris2.11 32-bit y y y y Solaris2.11 64-bit y y y y -Windows 7 y y y y -Windows 7 x64 y y y y -Windows 7 Cygwin n n n y -Windows 7 x64 Cygwin n n n y Windows 10 y y y y Windows 10 x64 y y y y Mac OS X Mountain Lion 10.8.5 64-bit y n y y @@ -1527,9 +1325,28 @@ The following platforms are not supported but have been tested for this release. #1 SMP ppc64 GNU/Linux IBM XL C/C++ for Linux, V13.1 (ostrich) and IBM XL Fortran for Linux, V15.1 - Fedora30 5.3.11-200.fc30.x86_64 - #1 SMP x86_64 GNU/Linux GNU gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1 20190827) - GNU Fortran (GCC) 9.2.1 20190827 (Red Hat 9.2.1 20190827) + Fedora33 5.11.18-200.fc33.x86_64 + #1 SMP x86_64 GNU/Linux GNU gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1) + GNU Fortran (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1) + clang version 11.0.0 (Fedora 11.0.0-2.fc33) + (cmake and autotools) + + Ubuntu20.04 5.8.0-53-generic-x86_64 + #60~20.04-Ubuntu SMP x86_64 GNU/Linux GNU gcc (GCC) 9.3.0-17ubuntu1 + GNU Fortran (GCC) 9.3.0-17ubuntu1 + clang version 10.0.0-4ubuntu1 + (cmake and autotools) + + Ubuntu20.10 5.8.0-53-generic-x86_64 + #60-Ubuntu SMP x86_64 GNU/Linux GNU gcc (GCC) 10.2.0-13ubuntu1 + GNU Fortran (GCC) 10.2.0-13ubuntu1 + Ubuntu clang version 11.0.0-2 + (cmake and autotools) + + SUSE15sp2 5.3.18-22-default + #1 SMP x86_64 GNU/Linux GNU gcc (SUSE Linux) 7.5.0 + GNU Fortran (SUSE Linux) 7.5.0 + clang version 7.0.1 (tags/RELEASE_701/final 349238) (cmake and autotools) @@ -1545,6 +1362,9 @@ Known Problems images, but since this is a collective operation, a deadlock is possible if one or more processes do not participate. + CPP ptable test fails on both VS2017 and VS2019 with Intel compiler, JIRA + issue: HDFFV-10628. This test will pass with VS2015 with Intel compiler. + Known problems in previous releases can be found in the HISTORY*.txt files in the HDF5 source. Please report any new problems found to help@hdfgroup.org. |