HDF5 version 1.10.1-pre1 released on 2017-03-23 ================================================================================ Note: this document is a work in progress. In particular not all bug fixes have been documented here, and issues reported in the pre-release versions will appear in "known problems" in the final 1.10.1 release. INTRODUCTION This document describes the differences between HDF5-1.10.0-patch1 and HDF5 1.10.1, and contains information on the platforms tested and known problems in HDF5-1.10.1. For more details check the HISTORY*.txt files in the HDF5 source. Links to HDF5 1.10.1 source code, documentation, and additional materials can be found on The HDF5 web page at: https://support.hdfgroup.org/HDF5/ The HDF5 1.10.1 release can be obtained from: https://support.hdfgroup.org/HDF5/release/obtain5110.html User documentation for the snapshot can be accessed directly at this location: https://support.hdfgroup.org/HDF5/doc1.10/ New features in the HDF5-1.10.x release series, including brief general descriptions of some new and modified APIs, are described in the "What's New in 1.10.1?" document: https://support.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew1101.html All new and modified APIs are listed in detail in the "HDF5 Software Changes from Release to Release" document, in the section "Release 1.8.19 (current release) versus Release 1.10.1 https://support.hdfgroup.org/HDF5/doc1.10/ADGuide/Changes.html If you have any questions or comments, please send them to the HDF Help Desk: help@hdfgroup.org CONTENTS - Major New Features Introduced in HDF5 1.10.1 - Other New Features - Support for new platforms and languages - Bug Fixes since HDF5-1.10.0-patch1 - Supported Platforms - Tested Configuration Features Summary - More Tested Platforms - Known Problems Major New Features Introduced in HDF5 1.10.1 ============================================ For links to the RFCs and documentation in this section please view https://support.hdfgroup.org/HDF5/docNewFeatures in a web browser. ________________________________________ Metadata Cache Image ________________________________________ HDF5 metadata is typically small, and scattered throughout the HDF5 file. This can affect performance, particularly on large HPC systems. The Metadata Cache Image feature can improve performance by writing the metadata cache in a single block on file close, and then populating the cache with the contents of this block on file open, thus avoiding the many small I/O operations that would otherwise be required on file open and close. See the RFC for complete details regarding this feature. Also, see the Fine Tuning the Metadata Cache documentation. ________________________________________ Metadata Cache Evict on Close ________________________________________ The HDF5 library's metadata cache is fairly conservative about holding on to HDF5 object metadata (object headers, chunk index structures, etc.), which can cause the cache size to grow, resulting in memory pressure on an application or system. The "evict on close" property will cause all metadata for an object to be evicted from the cache as long as metadata is not referenced from any other open object. See the Fine Tuning the Metadata Cache documentation for information on the APIs. ________________________________________ Paged Aggregation ________________________________________ The current HDF5 file space allocation accumulates small pieces of metadata and raw data in aggregator blocks which are not page aligned and vary widely in sizes. The paged aggregation feature was implemented to provide efficient paged access of these small pieces of metadata and raw data. See the RFC for details. Also, see the File Space Management documentation. ________________________________________ Page Buffering ________________________________________ Small and random I/O accesses on parallel file systems result in poor performance for applications. Page buffering in conjunction with paged aggregation can improve performance by giving an application control of minimizing HDF5 I/O requests to a specific granularity and alignment. See the RFC for details. Also, see the Page Buffering documentation. New Features ============ Configuration: ------------- - CMake minimum is now 3.2.2. (ADB 2017/01/10) - Tools folder is separated into source and test folders. This allows autotools to skip the make command and just execute the make check command. (HDFFV-9719 ADB 2016/10/27) Library: -------- - Paged Aggregation This is one of the file space management strategies. This strategy aggregates small metadata and raw data allocations into constant-sized well-aligned pages, which are suitable for page caching. Paged aggregation together with the page buffering feature will allow efficient I/O accesses. - Page Buffering The page buffering layer in the HDF5 library absorbs small accesses to the file system. Each page in memory corresponds to a page allocated in the file. Access to the file system is then performed as a single page or multiple of pages, if they are contiguous. This ensures that small accesses to the file system are avoided while providing another caching layer for improved I/O performance. This feature works in conjunction with the paged aggregation feature. Parallel Library: ----------------- - Fortran Library: ---------------- - C++ Library: ------------ - New wrappers for C APIs: (HDFFV-10004, HDFFV-10139, HDFFV-10145 - PRs #209, #232, #310, #334, #348) // Sets/Gets the strategy and the threshold value that the library will // will employ in managing file space. FileCreatPropList::setFileSpaceStrategy - H5Pset_file_space_strategy FileCreatPropList::getFileSpaceStrategy - H5Pget_file_space_strategy // Sets/Gets the file space page size for paged aggregation. FileCreatPropList::setFileSpacePagesize - H5Pset_file_space_page_size FileCreatPropList::getFileSpacePagesize - H5Pget_file_space_page_size // Checks if the given ID is valid. IdComponent::isValid - H5Iis_valid // Sets/Gets the number of soft or user-defined links that can be // traversed before a failure occurs. LinkAccPropList::setNumLinks - H5Pset_nlinks LinkAccPropList::getNumLinks - H5Pget_nlinks // Returns a copy of the creation property list of a datatype. DataType::getCreatePlist - H5Tget_create_plist // Opens/Closes an object within a group or a file, regardless object type Group::getObjId - H5Oopen Group::closeObjId - H5Oclose // Maps elements of a virtual dataset to elements of the source dataset. DSetCreatPropList::setVirtual - H5Pset_virtual // Gets general information about this file. H5File::getFileInfo - H5Fget_info2 // Returns the number of members in a type. IdComponent::getNumMembers - H5Inmembers // Determines if an element type exists. IdComponent::typeExists - H5Itype_exists // Determines if an object exists. H5Location::exists - H5Lexists. // Returns the header version of an HDF5 object. H5Object::objVersion - H5Oget_info for version -BMR, 2017/03/20 - New exception: ObjHeaderIException for H5O interface. (HDFFV-10145 - PR #334) -BMR, 2017/03/15 - New class LinkAccPropList for link access property list, to be used by wrappers of H5Lexists. (HDFFV-10145 - PR #232) -BMR, 2017/01/04 - New constructors to open datatypes in ArrayType, CompType, DataType, EnumType, FloatType, IntType, StrType, and VarLenType. (HDFFV-10056 - PR #209) -BMR, 2016/12/26 - New member functions DSetCreatPropList::setNbit() to setup N-bit compression for a dataset. (HDFFV-8623 patch 7) -BMR, 2016/04/25 ArrayType::getArrayNDims() const ArrayType::getArrayDims() const both to replace the non-const versions. (HDFFV-9725) -BMR, 2016/04/25 Tools: ------ - Add options to h5clear: -s: clear the status_flags field in the file's superblock -m: Remove the metadata cache image from the file (Pull Request #361 QK 2017/03/22) High-Level APIs: --------------- C Packet Table API ------------------ - Internal header file -------------------- - Documentation ------------- Support for new platforms, languages and compilers. ======================================= - Bug Fixes since HDF5-1.10.0-patch1 release ================================== Library ------- - SWMR implementation in the HDF5 1.10.0 and 1.10.0-patch1 releases has a broken metadata flush dependency that manifested itself with the following error at the end of the HDF5 error stack: H5Dint.c line 846 in H5D__swmr_setup(): dataspace chunk index must be 0 for SWMR access, chunkno = 1 major: Dataset minor: Bad value It was also reported at https://github.com/areaDetector/ADCore/issues/203 The flush dependency is fixed in this release. - Changed the plugins dlopen option from RTLD_NOW to RTLD_LAZY (PR 201 ADB 2016/12/12) - Fix error when copying dataset with attribute which is a compound datatype consisting of a variable length string. (HDFFV-7991 VC 2016/08/19, 2016/08/21, 2016/08/24) - H5DOappend will not fail if a dataset has no append callback registered. (HDFFV-9960 VC 2016/08/05, 2016/08/14) - Fix the problem where the committed datatype's file location is different from the file location of an attribute with that committed datatype. (HDFFV-9940 VC 2016/07/03, 2016/07/06) - (a) Throw an error instead of assertion when v1 btree level hits the 1 byte limit. (b) Modifications to better handle error recovery when conversion by h5format_convert fails. (HDFFV-9434 VC 2016/05/29) Configuration ------------- - Configuration will check for the strtoll and strtoull functions before using alternatives (PR 340 ADB 2017/03/17) - CMake uses a Windows pdb directory variable if available and will generate both static and shared pdb files. (HDFFV-9875 ADB 2017/02/06) - CMake now builds shared versions of tools. (HDFFV-10123 ADB 2017/02/01) Performance ------------- - Fortran -------- - Tools ----- - h5diff correctly ignores strpad in comparing strings. (HDFFV-10128 ADB 2017/03/03) - h5repack now correctly parses the command line filter options. (HDFFV-10046 ADB 2017/01/24) High-Level APIs: ------ - Fortran High-Level APIs: ------ - Documentation ------------- F90 APIs -------- - C++ APIs -------- - Due to the change in the C API, the overloaded functions of PropList::setProperty now need const for some arguments. They are planned for deprecation and are replaced by new versions with proper consts. (PR #344) -BMR, 2017/03/17 - The problem where a user-defined function cannot access both, attribute and dataset, using only one argument is now fixed. (HDFFV-9920 - PR #45) -BMR, 2016/10/11 - The high-level API Packet Table (PT) did not write data correctly when the datatype is a compound type that has string type as one of the members. This problem started in 1.8.15, after the fix of HDFFV-9042 was applied, which caused the Packet Table to use native type to access the data. It should be up to the application to specify whether the buffer to be read into memory is in the machine's native architecture. Thus, the PT is fixed to not use native type but to make a copy of the user's provided datatype during creation or the packet table's datatype during opening. If an application wishes to use native type to read the data, then the application will request that. However, the Packet Table doesn't provide a way to specify memory datatype in this release. This feature will be available in future releases, HDFFV-10023. (HDFFV-9758 - PRs #93 and #108) -BMR, 2016/10/27 - The obsolete macros, H5_NO_NAMESPACE and H5_NO_STD, are removed. (HDFFV-9532 - PR #92) -BMR, 2016/10/23 - In-memory array information, ArrayType::rank and ArrayType::dimensions, were removed. This is an implementation detail and should not affect applications. (HDFFV-9725) -BMR, 2016/04/25 Testing ------- - Supported Platforms =================== Linux 2.6.32-573.18.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) #1 SMP ppc64 GNU/Linux g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) (ostrich) GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) IBM XL C/C++ V13.1 IBM XL Fortran V15.1 Linux 3.10.0-327.10.1.el7 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: (kituo/moohan) Version 4.8.5 20150623 (Red Hat 4.8.5-4) Version 4.9.3, Version 5.2.0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 15.0.3.187 Build 20150407 MPICH 3.1.4 compiled with GCC 4.9.3 SunOS 5.11 32- and 64-bit Sun C 5.12 SunOS_sparc (emu) Sun Fortran 95 8.6 SunOS_sparc Sun C++ 5.12 SunOS_sparc Windows 7 Visual Studio 2012 w/ Intel Fortran 15 (cmake) Visual Studio 2013 w/ Intel Fortran 15 (cmake) Visual Studio 2015 w/ Intel Fortran 16 (cmake) Cygwin(CYGWIN_NT-6.1 2.2.1(0.289/5/3) gcc(4.9.3) compiler and gfortran) (cmake and autotools) Windows 7 x64 Visual Studio 2012 w/ Intel Fortran 15 (cmake) Visual Studio 2013 w/ Intel Fortran 15 (cmake) Visual Studio 2015 w/ Intel Fortran 16 (cmake) Visual Studio 2015 w/ Intel Parallel Studio 2017 (cmake) Windows 10 Visual Studio 2015 w/ Intel Fortran 16 (cmake) Windows 10 x64 Visual Studio 2015 w/ Intel Fortran 16 (cmake) Mac OS X Mt. Lion 10.8.5 Apple clang/clang++ version 5.1 from Xcode 5.1 64-bit gfortran GNU Fortran (GCC) 4.8.2 (swallow/kite) Intel icc/icpc/ifort version 15.0.3 Mac OS X Mavericks 10.9.5 Apple clang/clang++ version 6.0 from Xcode 6.2 64-bit gfortran GNU Fortran (GCC) 4.9.2 (wren/quail) Intel icc/icpc/ifort version 15.0.3 Mac OS X Yosemite 10.10.5 Apple clang/clang++ version 6.1 from Xcode 7.0 64-bit gfortran GNU Fortran (GCC) 4.9.2 (osx1010dev/osx1010test) Intel icc/icpc/ifort version 15.0.3 Mac OS X El Capitan 10.11.6 Apple clang/clang++ version 7.3.0 from Xcode 7.3 64-bit gfortran GNU Fortran (GCC) 5.2.0 (osx1010dev/osx1010test) Intel icc/icpc/ifort version 16.0.2 Tested Configuration Features Summary ===================================== In the tables below y = tested n = not tested in this release C = Cluster W = Workstation x = not working in this release dna = does not apply ( ) = footnote appears below second table = testing incomplete on this feature or platform Platform C F90/ F90 C++ zlib SZIP parallel F2003 parallel Solaris2.11 32-bit n y/y n y y y Solaris2.11 64-bit n y/n n y y y Windows 7 y y/y n y y y Windows 7 x64 y y/y y y y y Windows 7 Cygwin n y/n n y y y Windows 7 x64 Cygwin n y/n n y y y Windows 10 y y/y n y y y Windows 10 x64 y y/y n y y y Mac OS X Mountain Lion 10.8.5 64-bit n y/y n y y y Mac OS X Mavericks 10.9.5 64-bit n y/y n y y y Mac OS X Yosemite 10.10.5 64-bit n y/y n y y y Mac OS X El Capitan 10.11.6 64-bit n y/y n y y y CentOS 7.2 Linux 2.6.32 x86_64 PGI n y/y n y y y CentOS 7.2 Linux 2.6.32 x86_64 GNU y y/y y y y y CentOS 7.2 Linux 2.6.32 x86_64 Intel n y/y n y y y Linux 2.6.32-573.18.1.el6.ppc64 n y/y n y y y Platform Shared Shared Shared Thread- C libs F90 libs C++ libs safe Solaris2.11 32-bit y y y y Solaris2.11 64-bit y y y y Windows 7 y y y y Windows 7 x64 y y y y Windows 7 Cygwin n n n y Windows 7 x64 Cygwin n n n y Windows 10 y y y y Windows 10 x64 y y y y Mac OS X Mountain Lion 10.8.5 64-bit y n y y Mac OS X Mavericks 10.9.5 64-bit y n y y Mac OS X Yosemite 10.10.5 64-bit y n y y Mac OS X El Capitan 10.11.6 64-bit y n y y CentOS 7.2 Linux 2.6.32 x86_64 PGI y y y n CentOS 7.2 Linux 2.6.32 x86_64 GNU y y y y CentOS 7.2 Linux 2.6.32 x86_64 Intel y y y n Linux 2.6.32-573.18.1.el6.ppc64 y y y n Compiler versions for each platform are listed in the preceding "Supported Platforms" table. More Tested Platforms ===================== The following platforms are not supported but have been tested for this release. Linux 2.6.32-573.22.1.el6 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: (mayll/platypus) Version 4.4.7 20120313 Version 4.8.4 PGI C, Fortran, C++ for 64-bit target on x86-64; Version 16.10-0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 15.0.3.187 (Build 20150407) MPICH 3.1.4 compiled with GCC 4.9.3 Linux 2.6.32-573.18.1.el6.ppc64 MPICH mpich 3.1.4 compiled with #1 SMP ppc64 GNU/Linux IBM XL C/C++ for Linux, V13.1 (ostrich) and IBM XL Fortran for Linux, V15.1 Debian 8.4 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1 x86_64 GNU/Linux gcc, g++ (Debian 4.9.2-10) 4.9.2 GNU Fortran (Debian 4.9.2-10) 4.9.2 (cmake and autotools) Fedora 24 4.7.2-201.fc24.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux gcc, g++ (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3) GNU Fortran (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3) (cmake and autotools) Ubuntu 16.04.1 4.4.0-38-generic #57-Ubuntu SMP x86_64 GNU/Linux gcc, g++ (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 (cmake and autotools) Known Problems ============== This section will be populated with issues reported to help@hdfgroup.org during pre-release testing. Please report any issues.