HDF5 version 1.10.2-snap0 currently under development ================================================================================ INTRODUCTION This document describes the differences between HDF5-1.10.0-patch1 and HDF5 1.10.1, and contains information on the platforms tested and known problems in HDF5-1.10.1. For more details check the HISTORY*.txt files in the HDF5 source. Links to HDF5 1.10.1 source code, documentation, and additional materials can be found on The HDF5 web page at: https://support.hdfgroup.org/HDF5/ The HDF5 1.10.1 release can be obtained from: https://support.hdfgroup.org/HDF5/release/obtain5110.html User documentation for the snapshot can be accessed directly at this location: https://support.hdfgroup.org/HDF5/doc1.10/ New features in the HDF5-1.10.x release series, including brief general descriptions of some new and modified APIs, are described in the "What's New in 1.10.1?" document: https://support.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew1101.html All new and modified APIs are listed in detail in the "HDF5 Software Changes from Release to Release" document, in the section "Release 1.8.19 (current release) versus Release 1.10.1 https://support.hdfgroup.org/HDF5/doc1.10/ADGuide/Changes.html If you have any questions or comments, please send them to the HDF Help Desk: help@hdfgroup.org CONTENTS - New Features - Support for new platforms and languages - Bug Fixes since HDF5-1.10.0-patch1 - Supported Platforms - Tested Configuration Features Summary - More Tested Platforms - Known Problems New Features ============ Configuration: ------------- - CMake minimum is now 3.2.2. (ADB 2017/01/10) - Tools folder is separated into source and test folders. This allows autotools to skip the make command and just execute the make check command. (HDFFV-9719 ADB 2016/10/27) Library: -------- - Paged Aggregation This is one of the file space management strategies. This strategy aggregates small metadata and raw data allocations into constant-sized well-aligned pages, which are suitable for page caching. Paged aggregation together with the page buffering feature will allow efficient I/O accesses. - Page Buffering The page buffering layer in the HDF5 library absorbs small accesses to the file system. Each page in memory corresponds to a page allocated in the file. Access to the file system is then performed as a single page or multiple of pages, if they are contiguous. This ensures that small accesses to the file system are avoided while providing another caching layer for improved I/O performance. This feature works in conjunction with the paged aggregation feature. - Filter plugin API added to access the table of paths to search for a library. Java interface expanded with wrappers for the new functions. (HDFFV-10143 ADB 2017/04/04) Parallel Library: ----------------- - Fortran Library: ---------------- - C++ Library: ------------ - New wrappers for C APIs: (HDFFV-10004, HDFFV-10139, HDFFV-10145 - PRs #209, #232, #310, #334, #348) // Sets/Gets the strategy and the threshold value that the library will // will employ in managing file space. FileCreatPropList::setFileSpaceStrategy - H5Pset_file_space_strategy FileCreatPropList::getFileSpaceStrategy - H5Pget_file_space_strategy // Sets/Gets the file space page size for paged aggregation. FileCreatPropList::setFileSpacePagesize - H5Pset_file_space_page_size FileCreatPropList::getFileSpacePagesize - H5Pget_file_space_page_size // Checks if the given ID is valid. IdComponent::isValid - H5Iis_valid // Sets/Gets the number of soft or user-defined links that can be // traversed before a failure occurs. LinkAccPropList::setNumLinks - H5Pset_nlinks LinkAccPropList::getNumLinks - H5Pget_nlinks // Returns a copy of the creation property list of a datatype. DataType::getCreatePlist - H5Tget_create_plist // Opens/Closes an object within a group or a file, regardless object type Group::getObjId - H5Oopen Group::closeObjId - H5Oclose // Maps elements of a virtual dataset to elements of the source dataset. DSetCreatPropList::setVirtual - H5Pset_virtual // Gets general information about this file. H5File::getFileInfo - H5Fget_info2 // Returns the number of members in a type. IdComponent::getNumMembers - H5Inmembers // Determines if an element type exists. IdComponent::typeExists - H5Itype_exists // Determines if an object exists. H5Location::exists - H5Lexists. // Returns the header version of an HDF5 object. H5Object::objVersion - H5Oget_info for version -BMR, 2017/03/20 - New exception: ObjHeaderIException for H5O interface. (HDFFV-10145 - PR #334) -BMR, 2017/03/15 - New class LinkAccPropList for link access property list, to be used by wrappers of H5Lexists. (HDFFV-10145 - PR #232) -BMR, 2017/01/04 - New constructors to open datatypes in ArrayType, CompType, DataType, EnumType, FloatType, IntType, StrType, and VarLenType. (HDFFV-10056 - PR #209) -BMR, 2016/12/26 - New member functions DSetCreatPropList::setNbit() to setup N-bit compression for a dataset. (HDFFV-8623 patch 7) -BMR, 2016/04/25 ArrayType::getArrayNDims() const ArrayType::getArrayDims() const both to replace the non-const versions. (HDFFV-9725) -BMR, 2016/04/25 Tools: ------ - Add options to h5clear: -s: clear the status_flags field in the file's superblock -m: Remove the metadata cache image from the file (Pull Request #361 QK 2017/03/22) High-Level APIs: --------------- C Packet Table API ------------------ - Internal header file -------------------- - Documentation ------------- Support for new platforms, languages and compilers. ======================================= - Bug Fixes since HDF5-1.10.0-patch1 release ================================== Library ------- - Changed the plugins dlopen option from RTLD_NOW to RTLD_LAZY (PR 201 ADB 2016/12/12) - Fix error when copying dataset with attribute which is a compound datatype consisting of a variable length string. (HDFFV-7991 VC 2016/08/19, 2016/08/21, 2016/08/24) - H5DOappend will not fail if a dataset has no append callback registered. (HDFFV-9960 VC 2016/08/05, 2016/08/14) - Fix the problem where the committed datatype's file location is different from the file location of an attribute with that committed datatype. (HDFFV-9940 VC 2016/07/03, 2016/07/06) - (a) Throw an error instead of assertion when v1 btree level hits the 1 byte limit. (b) Modifications to better handle error recovery when conversion by h5format_convert fails. (HDFFV-9434 VC 2016/05/29) - Fix error when copying dataset with attribute which is a compound datatype consisting of a variable length string. (HDFFV-7991 VC 2016/08/19, 2016/08/21, 2016/08/24) - H5DOappend will not fail if a dataset has no append callback registered. (HDFFV-9960 VC 2016/08/05, 2016/08/14) - Fix the problem where the committed datatype's file location is different from the file location of an attribute with that committed datatype. (HDFFV-9940 VC 2016/07/03, 2016/07/06) - (a) Throw an error instead of assertion when v1 btree level hits the 1 byte limit. (b) Modifications to better handle error recovery when conversion by h5format_convert fails. (HDFFV-9434 VC 2016/05/29) Configuration ------------- - Configuration will check for the strtoll and strtoull functions before using alternatives (PR 340 ADB 2017/03/17) - CMake uses a Windows pdb directory variable if available and will generate both static and shared pdb files. (HDFFV-9875 ADB 2017/02/06) - CMake now builds shared versions of tools. (HDFFV-10123 ADB 2017/02/01) Performance ------------- - Fortran -------- - Tools ----- - h5diff correctly ignores strpad in comparing strings. (HDFFV-10128 ADB 2017/03/03) - h5repack now correctly parses the command line filter options. (HDFFV-10046 ADB 2017/01/24) - h5diff correctly indicates error when it cannot read data due to an unavailable filter plugin. (HDFFV-9994 ADB 2017/01/18) - h5repack allows the --enable-error-stack option on the command line. (HDFFV-775 ADB 2016/08/08) High-Level APIs: ------ - Fortran High-Level APIs: ------ - Documentation ------------- F90 APIs -------- - C++ APIs -------- - Due to the change in the C API, the overloaded functions of PropList::setProperty now need const for some arguments. They are planned for deprecation and are replaced by new versions with proper consts. (PR #344) -BMR, 2017/03/17 - The problem where a user-defined function cannot access both, attribute and dataset, using only one argument is now fixed. (HDFFV-9920 - PR #45) -BMR, 2016/10/11 - The high-level API Packet Table (PT) did not write data correctly when the datatype is a compound type that has string type as one of the members. This problem started in 1.8.15, after the fix of HDFFV-9042 was applied, which caused the Packet Table to use native type to access the data. It should be up to the application to specify whether the buffer to be read into memory is in the machine's native architecture. Thus, the PT is fixed to not use native type but to make a copy of the user's provided datatype during creation or the packet table's datatype during opening. If an application wishes to use native type to read the data, then the application will request that. However, the Packet Table doesn't provide a way to specify memory datatype in this release. This feature will be available in future releases, HDFFV-10023. (HDFFV-9758 - PRs #93 and #108) -BMR, 2016/10/27 - The obsolete macros, H5_NO_NAMESPACE and H5_NO_STD, are removed. (HDFFV-9532 - PR #92) -BMR, 2016/10/23 - In-memory array information, ArrayType::rank and ArrayType::dimensions, were removed. This is an implementation detail and should not affect applications. (HDFFV-9725) -BMR, 2016/04/25 Testing ------- - Supported Platforms =================== Linux 2.6.32-573.18.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) #1 SMP ppc64 GNU/Linux g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) (ostrich) GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) IBM XL C/C++ V13.1 IBM XL Fortran V15.1 Linux 3.10.0-327.10.1.el7 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: (kituo/moohan) Version 4.8.5 20150623 (Red Hat 4.8.5-4) Version 4.9.3, Version 5.2.0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 15.0.3.187 Build 20150407 MPICH 3.1.4 compiled with GCC 4.9.3 SunOS 5.11 32- and 64-bit Sun C 5.12 SunOS_sparc (emu) Sun Fortran 95 8.6 SunOS_sparc Sun C++ 5.12 SunOS_sparc Windows 7 Visual Studio 2012 w/ Intel Fortran 15 (cmake) Visual Studio 2013 w/ Intel Fortran 15 (cmake) Visual Studio 2015 w/ Intel Fortran 16 (cmake) Cygwin(CYGWIN_NT-6.1 2.2.1(0.289/5/3) gcc(4.9.3) compiler and gfortran) (cmake and autotools) Windows 7 x64 Visual Studio 2012 w/ Intel Fortran 15 (cmake) Visual Studio 2013 w/ Intel Fortran 15 (cmake) Visual Studio 2015 w/ Intel Fortran 16 (cmake) Visual Studio 2015 w/ Intel Parallel Studio 2017 (cmake) Windows 10 Visual Studio 2015 w/ Intel Fortran 16 (cmake) Windows 10 x64 Visual Studio 2015 w/ Intel Fortran 16 (cmake) Mac OS X Mt. Lion 10.8.5 Apple clang/clang++ version 5.1 from Xcode 5.1 64-bit gfortran GNU Fortran (GCC) 4.8.2 (swallow/kite) Intel icc/icpc/ifort version 15.0.3 Mac OS X Mavericks 10.9.5 Apple clang/clang++ version 6.0 from Xcode 6.2 64-bit gfortran GNU Fortran (GCC) 4.9.2 (wren/quail) Intel icc/icpc/ifort version 15.0.3 Mac OS X Yosemite 10.10.5 Apple clang/clang++ version 6.1 from Xcode 7.0 64-bit gfortran GNU Fortran (GCC) 4.9.2 (osx1010dev/osx1010test) Intel icc/icpc/ifort version 15.0.3 Mac OS X El Capitan 10.11.6 Apple clang/clang++ version 7.3.0 from Xcode 7.3 64-bit gfortran GNU Fortran (GCC) 5.2.0 (osx1010dev/osx1010test) Intel icc/icpc/ifort version 16.0.2 Tested Configuration Features Summary ===================================== In the tables below y = tested n = not tested in this release C = Cluster W = Workstation x = not working in this release dna = does not apply ( ) = footnote appears below second table = testing incomplete on this feature or platform Platform C F90/ F90 C++ zlib SZIP parallel F2003 parallel Solaris2.11 32-bit n y/y n y y y Solaris2.11 64-bit n y/n n y y y Windows 7 y y/y n y y y Windows 7 x64 y y/y y y y y Windows 7 Cygwin n y/n n y y y Windows 7 x64 Cygwin n y/n n y y y Windows 10 y y/y n y y y Windows 10 x64 y y/y n y y y Mac OS X Mountain Lion 10.8.5 64-bit n y/y n y y y Mac OS X Mavericks 10.9.5 64-bit n y/y n y y y Mac OS X Yosemite 10.10.5 64-bit n y/y n y y y Mac OS X El Capitan 10.11.6 64-bit n y/y n y y y CentOS 7.2 Linux 2.6.32 x86_64 PGI n y/y n y y y CentOS 7.2 Linux 2.6.32 x86_64 GNU y y/y y y y y CentOS 7.2 Linux 2.6.32 x86_64 Intel n y/y n y y y Linux 2.6.32-573.18.1.el6.ppc64 n y/y n y y y Platform Shared Shared Shared Thread- C libs F90 libs C++ libs safe Solaris2.11 32-bit y y y y Solaris2.11 64-bit y y y y Windows 7 y y y y Windows 7 x64 y y y y Windows 7 Cygwin n n n y Windows 7 x64 Cygwin n n n y Windows 10 y y y y Windows 10 x64 y y y y Mac OS X Mountain Lion 10.8.5 64-bit y n y y Mac OS X Mavericks 10.9.5 64-bit y n y y Mac OS X Yosemite 10.10.5 64-bit y n y y Mac OS X El Capitan 10.11.6 64-bit y n y y CentOS 7.2 Linux 2.6.32 x86_64 PGI y y y n CentOS 7.2 Linux 2.6.32 x86_64 GNU y y y y CentOS 7.2 Linux 2.6.32 x86_64 Intel y y y n Linux 2.6.32-573.18.1.el6.ppc64 y y y n Compiler versions for each platform are listed in the preceding "Supported Platforms" table. More Tested Platforms ===================== The following platforms are not supported but have been tested for this release. Linux 2.6.32-573.22.1.el6 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: (mayll/platypus) Version 4.4.7 20120313 Version 4.8.4 PGI C, Fortran, C++ for 64-bit target on x86-64; Version 16.10-0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 15.0.3.187 (Build 20150407) MPICH 3.1.4 compiled with GCC 4.9.3 Linux 2.6.32-573.18.1.el6.ppc64 MPICH mpich 3.1.4 compiled with #1 SMP ppc64 GNU/Linux IBM XL C/C++ for Linux, V13.1 (ostrich) and IBM XL Fortran for Linux, V15.1 Debian 8.4 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1 x86_64 GNU/Linux gcc, g++ (Debian 4.9.2-10) 4.9.2 GNU Fortran (Debian 4.9.2-10) 4.9.2 (cmake and autotools) Fedora 24 4.7.2-201.fc24.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux gcc, g++ (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3) GNU Fortran (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3) (cmake and autotools) Ubuntu 16.04.1 4.4.0-38-generic #57-Ubuntu SMP x86_64 GNU/Linux gcc, g++ (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 (cmake and autotools) Known Problems ==============