HDF5 version 1.11.0 currently under development ================================================================================ INTRODUCTION This document describes the differences between HDF5-1.10.0-patch1 and HDF5 1.10.1, and contains information on the platforms tested and known problems in HDF5-1.10.1. For more details check the HISTORY*.txt files in the HDF5 source. Links to HDF5 1.10.1 source code, documentation, and additional materials can be found on The HDF5 web page at: https://support.hdfgroup.org/HDF5/ The HDF5 1.10.1 release can be obtained from: https://support.hdfgroup.org/HDF5/release/obtain5110.html User documentation for the snapshot can be accessed directly at this location: https://support.hdfgroup.org/HDF5/doc1.10/ New features in the HDF5-1.10.x release series, including brief general descriptions of some new and modified APIs, are described in the "What's New in 1.10.1?" document: https://support.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew1101.html All new and modified APIs are listed in detail in the "HDF5 Software Changes from Release to Release" document, in the section "Release 1.8.19 (current release) versus Release 1.10.1 https://support.hdfgroup.org/HDF5/doc1.10/ADGuide/Changes.html If you have any questions or comments, please send them to the HDF Help Desk: help@hdfgroup.org CONTENTS - New Features - Support for new platforms and languages - Bug Fixes since HDF5-1.10.0-patch1 - Supported Platforms - Tested Configuration Features Summary - More Tested Platforms - Known Problems New Features ============ Configuration: ------------- - CMake minimum is now 3.2.2. (ADB 2017/01/10) - Tools folder is separated into source and test folders. This allows autotools to skip the make command and just execute the make check command. (HDFFV-9719 ADB 2016/10/27) Library: -------- - Paged Aggregation This is one of the file space management strategies. This strategy aggregates small metadata and raw data allocations into constant-sized well-aligned pages, which are suitable for page caching. Paged aggregation together with the page buffering feature will allow efficient I/O accesses. - Page Buffering The page buffering layer in the HDF5 library absorbs small accesses to the file system. Each page in memory corresponds to a page allocated in the file. Access to the file system is then performed as a single page or multiple of pages, if they are contiguous. This ensures that small accesses to the file system are avoided while providing another caching layer for improved I/O performance. This feature works in conjunction with the paged aggregation feature. Parallel Library: ----------------- - Fortran Library: ---------------- - C++ Library: ------------ - Tools: ------ - Add options to h5clear: -s: clear the status_flags field in the file's superblock -m: Remove the metadata cache image from the file (Pull Request #361 QK 2017/03/22) High-Level APIs: --------------- C Packet Table API ------------------ - Internal header file -------------------- - Documentation ------------- Support for new platforms, languages and compilers. ======================================= - Bug Fixes since HDF5-1.10.0-patch1 release ================================== Library ------- - Changed the plugins dlopen option from RTLD_NOW to RTLD_LAZY (PR 201 ADB 2016/12/12) - Fix error when copying dataset with attribute which is a compound datatype consisting of a variable length string. (HDFFV-7991 VC 2016/08/19, 2016/08/21, 2016/08/24) - H5DOappend will not fail if a dataset has no append callback registered. (HDFFV-9960 VC 2016/08/05, 2016/08/14) - Fix the problem where the committed datatype's file location is different from the file location of an attribute with that committed datatype. (HDFFV-9940 VC 2016/07/03, 2016/07/06) - (a) Throw an error instead of assertion when v1 btree level hits the 1 byte limit. (b) Modifications to better handle error recovery when conversion by h5format_convert fails. (HDFFV-9434 VC 2016/05/29) Configuration ------------- - Configuration will check for the strtoll and strtoull functions before using alternatives (PR 340 ADB 2017/03/17) - CMake uses a Windows pdb directory variable if available and will generate both static and shared pdb files. (HDFFV-9875 ADB 2017/02/06) - CMake now builds shared versions of tools. (HDFFV-10123 ADB 2017/02/01) Performance ------------- - Fortran -------- - Tools ----- - h5diff correctly ignores strpad in comparing strings. (HDFFV-10128 ADB 2017/03/03) - h5repack now correctly parses the command line filter options. (HDFFV-10046 ADB 2017/01/24) High-Level APIs: ------ - Fortran High-Level APIs: ------ - Documentation ------------- F90 APIs -------- - C++ APIs -------- - Testing ------- - Supported Platforms =================== Linux 2.6.32-573.22.1.el6 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: (mayll/platypus) Version 4.4.7 20120313 Version 4.8.4 PGI C, Fortran, C++ for 64-bit target on x86-64; Version 16.10-0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 15.0.3.187 (Build 20150407) MPICH 3.1.4 compiled with GCC 4.9.3 Linux 2.6.32-573.18.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) #1 SMP ppc64 GNU/Linux g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) (ostrich) GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) IBM XL C/C++ V13.1 IBM XL Fortran V15.1 Linux 3.10.0-327.10.1.el7 GNU C (gcc), Fortran (gfortran), C++ (g++) #1 SMP x86_64 GNU/Linux compilers: (kituo/moohan) Version 4.8.5 20150623 (Red Hat 4.8.5-4) Version 4.9.3, Version 5.2.0 Intel(R) C (icc), C++ (icpc), Fortran (icc) compilers: Version 15.0.3.187 Build 20150407 MPICH 3.1.4 compiled with GCC 4.9.3 SunOS 5.11 32- and 64-bit Sun C 5.12 SunOS_sparc (emu) Sun Fortran 95 8.6 SunOS_sparc Sun C++ 5.12 SunOS_sparc Windows 7 Visual Studio 2012 w/ Intel Fortran 15 (cmake) Visual Studio 2013 w/ Intel Fortran 15 (cmake) Visual Studio 2015 w/ Intel Fortran 16 (cmake) Cygwin(CYGWIN_NT-6.1 2.2.1(0.289/5/3) gcc(4.9.3) compiler and gfortran) (cmake and autotools) Windows 7 x64 Visual Studio 2012 w/ Intel Fortran 15 (cmake) Visual Studio 2013 w/ Intel Fortran 15 (cmake) Visual Studio 2015 w/ Intel Fortran 16 (cmake) Visual Studio 2015 w/ Intel Parallel Studio 2017 (cmake) Windows 10 Visual Studio 2015 w/ Intel Fortran 16 (cmake) Windows 10 x64 Visual Studio 2015 w/ Intel Fortran 16 (cmake) Mac OS X Mt. Lion 10.8.5 Apple clang/clang++ version 5.1 from Xcode 5.1 64-bit gfortran GNU Fortran (GCC) 4.8.2 (swallow/kite) Intel icc/icpc/ifort version 15.0.3 Mac OS X Mavericks 10.9.5 Apple clang/clang++ version 6.0 from Xcode 6.2 64-bit gfortran GNU Fortran (GCC) 4.9.2 (wren/quail) Intel icc/icpc/ifort version 15.0.3 Mac OS X Yosemite 10.10.5 Apple clang/clang++ version 6.1 from Xcode 7.0 64-bit gfortran GNU Fortran (GCC) 4.9.2 (osx1010dev/osx1010test) Intel icc/icpc/ifort version 15.0.3 Mac OS X El Capitan 10.11.6 Apple clang/clang++ version 7.3.0 from Xcode 7.3 64-bit gfortran GNU Fortran (GCC) 5.2.0 (osx1010dev/osx1010test) Intel icc/icpc/ifort version 16.0.2 Tested Configuration Features Summary ===================================== In the tables below y = tested n = not tested in this release C = Cluster W = Workstation x = not working in this release dna = does not apply ( ) = footnote appears below second table = testing incomplete on this feature or platform Platform C F90/ F90 C++ zlib SZIP parallel F2003 parallel Solaris2.11 32-bit n y/y n y y y Solaris2.11 64-bit n y/n n y y y Windows 7 y y/y n y y y Windows 7 x64 y y/y y y y y Windows 7 Cygwin n y/n n y y y Windows 7 x64 Cygwin n y/n n y y y Windows 10 y y/y n y y y Windows 10 x64 y y/y n y y y Mac OS X Mountain Lion 10.8.5 64-bit n y/y n y y y Mac OS X Mavericks 10.9.5 64-bit n y/y n y y ? Mac OS X Yosemite 10.10.5 64-bit n y/y n y y ? Mac OS X El Capitan 10.11.6 64-bit n y/y n y y ? CentOS 6.7 Linux 2.6.18 x86_64 GNU n y/y n y y y CentOS 6.7 Linux 2.6.18 x86_64 Intel n y/y n y y y CentOS 6.7 Linux 2.6.32 x86_64 PGI n y/y n y y y CentOS 7.2 Linux 2.6.32 x86_64 GNU y y/y y y y y CentOS 7.2 Linux 2.6.32 x86_64 Intel n y/y n y y y Linux 2.6.32-573.18.1.el6.ppc64 n y/n n y y y Platform Shared Shared Shared Thread- C libs F90 libs C++ libs safe Solaris2.11 32-bit y y y y Solaris2.11 64-bit y y y y Windows 7 y y y y Windows 7 x64 y y y y Windows 7 Cygwin n n n y Windows 7 x64 Cygwin n n n y Windows 10 y y y y Windows 10 x64 y y y y Mac OS X Mountain Lion 10.8.5 64-bit y n y y Mac OS X Mavericks 10.9.5 64-bit y n y y Mac OS X Yosemite 10.10.5 64-bit y n y y Mac OS X El Capitan 10.11.6 64-bit y n y y CentOS 6.7 Linux 2.6.18 x86_64 GNU y y y y CentOS 6.7 Linux 2.6.18 x86_64 Intel y y y n CentOS 6.7 Linux 2.6.32 x86_64 PGI y y y n CentOS 7.2 Linux 2.6.32 x86_64 GNU y y y n CentOS 7.2 Linux 2.6.32 x86_64 Intel y y y n Linux 2.6.32-573.18.1.el6.ppc64 y y y n Compiler versions for each platform are listed in the preceding "Supported Platforms" table. More Tested Platforms ===================== The following platforms are not supported but have been tested for this release. Linux 2.6.32-573.18.1.el6.ppc64 MPICH mpich 3.1.4 compiled with #1 SMP ppc64 GNU/Linux IBM XL C/C++ for Linux, V13.1 (ostrich) and IBM XL Fortran for Linux, V15.1 Debian 8.4 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1 x86_64 GNU/Linux gcc, g++ (Debian 4.9.2-10) 4.9.2 GNU Fortran (Debian 4.9.2-10) 4.9.2 (cmake and autotools) Fedora 24 4.7.2-201.fc24.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux gcc, g++ (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3) GNU Fortran (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3) (cmake and autotools) Ubuntu 16.04.1 4.4.0-38-generic #57-Ubuntu SMP x86_64 GNU/Linux gcc, g++ (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 (cmake and autotools) Known Problems ============== * "make check" fails on CYGWIN when building shared lib files is enabled. The default on Cygwin has been changed to disable shared. It can be enabled with the --enable-shared configure option but is likely to fail "make check" with GCC compilers. (LK -2015/04/16) * CLANG compiler with the options -fcatch-undefined-behavior and -ftrapv catches some undefined behavior in the alignment algorithm of the macro DETECT_I in H5detect.c (Issue 8147). Since the algorithm is trying to detect the alignment of integers, ideally the flag -fcatch-undefined-behavior shouldn't to be used for H5detect.c. In the future, we can separate flags for H5detect.c from the rest of the library. (SLU - 2013/10/16) * The 5.9 C++ compiler on Sun failed to compile a C++ test ttypes.cpp. It complains with this message: "/home/hdf5/src/H5Vprivate.h", line 130: Error: __func__ is not defined. The reason is that __func__ is a predefined identifier in C99 standard. The HDF5 C library uses it in H5private.h. The test ttypes.cpp includes H5private.h (H5Tpkg.h<-H5Fprivate.h<-H5Vprivate.h<-H5private.h). Sun's 5.9 C++ compiler doesn't support __func__, thus fails to compile the C++ test. But 5.11 C++ compiler does. To check whether your Sun C++ compiler knows this identifier, try to compile the following simple C++ program: #include int main(void) { printf("%s\n", __func__); return 0; } (SLU - 2012/11/5) * The C++ and FORTRAN bindings are not currently working on FreeBSD with the native release 8.2 compilers (4.2.1), but are working with gcc 4.6 from the ports (and probably gcc releases after that). (QAK - 2012/10/19) * The data conversion test dt_arith.c has failures (segmentation fault) from "long double" to other datatypes during hard conversion when the library is built with the default GCC 4.2.1 on Mac Lion system. It only happens with optimization (-O3, -O2, and -O1). Some newer versions of GCC do not have this problem. Users should disable optimization or try newer version of GCC. (Issue 8017. SLU - 2012/6/12) * The data conversion test dt_arith.c fails in "long double" to integer conversion on Ubuntu 11.10 (3.0.0.13 kernal) with GCC 4.6.1 if the library is built with optimization -O3 or -O2. The older GCC (4.5) or newer kernal (3.2.2 on Fedora) doesn't have the problem. Users should lower down the optimization level (-O1 or -O0) by defining CFLAGS in the command line of "configure" like: CFLAGS=-O1 ./configure It will overwrite the library's default optimization level. (Issue 7829. SLU - 2012/2/7) * --with-mpe configure option does not work with Mpich2. AKC - 2011/03/10) * While working on the 1.8.6 release of HDF5, a bug was discovered that can occur when reading from a dataset in parallel shortly after it has been written to collectively. The issue was exposed by a new test in the parallel HDF5 test suite, but had existed before that. We believe the problem lies with certain MPI implementations and/or filesystems. We have provided a pure MPI test program, as well as a standalone HDF5 program, that can be used to determine if this is an issue on your system. They should be run across multiple nodes with a varying number of processes. These programs can be found at: http://www.hdfgroup.org/ftp/HDF5/examples/known_problems/ * Parallel mode in AIX will fail some of the testcheck_version.sh tests where it treats "exit(134) the same as if process 0 had received an abort signal. This is fixed and will be available in the next release. AKC - 2009/11/3 * The PathScale MPI implementation, accessing a Panasas file system, would cause H5Fcreate() with H5F_ACC_EXCL to fail even when the file is not existing. This is due to the MPI_File_open() call failing if the amode has the MPI_MODE_EXCL bit set. (See bug 1468 for details.) AKC - 2009/8/11 * Parallel tests failed with 16 processes with data inconsistency at testphdf5 / dataset_readAll. Parallel tests also failed with 32 and 64 processes with collective abort of all ranks at t_posix_compliant / allwrite_allread_blocks with MPI IO. (CMC - 2009/04/28) * For SNL, spirit/liberty/thunderbird: The serial tests pass but parallel tests failed with MPI-IO file locking message. AKC - 2007/6/25. * On Intel 64 Linux cluster (RH 4, Linux 2.6.9) with Intel 10.0 compilers use -mp -O1 compilation flags to build the libraries. Higher level of optimization causes failures in several HDF5 library tests. * For HPUX 11.23 many tools tests failed for 64-bit version when linked to the shared libraries (tested for 1.8.0-beta2) * For SNL, Red Storm: only paralle HDF5 is supported. The serial tests pass and the parallel tests also pass with lots of non-fatal error messages. * on SUN 5.10 C++ test fails in the "Testing Shared Datatypes with Attributes" test * configuring with --enable-debug=all produces compiler errors on most platforms. Users who want to run HDF5 in debug mode should use --enable-debug rather than --enable-debug=all to enable debugging information on most modules. * On Mac OS 10.4, test/dt_arith.c has some errors in conversion from long double to (unsigned) long long and from (unsigned)long long to long double. * On Altix SGI with Intel 9.0 testmeta.c would not compile with -O3 optimization flag. * On VAX, Scaleoffset filter isn't supported. The filter cannot be applied to HDF5 data generated on VAX. Scaleoffset filter only supports IEEE standard for floating-point data. * On Cray X1, a lone colon on the command line of h5dump --xml (as in the testh5dumpxml.sh script) is misinterpereted by the operating system and causes an error. * On mpich 1.2.5 and 1.2.6, we found that if more than two processes contribute no IO and the application asks to do IO with collective, we found that when using 4 processors, a simple collective write will be hung sometimes. This can be verified with t_mpi test under testpar. * The dataset created or rewritten with the v1.6.3 library or after can't be read with the v1.6.2 library or before when Fletcher32 EDC(filter) is enabled. There was a bug in the calculating code of the Fletcher32 checksum in the library before v1.6.3. The checksum value wasn't consistent between big-endian and little-endian systems. This bug was fixed in Release 1.6.3. However, after fixing the bug, the checksum value is no longer the same as before on little-endian system. The library release after 1.6.4 can still read the dataset created or rewritten with the library of v1.6.2 or before. SLU - 2005/6/30 * For the version 6(6.02 and 6.04) of Portland Group compiler on AMD Opteron processor, there's a bug in the compiler for optimization(-O2). The library failed in several tests but all related to multi driver. The problem has been reported to the vendor. * On IBM AIX systems, parallel HDF5 mode will fail some tests with error messages like "INFO: 0031-XXX ...". This is from the command poe. Set the environment variable MP_INFOLEVEL to 0 to minimize the messages and run the tests again. The tests may fail with messages like "The socket name is already in use". HDF5 does not use sockets (except for stream-VFD). This is due to problems of the poe command trying to set up the debug socket. Check if there are many old /tmp/s.pedb.* staying around. These are sockets used by the poe command and left behind due to failed commands. Ask your system administrator to clean them out. Lastly, request IBM to provide a mean to run poe without the debug socket. * The C++ library's tests fails when compiling with PGI C++ compiler. The workaround until the problem is correctly handled is to use the flag "--instantiate=local" prior to the configure and build steps, as: setenv CXX "pgCC --instantiate=local" for pgCC 5.02 and higher * The stream-vfd test uses ip port 10007 for testing. If another application is already using that port address, the test will hang indefinitely and has to be terminated by the kill command. To try the test again, change the port address in test/stream_test.c to one not being used in the host. * The --enable-static-exec configure flag will only statically link libraries if the static version of that library is present. If only the shared version of a library exists (i.e., most system libraries on Solaris, AIX, and Mac, for example, only have shared versions), the flag should still result in a successful compilation, but note that the installed executables will not be fully static. Thus, the only guarantee on these systems is that the executable is statically linked with just the HDF5 library. * With the gcc 2.95.2 compiler, HDF 5 uses the `-ansi' flag during compilation. The ANSI version of the compiler complains about not being able to handle the `long long' datatype with the warning: warning: ANSI C does not support `long long' This warning is innocuous and can be safely ignored. * Certain platforms give false negatives when testing h5ls: - Cray J90 and Cray T90IEEE give errors during testing when displaying some floating-point values. These are benign differences due to the different precision in the values displayed and h5ls appears to be dumping floating-point numbers correctly. * Not all platforms behave correctly with szip's shared libraries. Szip is disabled in these cases, and a message is relayed at configure time. Static libraries should be working on all systems that support szip, and should be used when shared libraries are unavailable. There is also a configure error on Altix machines that incorrectly reports when a version of szip without an encoder is being used. * On some platforms that use Intel and Absoft compilers to build HDF5 fortran library, compilation may fail for fortranlib_test.f90, fflush1.f90 and fflush2.f90 complaining about exit subroutine. Comment out the line IF (total_error .ne. 0) CALL exit (total_error) * Information about building with PGI and Intel compilers is available in INSTALL file sections 5.7 and 5.8 * On at least one system, (SDSC DataStar), the scheduler (in this case LoadLeveler) sends job status updates to standard error when you run any executable that was compiled with the parallel compilers. This causes problems when running "make check" on parallel builds, as many of the tool tests function by saving the output from test runs, and comparing it to an exemplar. The best solution is to reconfigure the target system so it no longer inserts the extra text. However, this may not be practical. In such cases, one solution is to "setenv HDF5_Make_Ignore yes" prior to the configure and build. This will cause "make check" to continue after detecting errors in the tool tests. However, in the case of SDSC DataStar, it also leaves you with some 150 "failed" tests to examine by hand. A second solution is to write a script to run serial tests and filter out the text added by the scheduler. A sample script used on SDSC DataStar is given below, but you will probably have to customize it for your installation. Observe that the basic idea is to insert the script as the first item on the command line which executes the the test. The script then executes the test and filters out the offending text before passing it on. #!/bin/csh set STDOUT_FILE=~/bin/serial_filter.stdout set STDERR_FILE=~/bin/serial_filter.stderr rm -f $STDOUT_FILE $STDERR_FILE ($* > $STDOUT_FILE) >& $STDERR_FILE set RETURN_VALUE=$status cat $STDOUT_FILE tail +3 $STDERR_FILE exit $RETURN_VALUE You get the HDF make files and test scipts to execute your filter script by setting the environment variable "RUNSERIAL" to the full path of the script prior to running configure for parallel builds. Remember to "unsetenv RUNSERIAL" before running configure for a serial build. Note that the RUNSERIAL environment variable exists so that we can can prefix serial runs as necessary on the target system. On DataStar, no prefix is necessary. However on an MPICH system, the prefix might have to be set to something like "/usr/local/mpi/bin/mpirun -np 1" to get the serial tests to run at all. In such cases, you will have to include the regular prefix in your filter script. * H5Ocopy() does not copy reg_ref attributes correctly when shared-message is turn on. The value of the reference in the destination attriubte is wrong. This H5Ocopy problem will affect h5copy tool