HDF5 version 1.7.33 currently under development ================================================================================ INTRODUCTION This document describes the differences between HDF5-1.6.* and HDF5-1.7.*, and contains information on the platforms tested and known problems in HDF5-1.7.*. For more details check the HISTORY.txt file in the HDF5 source. The HDF5 documentation can be found on the NCSA ftp server (ftp.ncsa.uiuc.edu) in the directory: /HDF/HDF5/docs/ For more information look at the HDF5 home page at: http://hdf.ncsa.uiuc.edu/HDF5/ If you have any questions or comments, please send them to: hdfhelp@ncsa.uiuc.edu CONTENTS - New Features - Support for new platforms and languages - Bug Fixes since HDF5-1.6.0 - Platforms Tested - Known Problems New Features ============ Configuration: -------------- - Added some initial support for making purify (or similar memory checking products) happier by initializing buffers to zero and disabling the internal free list code. To take advantage of this, define 'H5_USING_PURIFY' in your CFLAGS when building the library. QAK - 2004/07/23 - Fixed the long compile time of H5detect.c when v7.x Intel Compiler is used with optimization NOT off. AKC - 2004/05/20 - Fixed configure setting of C++ for OSF1 platform. AKC - 2004/01/06 - Prefix default is changed from /usr/local to `pwd`/hdf5. AKC - 2003/07/09 Library: -------- - 4 new API functions, H5Tencode, H5Tdecode, H5Sencode, H5Sdecode were added to the library. Given object ID, these functions encode and decode HDF5 objects(data type and space) information into and from binary buffer. SLU - 2004/07/21 - Modified the way how HDF5 calculates 'pixels_per_scanline' parameter for SZIP compression. Now there is no restriction on the size and shape of the chunk except that the total number of elements in the chunk cannot be bigger than 'pixels_per_block' parameter provided by the user. EIP - 2004/07/21 - Added support for SZIP without encoder. Added H5Zget_filter_info and changed H5Pget_filter and H5Pget_filter_by_id to support this change. JL/NF - 2004/06/30 - SZIP always uses K13 compression. This flag no longer needs to be set when calling H5Pset_szip. If the flag for CHIP compression is set, it will be ignored (since the two are mutually exclusive). JL/NF - 2004/6/30 - A new API function H5Fget_name was added. It returns the name of the file by object(file, group, data set, named data type, attribute) ID. SLU - 2004/06/29 - Added support for user defined identifier types. NF/JL - 2004/06/29 - A new API function H5Fget_filesize was added. It returns the actual file size of the opened file. SLU - 2004/06/24 - New Feature of Data transformation is added. AKC - 2004/05/03. - New exception handler for datatype conversion is put in to replace the old overflow callback function. This exception handler is set through H5Pset_type_conv_cb function. SLU - 2004/4/27 - Added option that if $HDF5_DISABLE_VERSION_CHECK is set to 2, will suppress all library version mismatch warning messages. AKC - 2004/4/14 - A new type of dataspace, null dataspace(dataspace without any element) was added. SLU - 2004/3/24 - Data type conversion(software) from integer to float was added. SLU - 2004/3/13 - Data type conversion(software) from float to integer was added. Conversion from integer to float will be added later. SLU -2004/2/4 - Added new H5Premove_filter routine to remove I/O pipeline filters from dataset creation property lists. PVN - 2004/01/26 - Added new 'compare' callback parameter to H5Pregister & H5Pinsert routines. QAK - 2004/01/07 - Data type conversion(hardware) between integers and floats was added. SLU 2003/11/21 - New function H5Iget_file_id() was added. It returns file ID given an object(dataset, group, or attribute) ID. SLU 2003/10/29 - Added new fields to the H5G_stat_t for more information about an object's object header. QAK 2003/10/06 - Added new H5Fget_freespace() routine to query the free space in a given file. QAK 2003/10/06 - Added backward compatability with v1.6 for new Error API. SLU - 2003/09/24 - Changed 'objno' field in H5G_stat_t structure from 'unsigned long[2]' to 'haddr_t'. QAK - 2003/08/08 - Changed 'fileno' field in H5G_stat_t structure from 'unsigned long[2]' to 'unsigned long'. QAK - 2003/08/08 - Changed 'hobj_ref_t' type from structure with array field to 'haddr_t'. QAK - 2003/08/08 - Object references (hobj_ref_t) can now be compared with the 'objno' field in the H5G_stat_t struct for testing if two objects are the same within a file. QAK - 2003/08/08 - Switched over to new error API. SLU - 2003/07/25 Parallel Library: ----------------- - Add options of using atomicity and file-sync to test_mpio_1wMr. AKC - 2003/11/13 - Added parallel test, test_mpio_1wMr, which tests if the underlaying parallel I/O system is conforming to the POSIX write/read requirement. AKC - 2003/11/12 Fortran Library: ---------------- - added new functions h5fget_name_f and h5fget_filesize_f EIP 2004/07/08 - h5dwrite/read_f and h5awrite/read_f functions only accept dims parameter of the type INTEGER(HSIZE_T). Tools: ------ - h5repack.sh did not report errors encountered during tests. It does now. AKC - 2004/04/02 - Added the MPI-I/O and MPI-POSIX drivers to the list of VFL drivers available for h5dump and h5ls - Added option --vfd= to h5ls to allow a VFL driver to be selected by a user. - Added option -showconfig to compiler tools (h5cc,h5fc,h5c++). AKC - 2004/01/08 - Install the "h5cc" and "h5fc" tools as "h5pcc" and "h5pfc" respectively if library is built in parallel mode. WCW - 2003/11/04 - Added metadata benchmark (perform/perf_meta). SLU - 2003/10/03 - Changed output of "OID"s from h5dump from "-" to ":::" to ":" QAK - 2003/08/08 Support for new platforms, languages and compilers. ======================================= - Absoft compiler f95 v9.0 supported on Linux 2.4 EIP - 2004/07/29 - HDF5 Fortran APIs are supported on Mac OSX with IBM XL Fortran compiler version 8.1. This is a default compiler. - HDF5 Fortran APIs are supported on MAC OSX with Absoft F95 compiler version 8.2; set F9X environment varibale to f95, for example setenv F9X f95 Use --disable-shared --enable-static configure flags when Absoft compiler is used. EIP - 2004/07/27 - HDF5 Fortran APIs are supported on MAC OSX with IBM XL Fortran Compiler version 8.1 Use "--disable-shared --enable-static" configure flags along with the "--enable-fortran" flag to build Fortran library. EIP - 2004/01/07 Bug Fixes since HDF5-1.6.0 release ================================== Library ------- - Fixed obscure bug where a filter which failed during chunk allocation could allow library to write uncompressed data to disk but think the data was compressed. QAK - 2004/07/29 - Fixed bug where I/O to an extendible chunked dataset with zero-sized dimensions would cause library to fail an assertion. QAK - 2004/07/27 - Fixed bug where chunked datasets which have filters defined, allocation time set to "late" and whose chunks don't align with the dataspace bounds could have incorrect data stored when overwriting the entire dataset on the first write. QAK - 2004/07/27 - Added check to ensure that dataspaces have extents set. JML-2004/07/26 - Fixed bug on some Solaris systems where HDF5 would try to use gettimeofday() when that function didn't work properly. JML - 2004/07/23 - Fixed bug in H5Sset_extent_simple where setting maximum size to non-zero, then to zero would cause an error. JML - 2004/07/20 - Allow NULL pointer for buffer parameter to H5Dread & H5Dwrite when not writing data ("none" selection or hyperslab or point selection with no elements defined). QAK - 2004/07/20 - Calling H5Gcreate() on "/" or "." throws an error instead of failing quietly. JML - 2004/07/19 - Fixed bug where setting file address size to be very small could trigger an assert if the file grew to more than 64 KB. Now throws an error and data can be recovered. JL/NF - 2004/07/14 - Fixed bug where "resurrecting" a dataset was failing. QAK - 2004/07/14 - Fixed bug where incorrect data could be read from a chunked dataset after it was extended. QAK - 2004/07/12 - Fixed failure to read data back from file of compound type with variable-length string as field. SLU - 2004/06/10 - Fixed potential file corruption bug when a block of metadata could overlap the end of the internal metadata accumulator buffer and the buffer would be extended correctly, but would incorrectly change it's starting address. QAK - 2004/06/09 - Opaque datatype with no tag failed for some operations. Fixed. SLU - 2004/6/3 - Fixed potential file corruption bug where dimensions that were too large (a value greater than could be represented in 32-bits) could cause the incorrect amount of space to be allocated in a file for the raw data for the dataset. QAK - 2004/06/01 - Fixed dtypes "sw long double -> double" failure in QSC class machines. AKC - 2004/4/16 - Fixed problem with fletcher32 filter when converting data of different endianess. PVN - 2004/03/10 - Fixed problem with H5Tget_native_type() not handling opaque fields correctly. QAK - 2004/01/31 - Fixed several errors in B-tree deletion code which could cause a B-tree (used with groups and chunked datasets) to become corrupt with the right sequence of deleted objects. QAK - 2004/01/19 - Fixed small internal memory leaks of fill-value information. QAK - 2004/01/13 - Fixed bug that caused variable-length datatypes (strings or sequences) used for datasets in files with objects that were unlinked to fail to be read/written to a file. QAK - 2004/01/13 - Detect situation where szip 'pixels per block' is larger than the fastest changing dimension of a dataset's chunk size and disallow this (due to limits in szip library). QAK - 2003/12/31 - Fixed bug with flattened hyperslab selections that would generate incorrect hyperslab information with certain high-dimensionality combinations of start/stride/count/block information. QAK - 2003/12/31 - Fixed bug with variable-length datatypes used in compound datatypes. SLU - 2003/12/29 - Fixed bug in parallel I/O routines that would cause reads from "short datasets" (datasets which were only partially written out) to return invalid data. QAK & AKC - 2003/12/19 - Fixed bug where scalar dataspaces for attributes were reporting as simple dataspaces. QAK - 2003/12/13 - Fixed problem with selection offsets of hyperslab selections in chunked datasets causing the library to go into an infinite loop. QAK - 2003/12/13 - Fixed H5Giterate to avoid re-using index parameter after iteration callback has been called (allows iteration callback to modify the index parameter itself). QAK - 2003/12/06 - Fixed various floating-point conversion problems, including a change which could corrupt data when converting from double->float. QAK - 2003/11/24 - Changed "single process" metadata writing in library to collective I/O by all processes, in order to guarantee correct data being written with MPI-I/O. QAK - 2003/11/20 - Fixed problems with fill values and variable-length types and also I/O on VL values that were set to NULL. QAK - 2003/11/08 - Fixed problems with MPI datatypes that caused ASCI Q machine to hang. QAK - 2003/10/28 - Removed HDF5_MPI_PREFER_DERIVED_TYPES environment variable support, since it had no benefit. QAK - 2003/10/28 - Single hyperslab selections (which were set with only one call to H5Sselect_hyperslab) that had dimensions that could be "flattened" but were interspersed with dimensions that could not be flattened were not correctly handled, causing core dumps. QAK - 2003/10/25 - Fixed incorrect datatype of the third parameter to the Fortran90 h5pset(get)_cache_f subroutine (INTEGER to INTEGER(SIZE_T)) EIP - 2003/10/13 - Fixed problems with accessing variable-length data datatypes on Crays. QAK - 2003/10/10 - Fixed potential file corruption bug when too many object header messages (probably attributes, from a user perspective) were inserted into an object header and certain other conditions were met. QAK - 2003/10/08 - Changed implementation of internal ID searching algorithm to avoid O(n) behavior for many common cases. QAK - 2003/10/06 - Allow partial parallel writing to compact datasets. QAK - 2003/10/06 - Correctly create reference to shared datatype in attribute, instead of making a copy of the shared datatype in the attribute. QAK - 2003/10/01 - Revert changes which caused files >2GB to fail when created with MPI-I/O file driver on certain platforms. QAK - 2003/09/16 - Allow compound datatypes to grow in size. SLU - 2003/09/10 - Detect if a type is already packed before attempting to pack it again or check if it is locked. SLU - 2003/09/10 - Corrected bug when opening a file twice with read-only permission for one open and then closing the read-only access file ID would generate an error. QAK - 2003/09/10 - Corrected bug in repeated calls to H5Pget_access_plist() which would incorrectly manage reference counts of internal information and eventually blow up. QAK - 2003/09/02 - Return rank of the array datatype on successful call to H5Tget_array_dims(). QAK - 2003/08/30 - Corrected bug in H5Tdetect_class which was not correctly detecting datatype classes of fields in nested compound datatypes in some circumstances. QAK - 2003/08/30 - Corrected bug in sieve buffer code which could cause loss of data when a small dataset was created and deleted in quick succession. QAK - 2003/08/27 - Corrected bug in H5Gget_objname_by_idx which was not allowing NULL for the name when just querying for the object name's length. QAK - 2003/08/25 - Corrected bug in variable-length string handling which could generate a core dump on writing variable-length strings as part of a compound datatype on certain architectures. QAK - 2003/08/25 - Corrected bug in H5Tget_native_type which would incorrectly compute the size of certain compound datatypes and also incorrectly compute the offset of the last field for those compound datatypes. QAK - 2003/08/25 - Corrected bug in H5Tget_native_type which would drop string datatype metadata (padding, etc.) QAK - 2003/08/25 - Corrected bugs in H5Gget_num_objs, H5Gget_objname_by_idx and H5Gget_objtype_by_idx to allow them to accept location IDs, not just group IDs. QAK - 2003/08/21 - Corrected bug when using scalar dataspace for memory selection and operating on chunked dataset. QAK - 2003/08/18 - Corrected bugs with multiple '/' characters in names for H5Glink and H5Gunlink. QAK - 2003/08/16 - Corrected bug with user blocks that didn't allow a user block to be inserted in front of a file after the file was created. QAK - 2003/08/13 - Corrected errors with using point selections to access data in chunked datasets. QAK - 2003/07/23 - Corrected error with variable-length datatypes and chunked datasets caused H5Dwrite to fail sometimes. QAK - 2003/07/19 - Modified library and file format to support storing indexed storage (chunked dataset) B-tree's with non-default internal 'K' values. QAK - 2003/07/15 - Returned H5T_BKG_TEMP support to library after it was accidentally removed. QAK - 2003/07/14 Configuration ------------- - Parallel I/O with the MPI-I/O driver will no longer work if the filesystem is not POSIX compliant. The "HDF5_MPI_1_METAWRITE" environment variable has been removed. QAK - 2004/01/30 - Fixed the error that cause "make install" to fail because of the macro definition syntax of "prefix?=..." AKC - 2003/07/22 Performance ------------- - More optimizations to inner loops of datatype conversions for integers and floats which give a 10-50% speedup. QAK - 2003/11/07 - Hoisted invariant 'if/else's out of inner datatype conversion loop for integer and floating-point values, giving about a 20% speedup. QAK - 2003/10/20 Tools ----- - Fixed h5redeploy which sometimes complain too many argument for the test command. (The complain did not hinder the h5redploy to proceed correctly.) AKC - 2003/11/03 - Fixed a segmentation fault of h5diff when percentage option is used. AKC - 2003/08/27 - Switched away from tools using internal "fixtype" function(s) to use H5Tget_native_type() internally. QAK - 2003/08/25 Documentation ------------- Platforms Tested ================ AIX 5.1 (32 and 64-bit) xlc 6.0.0.2 xlf 8.1.0.3 xlC 6.0.0.4 xlc 5.0.2.5 xlf 7.1.1.2 xlC 5.0.2.5 mpcc_r 5.0.2.5 mpxlf_r 7.1.1.2 poe 3.2.0.10 Cray T3E sn6606 2.0.6.08 Cray Standard C Version 6.6.0.2 Cray Fortran Version 3.6.0.0.2 mpt 2.2.0.0 Cray SV1 sn9617 10.0.1.2 Cray Standard C Version 6.6.0.2 mpt 2.2.0.0 Cray Fortran Version 3.6.0.0.2 Cray T90IEEE 10.0.1.01y Cray Standard C Version 6.4.0.2.3 Cray Fortran Version 3.4.0.3 mpt 2.1.0.0 FreeBSD 4.9 gcc 2.95.4 g++ 2.95.4 HP-UX B.11.00 HP C HP92453-01 A.11.01.20 HP F90 v2.4 HP ANSI C++ B3910B A.03.13 MPIch 1.2.4 IRIX 6.5 MIPSpro cc 7.30 IRIX64 6.5 (64 & n32) MIPSpro cc 7.3.1.3m F90 MIPSpro 7.3.1.3m (64 only) MPIch 1.2.4 Linux 2.4.18 gcc 2.96, 3.2.2, 3.2.3 g++ 3.2.2, 3.2.3 Intel(R) C++ Version 7.1 Intel(R) Fortran Compiler Version 7.1 PGI compilers (pgcc, pgf90, pgCC) version 5.0-2 MPIch 1.2.4 Absoft Fortran v9.0 OSF1 V5.1 Compaq C V6.4-014 Compaq C V6.3-027 Compaq Fortran V5.5-1877 Compaq C++ V6.5-014 MPI_64bit_R5 g++ version 3.0 for C++ SunOS 5.7 WorkShop Compilers 5.0 98/12/15 C 5.0 (Solaris 2.7) WorkShop Compilers 5.0 98/12/15 C++ 5.0 WorkShop Compilers 5.0 98/10/25 FORTRAN 90 2.0 Patch 107356-04 SunOS 5.8/32 Sun WorkShop 6 update 2 C 5.3 (Solaris 2.8) Sun WorkShop 6 update 2 Fortran 90 Sun WorkShop 6 update 2 C++ 5.3 SunOS 5.8/64 Sun WorkShop 6 update 2 C 5.3 (Solaris 2.8) Sun WorkShop 6 update 2 Fortran 90 Sun WorkShop 6 update 2 C++ 5.3 TFLOPS r1.0.4 v4.3.3 i386 pgcc Rel 3.1-4i with mpich-1.2.4 with local modifications IA-32 Linux 2.4.9 gcc 2.96 Intel(R) C++ Version 7.0 Intel(R) Fortran Compiler Version 7.0 IA-64 Linux 2.4.16 ia64 gcc version 2.96 20000731 Intel(R) C++ Version 7.0 Intel(R) Fortran Compiler Version 7.0 Windows 2000 (NT5.0) MSVC++ 6.0 DEC Visual Fortran 6.0 Intel C and F90 compilers version 7.1 Code Warrior 8.0 Windows XP MSVC++.NET MAC OS X Darwin 6.5 gcc and g++ Apple Computer, Inc. GCC version 1161, based on gcc version 3.1 IBM XL Fortran compiler version 8.1 Absoft Fortran v8.2 Supported Configuration Features Summary ======================================== In the tables below y = tested and supported n = not supported or not tested in this release x = not working in this release dna = does not apply ( ) = footnote appears below second table Platform C C F90 F90 C++ Shared zlib parallel parallel libraries (4) Solaris2.7 64-bit y y (1) y y (1) y y y Solaris2.7 32-bit y y (1) y y (1) y y y Solaris2.8 64-bit y y (1) y y (1) y y y Solaris2.8 32-bit y y y y (1) y y y IRIX6.5 y y (1) n n n y y IRIX64_6.5 64-bit y y (2) y y y y y IRIX64_6.5 32-bit y y (2) n n n y y HPUX11.00 y y (1) y y y y y OSF1 v5.1 y y y y y y y T3E y y (5) y y (5) n n y SV1 y y (5) y y (5) n n y T90 IEEE y y (5) y y (5) n n y TFLOPS n y (1) n n n n y AIX-5.1 32-bit y y y y y n y AIX-5.1 64-bit y y y y y n y WinXP (6) y n n n y y y WinXP Intel y n n n y y y Win2000 y n y n y y y Win2000 Intel y n y n y y y WinNT CW y n n n n n y Mac OS X 10.2 y n n n y y y FreeBSD y y (1) n n y y y Linux 2.4 gcc (3) y y (1) y (PGI) n y y y Linux 2.4 Intel (3) y n y n y n y Linux 2.4 PGI (3) y n y n y n y Linux 2.4 IA32 Intel y n y n y n y Linux 2.4 IA64 Intel y n y n y n y ASCII Table 2 -- for RELEASE.txt Platform static- Thread- SZIP GASS STREAM- High-level H4/H5 exec safe VFD APIs tools (7) Solaris2.7 64-bit x y y n y y n Solaris2.7 32-bit x y y n y y y Solaris2.8 64-bit x y y n y y n Solaris2.8 32-bit x y y n y y y IRIX6.5 x n y n y y y IRIX64_6.5 64-bit x y y y y y y IRIX64_6.5 32-bit x y y y y y y HPUX11.00 x n y n y y y OSF1 v5.1 y n y n y y y T3E y n n n y y y SV1 y n n n y y y T90 IEEE y n n n y y n TFLOPS y n n n n n n AIX-5.1 32-bit y n y n y y y AIX-5.1 64-bit y n y n y y y WinXP (6) y n y n n y y WinXP Intel y n y n n y y Win2000 y n y n n y y Win2000 Intel y n y n n y y WinNT CW y n y n n y y Mac OS X 10.2 y n y n y y n FreeBSD y y y n y y y Linux 2.4 gcc (3) y y y n y y y Linux 2.4 Intel (3) y n y n y n n Linux 2.4 PGI (3) y n y n y n n Linux 2.4 IA32 Intel y n y n y y y Linux 2.4 IA64 Intel y n y n y y y Notes: (1) Using mpich 1.2.4. (2) Using mpt and mpich 1.2.4. (3) Linux 2.4 with GNU, Intel, and PGI compilers, respectively. (4) Shared libraries are provided only for the C library, except on Windows where they are provided for C and C++. (5) Using mpt. (6) Binaries only; source code for this platform is not being released at this time. (7) Includes the H4toH5 Library and the h4toh5 and h5toh4 utilities. Compiler versions for each platform are listed in the preceding "Platforms Tested" table. Known Problems ============== * PGI C++ compiler fails when compiling the C++ library's tests. Therefore, we cannot verify that the C++ library built with the PGI C++ compiler is correct. * The h5dump tests may fail to match the expected output on some platforms (e.g. parallel jobs, Windows) where the error messages directed to "stderr" do not appear in the "right order" with output from stdout. This is not an error. * The stream-vfd test uses ip port 10007 for testing. If another application is already using that port address, the test will hang indefinitely and has to be terminated by the kill command. To try the test again, change the port address in test/stream_test.c to one not being used in the host. * The --enable-static-exec configure flag fails to compile for Solaris platforms. This is due to the fact that not all of the system libraries on Solaris are available in a static format. The --enable-static-exec configure flag also fails to correctly compile on IBM SP2 platform for the serial mode. The parallel mode works fine with this option. It is suggested that you don't use this option on these platforms during configuration. * With the gcc 2.95.2 compiler, HDF 5 uses the `-ansi' flag during compilation. The ANSI version of the compiler complains about not being able to handle the `long long' datatype with the warning: warning: ANSI C does not support `long long' This warning is innocuous and can be safely ignored. * The Stream VFD was not tested yet under Windows. It is not supported in the TFLOPS machine. * The ./dsets tests failed in the TFLOPS machine if the test program, dsets.c, is compiled with the -O option. The hdf5 library still works correctly with the -O option. The test program works fine if it is compiled with -O1 or -O0. Only -O (same as -O2) causes the test program to fail. * Certain platforms give false negatives when testing h5ls: - Cray J90 and Cray T90IEEE give errors during testing when displaying some floating-point values. These are benign differences due to the different precision in the values displayed and h5ls appears to be dumping floating-point numbers correctly. * Before building HDF5 F90 Library from source on Crays replace H5Aff.f90, H5Dff.f90 and H5Pff.f90 files in the fortran/src subdirectory in the top level directory with the Cray-specific files from the site: ftp://hdf.ncsa.uiuc.edu/pub/outgoing/hdf5/hdf5-1.6.2/F90_source_for_Crays * On some platforms that use Intel and Absoft compilers to build HDF5 fortran library, compilation may fail for fortranlib_test.f90, fflush1.f90 and fflush2.f90 complaining about exit subroutine. Comment out the line IF (total_error .ne. 0) CALL exit (total_error) * On IA32 and IA64 systems, if you use a compiler other than GCC (such as Intel's ecc or icc compilers), you will need to modify the generated "libtool" program after configuration is finished. On or around line 104 of the libtool file, there are lines which look like: # How to pass a linker flag through the compiler. wl="" change these lines to this: # How to pass a linker flag through the compiler. wl="-Wl," UPDATE: This is now done automatically by the configure script. However, if you still experience a problem, you may want to check this line in the libtool file and make sure that it has the correct value. * Information about building with PGI and Intel compilers is available in INSTALL file sections 5.7 and 5.8