path: root/release_docs/RELEASE.txt

HDF5 version 1.10.5-snap1 currently under development
================================================================================


INTRODUCTION

This document describes the differences between this release and the previous
HDF5 release. It contains information on the platforms tested and known
problems in this release. For more details check the HISTORY*.txt files in the
HDF5 source.

Note that documentation in the links below will be updated at the time of each
final release.

Links to HDF5 documentation can be found on The HDF5 web page:

     https://portal.hdfgroup.org/display/HDF5/HDF5

The official HDF5 releases can be obtained from:

     https://www.hdfgroup.org/downloads/hdf5/

Changes from Release to Release and New Features in the HDF5-1.10.x release series
can be found at:

     https://portal.hdfgroup.org/display/HDF5/HDF5+Application+Developer%27s+Guide

If you have any questions or comments, please send them to the HDF Help Desk:

     help@hdfgroup.org


CONTENTS

- New Features
- Support for new platforms and languages
- Bug Fixes since HDF5-1.10.3
- Supported Platforms
- Tested Configuration Features Summary
- More Tested Platforms
- Known Problems
- CMake vs. Autotools installations


New Features
============

    Configuration:
    -------------
    - Rework CMake configure files to eliminate developer CMP0075 warning

      Renamed varname to HDF5_REQUIRED_LIBRARIES as the contents were not
      required for configuration. Also moved check includes calls to top of
      files.

      (ADB - 2019/01/03, HDFFV-10546)

    - Keep stderr and stdout separate in tests

      Changed test handling of output capture. Tests now keep the stderr
      output separate from the stdout output. It is up to the test to decide
      which output to check against a reference. Also added the option
      to grep for a string in either output.

      (ADB - 2018/12/12, HDFFV-10632)

    - Add toolchain and cross-compile support

      Added info on using a toolchain file to INSTALL_CMAKE.txt. A
      toolchain file is also used in cross-compiling, which requires
      CMAKE_CROSSCOMPILING_EMULATOR to be set. To help with cross-compiling
      the fortran configure process, the HDF5UseFortran.cmake file macros
      were improved. Fixed a Fortran configure file issue that incorrectly
      used #cmakedefine instead of #define.

      (ADB - 2018/10/04, HDFFV-10594)

    - Add warning flags for Intel compilers

      Identified Intel compiler specific warnings flags that should be used
      instead of GNU flags.

      (ADB - 2018/10/04, TRILABS-21)

    - Add default rpath to targets

      Default rpaths should be set in shared executables and
      libraries to allow the use of loading dependent libraries
      without requiring LD_LIBRARY_PATH to be set. The default
      path should be relative using @rpath on osx and $ORIGIN
      on linux. Windows is not affected.

      (ADB - 2018/09/26, HDFFV-10594)

    - Incorrectly installed private header files were removed from
      CMake installs.

      The CMake build files incorrectly flagged the following header files
      as public and installed them. They are private and will no longer be
      installed.

        HDF5 library private package files (H5Xpkg.h)
        H5Edefin.h
        H5Einit.h
        H5Eterm.h
        H5LTparse.h
        h5diff.h
        h5tools_dump.h
        h5tools.h
        h5tools_ref.h
        h5tools_str.h
        h5tools_utils.h
        h5trav.h

      (DER - 2018/10/26, HDFFV-10614, 10609)

    - Autotools installs now install H5FDwindows.h

      This is simply to align the installed header files between the
      autotools and CMake. H5FDwindows.h has no functionality on
      non-Windows systems.

      (DER - 2018/10/26, HDFFV-10614)


    Library:
    --------
    - Added new chunk query functions

      The following public functions were added to discover information about
      the chunks in an HDF5 file.
        herr_t H5Dget_num_chunks(dset_id, fspace_id, *nchunks)
        herr_t H5Dget_chunk_info_by_coord(dset_id, *coord, *filter_mask, *addr, *size)
        herr_t H5Dget_chunk_info(dset_id, fspace_id, index, *coord, *filter_mask, *addr, *size)

      (BMR - 2018/11/07, HDFFV-10615)

    - Allow pre-generated H5Tinit.c and H5make_libsettings.c to be used.

      Rather than always running H5detect and generating H5Tinit.c and
      H5make_libsettings.c, supply a location for those files.

      (ADB - 2018/09/18, HDFFV-10332)

    - Several empty public header files where removed from the distribution

      The following files were empty placeholders. They are for internal
      packages that are unlikely to ever have public functionality and have
      thus been removed.

        H5Bpublic.h
        H5B2public.h
        H5FSpublic.h
        H5HFpublic.h
        H5HGpublic.h
        H5HLpublic.h

      They were only installed in CMake builds.

      (DER - 2018/10/26, HDFFV-10614)

    - Add ability to minimze dataset object headers.

      Creation of many, very small datasets resulted in extensive file bloat
      due to extra space in the dataset object headers -- this space is
      allocated by default to allow for the insertion of a small number of
      attributes within the object header and not require a continuation
      block, an unnecessary provision in the target use case.

      Inform the library to expect no attributes on created datasets, and to
      allocate the least space possible for the object headers.
      NOTE: A continuation block is created if attributes are added to a
      'minimized' dataset, which can reduce performance.
      NOTE: Some extra space is allocated for attributes essential to the
      correct behavior of the object header (store creation times, e.g.). This
      does not violate the design principle, as the space is calculated and
      allocated as needed at the time of dataset object header creation --
      unused space is not generated.
      New API calls:
      H5Fget_dset_no_attrs_hint
      H5Fset_dset_no_attrs_hint
      H5Pget_dset_no_attrs_hint
      H5Pset_dset_no_attrs_hint

      (JOS - 2019/01/04, TRILAB-45)


     Parallel Library:
    -----------------
    - All MPI-1 API calls have been replaced with MPI-2 equivalents.

      This was done to better support OpenMPI, as default builds no longer
      include MPI-1 support (as of OpenMPI 4.0).

      (DER - 2018/12/30, HDFFV-10566)


    Fortran Library:
    ----------------
    - Added wrappers for dataset object header minimization calls.
      (see the note for TRILAB-45, above)

      New API calls:

      h5fget_dset_no_attrs_hint_f
      h5fset_dset_no_attrs_hint_f
      h5pget_dset_no_attrs_hint_f
      h5pset_dset_no_attrs_hint_f

      (DER - 2019/01/09, TRILAB-45)

    C++ Library:
    ------------
    -

    Java Library:
    ----------------
    - Add new functions to java interface

      Added wrappers for:
      H5Fset_libver_bounds
      H5Fget_dset_no_attrs_hint/H5Fset_dset_no_attrs_hint
      H5Pget_dset_no_attrs_hint/H5Pset_dset_no_attrs_hint

      (ADB - 2019/01/07, HDFFV-10664)

    - Fix java unit tests when Time is a natural number

      Time substitution in java/test/junit.sh.in doesn't
      handle the case when Time is a natural number. Fixed
      the regular expression.

      (ADB - 2019/01/07, HDFFV-10674)

    - Duplicate the data read/write functions of Datasets for Attributes.

      Region references could not be displayed for attributes as they could
      for datasets. Datasets had overloaded read and write functions for different
      datatypes that were not available for attributes. After adding similar
      functions, attribute region references work normally.

      (ADB - 2018/12/12, HDFVIEW-4)


    Tools:
    ------
    - The h5repart -family-to-sec2 argument was changed to -family-to-single

      In order to better support other single-file VFDs which could work with
      h5repart, the -family-to-sec2 argument was renamed to -family-to-single.
      This is just a name change and the functionality of the argument has not
      changed.

      The -family-to-sec2 argument has been kept for backwards-compatibility.
      This argument should be considered deprecated.

      (DER - 2018/11/14, HDFFV-10633)

    High-Level APIs:
    ---------------
    -

    C Packet Table API
    ------------------
    -

    Internal header file
    --------------------
    -

    Documentation
    -------------
    -

Support for new platforms, languages and compilers.
=======================================
    -

Bug Fixes since HDF5-1.10.3 release
==================================

    Library
    -------
    - Deleting attributes in dense storage

      The library aborts with "infinite loop closing library" after
      attributes in dense storage are created and then deleted.

      When deleting the attribute nodes from the name index v2 B-tree,
      if an attribute is found in the intermediate B-tree nodes,
      which may be merged/redistributed in the process, we need to 
      free the dynamically allocated spaces for the intermediate 
      decoded attribute.

      (VC - 2018/12/26, HDFFV-10659)

    - A bug was discovered in the parallel library where an application
      would eventually consume all of the available MPI communicators
      when continually writing to a compressed dataset in parallel. This
      was due to internal copies of an HDF5 File Access Property List,
      which each contained a copy of the MPI communicator, not being
      closed at the end of each write operation. This problem was
      exacerbated by larger numbers of processors.

      (JTH - 2018/12/05, HDFFV-10629)

    - Allow H5detect and H5make_libsettings to take a file as an argument.

      Rather than only writing to stdout, add a command argument to name
      the file that H5detect and H5make_libsettings will use for output.
      Without an argument, stdout is still used, so backwards compatibility
      is maintained.

      (ADB - 2018/09/05, HDFFV-9059)

    - A bug was discovered in the parallel library where an application
      would hang if a collective read/write of a chunked dataset occurred
      when collective metadata reads were enabled and some of the ranks
      had no selection in the dataset's dataspace. The ranks which had no
      selection in the dataset's dataspace called H5D__chunk_addrmap() to
      retrieve the lowest chunk address in the dataset. This is because we
      require reads/writes to be performed in strictly non-decreasing order
      of chunk address in the file.

      When the chunk index used was a version 1 or 2 B-tree, these
      non-participating ranks would issue a collective MPI_Bcast() call
      that the participating ranks would not issue, causing the hang. Since
      the non-participating ranks are not actually reading/writing anything,
      the H5D__chunk_addrmap() call can be safely removed and the address used
      for the read/write can be set to an arbitrary number (0 was chosen).

      (JTH - 2018/08/25, HDFFV-10501)

    - There was an incorrect protection against division by zero reported
      to The HDF Group as issue #CVE-2018-17233.

      Protection against division by zero was added to address the issue
      #CVE-2018-17233.  In addition, several similar occurrences in the same
      file were fixed as well.

      (BMR - 2018/02/26, HDFFV-10577)


    Java Library:
    ----------------
    - JNI native library dependencies

      The build for the hdf5_java native library used the wrong
      hdf5 target library for CMake builds. Correcting the hdf5_java
      library to build with the shared hdf5 library required testing
      paths to change also.

      (ADB - 2018/08/31, HDFFV-10568)
     - Java iterator callbacks

      Change global callback object to a small stack structure in order
      to fix a runtime crash. This crash was discovered when iterating
      through a file with nested group members. The global variable
      visit_callback is overwritten when recursion starts. When recursion
      completes, visit_callback will be pointing to the wrong callback method.

      (ADB - 2018/08/15, HDFFV-10536)

    - Java HDFLibraryException class

      Change parent class from Exception to RuntimeException.

      (ADB - 2018/07/30, HDFFV-10534)

    - JNI Read and Write

      Refactored variable-length functions, H5DreadVL and H5AreadVL,
      to correct dataset and attribute reads. New write functions,
      H5DwriteVL and H5AwriteVL, are under construction.

      (ADB - 2018/06/02, HDFFV-10519)

    Fortran
    --------

    - Made Fortran specific subroutines PRIVATE in generic procedures.

      Affected generic procedures were functions in H5A, H5D, H5P, H5R and H5T.

      (MSB, 2018/12/04, HDFFV-10511)

Supported Platforms
===================

    Linux 2.6.32-696.16.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
    #1 SMP ppc64 GNU/Linux        g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
    (ostrich)                     GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
                                  IBM XL C/C++ V13.1
                                  IBM XL Fortran V15.1

    Linux 3.10.0-327.10.1.el7   GNU C (gcc), Fortran (gfortran), C++ (g++)
    #1 SMP x86_64 GNU/Linux       compilers:
    (kituo/moohan)                Version 4.8.5 20150623 (Red Hat 4.8.5-4)
                                    Version 4.9.3, Version 5.2.0
                                  Intel(R) C (icc), C++ (icpc), Fortran (icc)
                                  compilers:
                                     Version 17.0.0.098 Build 20160721
                                  MPICH 3.1.4 compiled with GCC 4.9.3

    SunOS 5.11 32- and 64-bit     Sun C 5.12 SunOS_sparc
    (emu)                         Sun Fortran 95 8.6 SunOS_sparc
                                  Sun C++ 5.12 SunOS_sparc

    Windows 7                     Visual Studio 2015 w/ Intel Fortran 16 (cmake)

    Windows 7 x64                 Visual Studio 2013
                                  Visual Studio 2015 w/ Intel Fortran 16 (cmake)
                                  Visual Studio 2015 w/ Intel C, Fortran 2018 (cmake)
                                  Visual Studio 2015 w/ MSMPI 8 (cmake)

    Windows 10                    Visual Studio 2015 w/ Intel Fortran 18 (cmake)

    Windows 10 x64                Visual Studio 2015 w/ Intel Fortran 18 (cmake)
                                  Visual Studio 2017 w/ Intel Fortran 18 (cmake)

    Mac OS X Yosemite 10.10.5     Apple clang/clang++ version 6.1 from Xcode 7.0
    64-bit                        gfortran GNU Fortran (GCC) 4.9.2
    (osx1010dev/osx1010test)      Intel icc/icpc/ifort version 15.0.3

    Mac OS X El Capitan 10.11.6   Apple clang/clang++ version 7.3.0 from Xcode 7.3
    64-bit                        gfortran GNU Fortran (GCC) 5.2.0
    (osx1011dev/osx1011test)      Intel icc/icpc/ifort version 16.0.2

    Mac OS Sierra 10.12.6         Apple LLVM version 8.1.0 (clang/clang++-802.0.42)
    64-bit                        gfortran GNU Fortran (GCC) 7.1.0
    (swallow/kite)                Intel icc/icpc/ifort version 17.0.2


Tested Configuration Features Summary
=====================================

    In the tables below
          y   = tested
          n   = not tested in this release
          C   = Cluster
          W   = Workstation
          x   = not working in this release
          dna = does not apply
          ( ) = footnote appears below second table
          <blank> = testing incomplete on this feature or platform

Platform                              C         F90/   F90      C++  zlib  SZIP
                                      parallel  F2003  parallel
Solaris2.11 32-bit                      n        y/y    n        y    y     y
Solaris2.11 64-bit                      n        y/n    n        y    y     y
Windows 7                               y        y/y    n        y    y     y
Windows 7 x64                           y        y/y    y        y    y     y
Windows 7 Cygwin                        n        y/n    n        y    y     y
Windows 7 x64 Cygwin                    n        y/n    n        y    y     y
Windows 10                              y        y/y    n        y    y     y
Windows 10 x64                          y        y/y    n        y    y     y
Mac OS X Mavericks 10.9.5 64-bit        n        y/y    n        y    y     y
Mac OS X Yosemite 10.10.5 64-bit        n        y/y    n        y    y     y
Mac OS X El Capitan 10.11.6 64-bit      n        y/y    n        y    y     y
Mac OS Sierra 10.12.6 64-bit            n        y/y    n        y    y     y
CentOS 7.2 Linux 2.6.32 x86_64 PGI      n        y/y    n        y    y     y
CentOS 7.2 Linux 2.6.32 x86_64 GNU      y        y/y    y        y    y     y
CentOS 7.2 Linux 2.6.32 x86_64 Intel    n        y/y    n        y    y     y
Linux 2.6.32-573.18.1.el6.ppc64         n        y/y    n        y    y     y


Platform                                 Shared  Shared    Shared    Thread-
                                         C libs  F90 libs  C++ libs  safe
Solaris2.11 32-bit                         y       y         y         y
Solaris2.11 64-bit                         y       y         y         y
Windows 7                                  y       y         y         y
Windows 7 x64                              y       y         y         y
Windows 7 Cygwin                           n       n         n         y
Windows 7 x64 Cygwin                       n       n         n         y
Windows 10                                 y       y         y         y
Windows 10 x64                             y       y         y         y
Mac OS X Mavericks 10.9.5 64-bit           y       n         y         y
Mac OS X Yosemite 10.10.5 64-bit           y       n         y         y
Mac OS X El Capitan 10.11.6 64-bit         y       n         y         y
Mac OS Sierra 10.12.6 64-bit               y       n         y         y
CentOS 7.2 Linux 2.6.32 x86_64 PGI         y       y         y         n
CentOS 7.2 Linux 2.6.32 x86_64 GNU         y       y         y         y
CentOS 7.2 Linux 2.6.32 x86_64 Intel       y       y         y         n
Linux 2.6.32-573.18.1.el6.ppc64            y       y         y         n

Compiler versions for each platform are listed in the preceding
"Supported Platforms" table.


More Tested Platforms
=====================
The following platforms are not supported but have been tested for this release.

    Linux 2.6.32-573.22.1.el6    GNU C (gcc), Fortran (gfortran), C++ (g++)
    #1 SMP x86_64 GNU/Linux       compilers:
    (mayll/platypus)                 Version 4.4.7 20120313
                                     Version 4.9.3, 5.3.0, 6.2.0
                                  PGI C, Fortran, C++ for 64-bit target on
                                  x86-64;
                                     Version 17.10-0
                                  Intel(R) C (icc), C++ (icpc), Fortran (icc)
                                  compilers:
                                     Version 17.0.4.196 Build 20170411
                                  MPICH 3.1.4 compiled with GCC 4.9.3

    Linux 3.10.0-327.18.2.el7     GNU C (gcc) and C++ (g++) compilers
    #1 SMP x86_64 GNU/Linux          Version 4.8.5 20150623 (Red Hat 4.8.5-4)
    (jelly)                       with NAG Fortran Compiler Release 6.1(Tozai)
                                  GCC Version 7.1.0
                                  OpenMPI 3.0.0-GCC-7.2.0-2.29,
                                     3.1.0-GCC-7.2.0-2.29
                                  Intel(R) C (icc) and C++ (icpc) compilers
                                     Version 17.0.0.098 Build 20160721
                                  with NAG Fortran Compiler Release 6.1(Tozai)

    Linux 3.10.0-327.10.1.el7     MPICH 3.2 compiled with GCC 5.3.0
    #1 SMP x86_64 GNU/Linux
    (moohan)

    Linux 2.6.32-573.18.1.el6.ppc64  MPICH mpich 3.1.4 compiled with
    #1 SMP ppc64 GNU/Linux           IBM XL C/C++ for Linux, V13.1
    (ostrich)                        and IBM XL Fortran for Linux, V15.1

    Debian 8.4 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1 x86_64 GNU/Linux
                                  gcc, g++ (Debian 4.9.2-10) 4.9.2
                                  GNU Fortran (Debian 4.9.2-10) 4.9.2
                                  (cmake and autotools)

    Fedora 24  4.7.2-201.fc24.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
                                  gcc, g++ (GCC) 6.1.1 20160621
                                      (Red Hat 6.1.1-3)
                                  GNU Fortran (GCC) 6.1.1 20160621
                                      (Red Hat 6.1.1-3)
                                  (cmake and autotools)

    Ubuntu 16.04.1 4.4.0-38-generic #57-Ubuntu SMP x86_64 GNU/Linux
                                  gcc, g++ (Ubuntu 5.4.0-6ubuntu1~16.04.2)
                                      5.4.0 20160609
                                  GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.2)
                                      5.4.0 20160609
                                  (cmake and autotools)


Known Problems
==============

    At present, metadata cache images may not be generated by parallel
    applications.  Parallel applications can read files with metadata cache
    images, but since this is a collective operation, a deadlock is possible
    if one or more processes do not participate.

    Three tests fail with OpenMPI 3.0.0/GCC-7.2.0-2.29:
        testphdf5 (ecdsetw, selnone, cchunk1, cchunk3, cchunk4, and actualio)
        t_shapesame (sscontig2)
        t_pflush1/fails on exit
    The first two tests fail attempting collective writes.

    Known problems in previous releases can be found in the HISTORY*.txt files
    in the HDF5 source. Please report any new problems found to
    help@hdfgroup.org.


CMake vs. Autotools installations
=================================
While both build systems produce similar results, there are differences.
Each system produces the same set of folders on linux (only CMake works
on standard Windows); bin, include, lib and share. Autotools places the
COPYING and RELEASE.txt file in the root folder, CMake places them in
the share folder.

The bin folder contains the tools and the build scripts. Additionally, CMake
creates dynamic versions of the tools with the suffix "-shared". Autotools
installs one set of tools depending on the "--enable-shared" configuration
option.
  build scripts
  -------------
  Autotools: h5c++, h5cc, h5fc
  CMake: h5c++, h5cc, h5hlc++, h5hlcc

The include folder holds the header files and the fortran mod files. CMake
places the fortran mod files into separate shared and static subfolders,
while Autotools places one set of mod files into the include folder. Because
CMake produces a tools library, the header files for tools will appear in
the include folder.

The lib folder contains the library files, and CMake adds the pkgconfig
subfolder with the hdf5*.pc files used by the bin/build scripts created by
the CMake build. CMake separates the C interface code from the fortran code by
creating C-stub libraries for each Fortran library. In addition, only CMake
installs the tools library. The names of the szip libraries are different
between the build systems.

The share folder will have the most differences because CMake builds include
a number of CMake specific files for support of CMake's find_package and support
for the HDF5 Examples CMake project.