Merge branch 'develop' into cu_devinactive/cu_dev

1 files changed, 319 insertions, 6 deletions

diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt
index 665a156..1c5d96d 100644
--- a/release_docs/RELEASE.txt
+++ b/release_docs/RELEASE.txt
@@ -48,6 +48,56 @@ New Features
 
     Configuration:
     -------------
+    - CMake option to build the HDF filter plugins project as an external project
+
+      The HDF filter plugins project is a collection of registered compression
+      filters that can be dynamically loaded when needed to access data stored
+      in a hdf5 file. This CMake-only option allows the plugins to be built and
+      distributed with the hdf5 library and tools. Like the options for szip and
+      zlib, either a tgz file or a git repository can be specified for the source.
+
+      The necessary options are (see the INSTALL_CMake.txt file):
+          HDF5_ENABLE_PLUGIN_SUPPORT
+          PLUGIN_TGZ_NAME or PLUGIN_GIT_URL
+      There are more options necessary for various filters and the plugin project
+      documents should be referenced.
+
+      (ADB - 2020/09/27, OESS-98)
+
+    - Added CMake option to format source files
+
+      HDF5_ENABLE_FORMATTERS option will enable creation of targets using the
+      pattern - HDF5_*_SRC_FORMAT - where * corresponds to the source folder
+      or tool folder. All sources can be formatted by executing the format target;
+         make format
+
+      (ADB - 2020/08/24)
+
+    - CMake option to link the generated Fortran MOD files into the include
+      directory.
+
+      The Fortran generation of MOD files by a Fortran compile can produce
+      different binary files between SHARED and STATIC compiles with different
+      compilers and/or different platforms. Note that it has been found that
+      different versions of Fortran compilers will produce incompatible MOD
+      files. Currently, CMake will locate these MOD files in subfolders of
+      the include directory and add that path to the Fortran library target
+      in the CMake config file, which can be used by the CMake find library
+      process. For other build systems using the binary from a CMake install,
+      a new CMake configuration can be used to copy the pre-chosen version
+      of the Fortran MOD files into the install include directory.
+
+      The default will depend on the configuration of
+      BUILD_STATIC_LIBS and BUILD_SHARED_LIBS:
+            YES                   YES         Default to SHARED
+            YES                   NO          Default to STATIC
+            NO                    YES         Default to SHARED
+            NO                    NO          Default to SHARED
+      The defaults can be overriden by setting the config option
+         HDF5_INSTALL_MOD_FORTRAN to one of NO, SHARED, or STATIC
+
+      (ADB - 2020/07/9, HDFFV-11116)
+
     - CMake option to use AEC (open source SZip) library instead of SZip
 
       The open source AEC library is a replacement library for SZip. In
@@ -305,9 +355,102 @@ New Features
 
       (ADB - 2018/07/16)
 
+    - Add file locking configure and CMake options
+
+      HDF5 1.10.0 introduced a file locking scheme, primarily to help
+      enforce SWMR setup. Formerly, the only user-level control of the scheme
+      was via the HDF5_USE_FILE_LOCKING environment variable.
+
+      This change introduces configure-time options that control whether
+      or not file locking will be used and whether or not the library
+      ignores errors when locking has been disabled on the file system
+      (useful on some HPC Lustre installations).
+
+      In both the Autotools and CMake, the settings have the effect of changing
+      the default property list settings (see the H5Pset/get_file_locking()
+      entry, below).
+
+      The yes/no/best-effort file locking configure setting has also been
+      added to the libhdf5.settings file.
+
+      Autotools:
+
+        An --enable-file-locking=(yes|no|best-effort) option has been added.
+
+        yes:          Use file locking.
+        no:           Do not use file locking.
+        best-effort:  Use file locking and ignore "disabled" errors.
+
+      CMake:
+
+        Two self-explanatory options have been added:
+
+        HDF5_USE_FILE_LOCKING
+        HDF5_IGNORE_DISABLED_FILE_LOCKS
+
+        Setting both of these to ON is the equivalent to the Autotools'
+        best-effort setting.
+
+      NOTE:
+      The precedence order of the various file locking control mechanisms is:
+
+        1) HDF5_USE_FILE_LOCKING environment variable (highest)
+
+        2) H5Pset_file_locking()
+
+        3) configure/CMake options (which set the property list defaults)
+
+        4) library defaults (currently best-effort)
+
+      (DER - 2020/07/30, HDFFV-11092)
+
 
     Library:
     --------
+    - Add new public function H5Ssel_iter_reset
+
+      This function resets a dataspace selection iterator back to an
+      initial state so that it may be used for iteration once more.
+      This can be useful when needing to iterate over a selection
+      multiple times without having to repeatedly create/destroy
+      a selection iterator for that dataspace selection.
+
+      (JTH - 2020/09/18)
+
+    - Remove HDFS VFD stubs
+
+      The original implementation of the HDFS VFD included non-functional
+      versions of the following public API calls when the HDFS VFD is
+      not built as a part of the HDF5 library:
+
+      * H5FD_hdfs_init()
+      * H5Pget_fapl_hdfs()
+      * H5Pset_fapl_hdfs()
+
+      They will remain present in HDF5 1.10 and HDF5 1.12 releases
+      for binary compatibility purposes but have been removed as of 1.14.0.
+
+      Note that this has nothing to do with the real HDFS VFD API calls
+      that are fully functional when the HDFS VFD is configured and built.
+
+      We simply changed:
+
+      #ifdef LIBHDFS
+        <real API call>
+      #else
+        <useless stub>
+      #endif
+
+      to:
+
+      #ifdef LIBHDFS
+        <real API call>
+      #endif
+
+      Which is how the other optional VFDs are handled.
+
+      (DER - 2020/08/27)
+
     - Add Mirror VFD
 
       Use TCP/IP sockets to perform write-only (W/O) file I/O on a remote
@@ -482,6 +625,32 @@ New Features
 
       (DER - 2020/03/18, HDFFV-11057)
 
+    - Add BEST_EFFORT value to HDF5_USE_FILE_LOCKING environment variable
+
+      This change adds a BEST_EFFORT to the TRUE/FALSE, 1/0 settings that
+      were previously accepted. This option turns on file locking but
+      ignores locking errors when the library detects that file locking
+      has been disabled on a file system (useful on some HPC Lustre
+      installations).
+
+      The capitalization of BEST_EFFORT is mandatory.
+
+      See the configure option discussion for HDFFV-11092 (above) for more
+      information on the file locking feature and how it's controlled.
+
+      (DER - 2020/07/30, HDFFV-11092)
+
+
+    - Add H5Pset/get_file_locking() API calls
+
+      This change adds new API calls which can be used to set or get the
+      file locking parameters. The single API call sets both the "use file
+      locking" flag and the "ignore disabled file locking" flag.
+
+      See the configure option discussion for HDFFV-11092 (above) for more
+      information on the file locking feature and how it's controlled.
+
+      (DER - 2020/07/30, HDFFV-11092)
 
     Parallel Library:
     -----------------
@@ -512,11 +681,21 @@ New Features
 
       (MSB, 2019/01/08, HDFFV-10443)
 
+    - Add wrappers for H5Pset/get_file_locking() API calls
+
+      h5pget_file_locking_f()
+      h5pset_file_locking_f()
+
+      See the configure option discussion for HDFFV-11092 (above) for more
+      information on the file locking feature and how it's controlled.
+
+      (DER - 2020/07/30, HDFFV-11092)
+
     C++ Library:
     ------------
     - Added new wrappers for H5Pset/get_create_intermediate_group()
-        LinkCreatPropList::setCreateIntermediateGroup()
         LinkCreatPropList::getCreateIntermediateGroup()
+        LinkCreatPropList::setCreateIntermediateGroup()
 
       (BMR - 2019/04/22, HDFFV-10622)
 
@@ -525,6 +704,16 @@ New Features
 
       (BMR - 2019/02/14, HDFFV-10532)
 
+    - Add wrappers for H5Pset/get_file_locking() API calls
+
+      FileAccPropList::setFileLocking()
+      FileAccPropList::getFileLocking()
+
+      See the configure option discussion for HDFFV-11092 (above) for more
+      information on the file locking feature and how it's controlled.
+
+      (DER - 2020/07/30, HDFFV-11092)
+
 
     Java Library:
     ----------------
@@ -562,9 +751,34 @@ New Features
 
       (DER - 2018/12/08, HDFFV-10252)
 
+    - Add wrappers for H5Pset/get_file_locking() API calls
+
+      H5Pset_file_locking()
+      H5Pget_use_file_locking()
+      H5Pget_ignore_disabled_file_locking()
+
+      Unlike the C++ and Fortran wrappers, there are separate getters for the
+      two file locking settings, each of which returns a boolean value.
+
+      See the configure option discussion for HDFFV-11092 (above) for more
+      information on the file locking feature and how it's controlled.
+
+      (DER - 2020/07/30, HDFFV-11092)
+
 
     Tools:
     ------
+    - h5repack added options to control how external links are handled.
+
+      Currently h5repack preserves external links and cannot copy and merge
+      data from the external files. Two options, merge and prune, were added to
+      control how to merge data from an external link into the resulting file.
+       --merge             Follow external soft link recursively and merge data.
+       --prune             Do not follow external soft links and remove link.
+       --merge --prune     Follow external link, merge data and remove dangling link.
+
+      (ADB - 2020/08/05, HDFFV-9984)
+
     - h5repack was fixed to repack the reference attributes properly.
       The code line that checks if the update of reference inside a compound
       datatype is misplaced outside the code block loop that carries out the
@@ -631,6 +845,50 @@ Bug Fixes since HDF5-1.10.3 release
 
     Library
     -------
+    - Creation of dataset with optional filter
+
+      When the combination of type, space, etc doesn't work for filter
+      and the filter is optional, it was supposed to be skipped but it was
+      not skipped and the creation failed.
+
+      Allowed the creation of the dataset in such situation.
+
+      (BMR - 2020/8/13, HDFFV-10933)
+
+    - Explicitly declared dlopen to use RTLD_LOCAL
+
+      dlopen documentation states that if neither RTLD_GLOBAL nor
+      RTLD_LOCAL are specified, then the default behavior is unspecified.
+      The default on linux is usually RTLD_LOCAL while macos will default
+      to RTLD_GLOBAL.
+
+      (ADB - 2020/08/12, HDFFV-11127)
+
+    - Fixed issues CVE-2018-13870 and CVE-2018-13869
+
+      When a buffer overflow occurred because a name length was corrupted
+      and became very large, h5dump crashed on memory access violation.
+
+      A check for reading pass the end of the buffer was added to multiple
+      locations to prevent the crashes and h5dump now simply fails with an
+      error message when this error condition occurs.
+
+      (BMR - 2020/07/22, HDFFV-11120 and HDFFV-11121)
+
+    - Fixed the segmentation fault when reading attributes with multiple threads
+
+      It was reported that the reading of attributes with variable length string
+      datatype will crash with segmentation fault particularly when the number of
+      threads is high (>16 threads).  The problem was due to the file pointer that
+      was set in the variable length string datatype for the attribute.  That file
+      pointer was already closed when the attribute was accessed.
+
+      The problem was fixed by setting the file pointer to the current opened file pointer
+      when the attribute was accessed.  Similar patch up was done before when reading
+      dataset with variable length string datatype.
+
+      (VC - 2020/07/13, HDFFV-11080)
+
     - Fixed CVE-2020-10810
 
       The tool h5clear produced a segfault during an error recovery in
@@ -912,6 +1170,21 @@ Bug Fixes since HDF5-1.10.3 release
 
       (DER - 2019/12/09, HDFFV-10945)
 
+    - H5Sset_extent_none() sets the dataspace class to H5S_NO_CLASS which
+      causes asserts/errors when passed to other dataspace API calls.
+
+      H5S_NO_CLASS is an internal class value that should not have been
+      exposed via a public API call.
+
+      In debug builds of the library, this can cause asserts to trip. In
+      non-debug builds, it will produce normal library errors.
+
+      The new library behavior is for H5Sset_extent_none() to convert
+      the dataspace into one of type H5S_NULL, which is better handled
+      by the library and easier for developers to reason about.
+
+      (DER - 2020/07/27, HDFFV-11027)
+
 
     Java Library:
     ----------------
@@ -989,11 +1262,51 @@ Bug Fixes since HDF5-1.10.3 release
 
     Tools
     -----
-    -
+    - The tools library was updated by standardizing the error stack process.
+
+      General sequence is:
+          h5tools_setprogname(PROGRAMNAME);
+          h5tools_setstatus(EXIT_SUCCESS);
+          h5tools_init();
+          ... process the command-line (check for error-stack enable) ...
+          h5tools_error_report();
+          ... (do work) ...
+          h5diff_exit(ret);
+
+      (ADB - 2020/07/20, HDFFV-11066)
+
+    - h5diff fixed a command line parsing error.
+
+      h5diff would ignore the argument to -d (delta) if it is smaller than DBL_EPSILON.
+      The macro H5_DBL_ABS_EQUAL was removed and a direct value comparision was used.
+
+      (ADB - 2020/07/20, HDFFV-10897)
+
+    - h5diff added a command line option to ignore attributes.
+
+      h5diff would ignore all objects with a supplied path if the exclude-path argument is used.
+      Adding the exclude-attribute argument will only eclude attributes, with the supplied path,
+      from comparision.
+
+      (ADB - 2020/07/20, HDFFV-5935)
+
+    - h5diff added another level to the verbose argument to print filenames.
+
+      Added verbose level 3 that is level 2 plus the filenames. The levels are:
+          0 : Identical to '-v' or '--verbose'
+          1 : All level 0 information plus one-line attribute status summary
+          2 : All level 1 information plus extended attribute status report
+          3 : All level 2 information plus file names
+
+      (ADB - 2020/07/20, HDFFV-10005)
 
     High-Level APIs:
     ------
-    -
+    - The H5DSis_scale function was updated to return "not a dimension scale" (0)
+      instead of failing (-1), when CLASS or DIMENSION_SCALE attributes are
+      not written according to Dimension Scales Specification.
+
+     (EIP - 2020/08/12, HDFFV-10436)
 
     Fortran High-Level APIs:
     ------
@@ -1015,9 +1328,9 @@ Bug Fixes since HDF5-1.10.3 release
     -------
     - Stopped java/test/junit.sh.in installing libs for testing under ${prefix}
 
-      Lib files needed are now copied to a subdirectory in the java/test 
-      directory, and on Macs the loader path for libhdf5.xxxs.so is changed 
-      in the temporary copy of libhdf5_java.dylib. 
+      Lib files needed are now copied to a subdirectory in the java/test
+      directory, and on Macs the loader path for libhdf5.xxxs.so is changed
+      in the temporary copy of libhdf5_java.dylib.
 
       (LRK, 2020/7/2, HDFFV-11063)