summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorLarry Knox <lrknox@hdfgroup.org>2019-02-20 20:10:48 (GMT)
committerLarry Knox <lrknox@hdfgroup.org>2019-02-20 20:10:48 (GMT)
commit0c6110e586985aa6d2356b739179130c5d79e9c1 (patch)
treebd766348e7c5071b7d7b10799a10ed74d821132f
parentae0a8a197b7c2fbcaa3d5bfa20f6ad212bd69365 (diff)
parent1353f9552eaabf9f17b9043ed1d9a3d97fb5ba87 (diff)
downloadhdf5-0c6110e586985aa6d2356b739179130c5d79e9c1.zip
hdf5-0c6110e586985aa6d2356b739179130c5d79e9c1.tar.gz
hdf5-0c6110e586985aa6d2356b739179130c5d79e9c1.tar.bz2
Merge pull request #1567 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10 to hdf5_1_10
* commit '1353f9552eaabf9f17b9043ed1d9a3d97fb5ba87': Add new files to MANIFEST. Update so version numbers. fix 2 typos.
-rw-r--r--MANIFEST2
-rw-r--r--config/lt_vers.am24
-rw-r--r--[-rwxr-xr-x]release_docs/HISTORY-1_10.txt0
-rw-r--r--release_docs/README_HDF5_CMake23
-rw-r--r--release_docs/README_HPC206
-rw-r--r--release_docs/RELEASE.txt4
6 files changed, 245 insertions, 14 deletions
diff --git a/MANIFEST b/MANIFEST
index c16dde5..66cbb00 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -462,6 +462,8 @@
./release_docs/INSTALL_Warnings.txt
./release_docs/INSTALL_Windows.txt
./release_docs/RELEASE.txt
+./release_docs/README_HDF5_CMake
+./release_docs/README_HPC
./release_docs/USING_HDF5_CMake.txt
./release_docs/USING_HDF5_VS.txt
diff --git a/config/lt_vers.am b/config/lt_vers.am
index 8b63c22..84bff2d 100644
--- a/config/lt_vers.am
+++ b/config/lt_vers.am
@@ -16,7 +16,7 @@
## See libtool versioning documentation online.
## After making changes, run bin/reconfigure to update other configure related
## files like Makefile.in.
-LT_VERS_INTERFACE = 103
+LT_VERS_INTERFACE = 104
LT_VERS_REVISION = 0
LT_VERS_AGE = 0
@@ -39,31 +39,31 @@ LT_VERS_AGE = 0
## the effects of the H5_V1_x_COMPAT flag.
##
## Version numbers for wrapper shared library files.
-LT_CXX_VERS_INTERFACE = 103
+LT_CXX_VERS_INTERFACE = 104
LT_CXX_VERS_REVISION = 0
-LT_CXX_VERS_AGE = 0
+LT_CXX_VERS_AGE = 1
-LT_F_VERS_INTERFACE = 101
-LT_F_VERS_REVISION = 2
-LT_F_VERS_AGE = 1
+LT_F_VERS_INTERFACE = 102
+LT_F_VERS_REVISION = 0
+LT_F_VERS_AGE = 0
LT_HL_VERS_INTERFACE = 101
-LT_HL_VERS_REVISION = 1
+LT_HL_VERS_REVISION = 2
LT_HL_VERS_AGE = 1
LT_HL_CXX_VERS_INTERFACE = 101
-LT_HL_CXX_VERS_REVISION = 2
+LT_HL_CXX_VERS_REVISION = 3
LT_HL_CXX_VERS_AGE = 1
LT_HL_F_VERS_INTERFACE = 100
-LT_HL_F_VERS_REVISION = 3
+LT_HL_F_VERS_REVISION = 4
LT_HL_F_VERS_AGE = 0
-LT_JAVA_VERS_INTERFACE = 103
+LT_JAVA_VERS_INTERFACE = 104
LT_JAVA_VERS_REVISION = 0
-LT_JAVA_VERS_AGE = 3
+LT_JAVA_VERS_AGE = 4
LT_TOOLS_VERS_INTERFACE = 101
-LT_TOOLS_VERS_REVISION = 1
+LT_TOOLS_VERS_REVISION = 2
LT_TOOLS_VERS_AGE = 1
diff --git a/release_docs/HISTORY-1_10.txt b/release_docs/HISTORY-1_10.txt
index ad8beb2..ad8beb2 100755..100644
--- a/release_docs/HISTORY-1_10.txt
+++ b/release_docs/HISTORY-1_10.txt
diff --git a/release_docs/README_HDF5_CMake b/release_docs/README_HDF5_CMake
new file mode 100644
index 0000000..484710d
--- /dev/null
+++ b/release_docs/README_HDF5_CMake
@@ -0,0 +1,23 @@
+This tar file contains
+
+ build-unix.sh script to build HDF5 with CMake on unix machines
+ build-unix-hpc.sh script to build HDF5 with CMake on unix machines and run
+ tests with batch scripts (sbatch).
+ CTestScript.cmake
+ HDF5config.cmake CMake scripts for building HDF5
+ HDF5options.cmake
+ hdf5-1.10.5-pre1 HDF5 1.10.5-pre1 source
+ SZip.tar.gz source for building SZIP
+ ZLib.tar.gz source for building ZLIB
+
+For more information about building HDF5 with CMake, see USING_HDF5_CMake.txt in
+hdf5-1.10.5-pre1/release_docs, or
+https://portal.hdfgroup.org/display/support/Building+HDF5+with+CMake.
+
+For more information about building HDF5 with CMake on HPC machines, including
+cross compiling on Cray XC40, see README_HPC in hdf5-1.10.5-pre1/release_docs.
+
+
+
+
+
diff --git a/release_docs/README_HPC b/release_docs/README_HPC
new file mode 100644
index 0000000..67a5d6c
--- /dev/null
+++ b/release_docs/README_HPC
@@ -0,0 +1,206 @@
+************************************************************************
+* Using CMake to build and test HDF5 source on HPC machines *
+************************************************************************
+
+ Contents
+
+Section I: Prerequisites
+Section II: Obtain HDF5 source
+Section III: Using ctest command to build and test
+Section IV: Cross compiling
+Section V: Manual alternatives
+Section VI: Other cross compiling options
+
+************************************************************************
+
+========================================================================
+I. Prerequisites
+========================================================================
+ 1. Create a working directory that is accessible from the compute nodes for
+ running tests; the working directory should be in a scratch space or a
+ parallel file system space since testing will use this space. Building
+ from HDF5 source in a 'home' directory typically results in test
+ failures and should be avoided.
+
+ 2. Load modules for desired compilers, module for cmake version 3.10 or greater,
+ and set any needed environment variables for compilers (i.e., CC, FC, CXX).
+ Unload any problematic modules (i.e., craype-hugepages2M).
+
+========================================================================
+II. Obtain HDF5 source
+========================================================================
+Obtain HDF5 source code from the HDF5 repository using a git command or
+from a release tar file in a working directory:
+
+ git clone https://git@bitbucket.hdfgroup.org/scm/hdffv/hdf5.git
+ [-b branch] [source directory]
+
+If no branch is specified, then the 'develop' version will be checked out.
+If no source directory is specified, then the source will be located in the
+'hdf5' directory. The Cmake scripts expect the source to be in a directory
+named hdf5-<version string>, where 'version string' uses the format '1.xx.xx'.
+For example, for the current 'develop' version, the "hdf5" directory should
+be renamed "hdf5-1.11.4", or for the first hdf5_1_10_5 pre-release version,
+it should be renamed "hdf5-1.10.5-pre1".
+
+If the version number is not known a priori, the version string
+can be obtained by running bin/h5vers in the top level directory of the source clone, and
+the source directory renamed 'hdf5-<version string>'.
+
+Release or snapshot tar files may also be extracted and used.
+
+========================================================================
+III. Using ctest command to build and test
+========================================================================
+
+The ctest command [1]:
+
+ ctest -S HDF5config.cmake,BUILD_GENERATOR=Unix -C Release -V -O hdf5.log
+
+will configure, build, test and package HDF5 from the downloaded source
+after the setup steps outlined below are followed.
+
+CMake option variables are available to allow running test programs in batch
+scripts on compute nodes and to cross-compile for compute node hardware using
+a cross-compiling emulator. The setup steps will make default settings for
+parallel or serial only builds available to the CMake command.
+
+ 1. For the current 'develop' version the "hdf5" directory should be renamed
+ "hdf5-1.11.4".
+
+ 2. Three cmake script files need to be copied to the working directory, or
+ have symbolic links to them, created in the working directory:
+
+ hdf5-1.11.4/config/cmake/scripts/HDF5config.cmake
+ hdf5-1.11.4/config/cmake/scripts/CTestScript.cmake
+ hdf5-1.11.4/config/cmake/scripts/HDF5options.cmake
+
+ should be copied to the working directory.
+
+ 3. The resulting contents of the working directory are then:
+
+ CTestScript.cmake
+ HDF5config.cmake
+ HDF5options.cmake
+ hdf5-1.11.4
+
+ Additionally, when the ctest command runs [1], it will add a build directory
+ in the working directory.
+
+ 4. The following options (among others) can be added to the ctest
+ command [1], following '-S HDF5config.cmake,' and separated by ',':
+
+ HPC=sbatch (or 'bsub' or 'raybsub') indicates which type of batch
+ files to use for running tests. If omitted, test
+ will run on the local machine or login node.
+
+ KNL=true to cross-compile for KNL compute nodes on CrayXC40
+ (see section IV)
+
+ MPI=true enables parallel, disables c++, java, and threadsafe
+
+ LOCAL_BATCH_SCRIPT_ARGS="--account=<account#>" to supply user account
+ information for batch jobs
+
+ The HPC options will add BUILD_GENERATOR=Unix for the three HPC options.
+ An example ctest command for a parallel build on a system using sbatch is
+
+ ctest -S HDF5config.cmake,HPC=sbatch,MPI=true -C Release -V -O hdf5.log
+
+ Adding the option 'KNL=true' to the above list will compile for KNL nodes,
+ for example, on 'mutrino' and other CrayXC40 machines.
+
+ Changing -V to -VV will produce more logging information in HDF5.log.
+
+ More detailed CMake information can be found in the HDF5 source in
+ release_docs/INSTALL_CMake.txt.
+
+========================================================================
+IV. Cross-compiling
+========================================================================
+For cross-compiling on Cray, set environment variables CC=cc, FC=ftn
+and CXX=CC (for c++) after all compiler modules are loaded since switching
+compiler modules may unset or reset these variables.
+
+CMake provides options for cross-compiling. To cross-compile for KNL hardware
+on mutrino and other CrayXC40 machines, add HPC=sbatch,KNL=true to the
+ctest command line. This will set the following options from the
+config/cmake/scripts/HPC/sbatch-HDF5options.cmake file:
+
+ set (COMPILENODE_HWCOMPILE_MODULE "craype-haswell")
+ set (COMPUTENODE_HWCOMPILE_MODULE "craype-mic-knl")
+ set (LOCAL_BATCH_SCRIPT_NAME "knl_ctestS.sl")
+ set (LOCAL_BATCH_SCRIPT_PARALLEL_NAME "knl_ctestP.sl")
+ set (ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DCMAKE_TOOLCHAIN_FILE:STRING=config/toolchain/crayle.cmake")
+
+On the Cray XC40 the craype-haswell module is needed for configuring, and the
+craype-mic-knl module is needed for building to run on the KNL nodes. CMake
+with the above options will swap modules after configuring is complete,
+but before compiling programs for KNL.
+
+The sbatch script arguments for running jobs on KNL nodes may differ on CrayXC40
+machines other than mutrino. The batch scripts knl_ctestS.sl and knl_ctestP.sl
+have the correct arguments for mutrino: "#SBATCH -p knl -C quad,cache". For
+cori, another CrayXC40, that line is replaced by "#SBATCH -C knl,quad,cache".
+For cori (and other machines), the values in LOCAL_BATCH_SCRIPT_NAME and
+LOCAL_BATCH_SCRIPT_PARALLEL_NAME in the config/cmake/scripts/HPC/sbatch-HDF5options.cmake
+file can be replaced by cori_knl_ctestS.sl and cori_knl_ctestS.sl, or the lines
+can be edited in the batch files in hdf5-1.11.4/bin/batch.
+
+========================================================================
+V. Manual alternatives
+========================================================================
+If using ctest is undesirable, one can create a build directory and run the cmake
+configure command, for example
+
+"/projects/Mutrino/hpcsoft/cle6.0/common/cmake/3.10.2/bin/cmake"
+-C "<working directory>/hdf5-1.11.4/config/cmake/cacheinit.cmake"
+-DCMAKE_BUILD_TYPE:STRING=Release -DHDF5_BUILD_FORTRAN:BOOL=ON
+-DHDF5_BUILD_JAVA:BOOL=OFF
+-DCMAKE_INSTALL_PREFIX:PATH=<working directory>/HDF_Group/HDF5/1.11.4
+-DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF
+-DHDF5_ENABLE_PARALLEL:BOOL=ON -DHDF5_BUILD_CPP_LIB:BOOL=OFF
+-DHDF5_BUILD_JAVA:BOOL=OFF -DHDF5_ENABLE_THREADSAFE:BOOL=OFF
+-DHDF5_PACKAGE_EXTLIBS:BOOL=ON -DLOCAL_BATCH_TEST:BOOL=ON
+-DMPIEXEC_EXECUTABLE:STRING=srun -DMPIEXEC_NUMPROC_FLAG:STRING=-n
+-DMPIEXEC_MAX_NUMPROCS:STRING=6
+-DCMAKE_TOOLCHAIN_FILE:STRING=config/toolchain/crayle.cmake
+-DLOCAL_BATCH_SCRIPT_NAME:STRING=knl_ctestS.sl
+-DLOCAL_BATCH_SCRIPT_PARALLEL_NAME:STRING=knl_ctestP.sl -DSITE:STRING=mutrino
+-DBUILDNAME:STRING=par-knl_GCC493-SHARED-Linux-4.4.156-94.61.1.16335.0.PTF.1107299-default-x86_64
+"-GUnix Makefiles" "" "<working directory>/hdf5-1.11.4"
+
+followed by make and batch jobs to run tests.
+
+To cross-compile on CrayXC40, run the configure command with the craype-haswell
+module loaded, then switch to the craype-mic-knl module for the build process.
+
+Tests on machines using slurm can be run with
+
+"sbatch -p knl -C quad,cache ctestS.sl"
+
+or
+
+"sbatch -p knl -C quad,cache ctestP.sl"
+
+for parallel builds.
+
+Tests on machines using LSF will typically use "bsub ctestS.lsf", etc.
+
+========================================================================
+VI. Other cross compiling options
+========================================================================
+Settings for two other cross-compiling options are also in the config/toolchain
+files which do not seem to be necessary with the Cray PrgEnv-* modules
+
+1. HDF5_USE_PREGEN. This option, along with the HDF5_USE_PREGEN_DIR CMake
+ variable would allow the use of an appropriate H5Tinit.c file with type
+ information generated on a compute node to be used when cross compiling
+ for those compute nodes. The use of the variables in lines 110 and 111
+ of HDF5options.cmake file seem to preclude needing this option with the
+ available Cray modules and CMake option.
+
+2. HDF5_BATCH_H5DETECT and associated CMake variables. This option when
+ properly configured will run H5detect in a batch job on a compute node
+ at the beginning of the CMake build process. It was also found to be
+ unnecessary with the available Cray modules and CMake options.
diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt
index 939b586..baa0456 100644
--- a/release_docs/RELEASE.txt
+++ b/release_docs/RELEASE.txt
@@ -199,7 +199,7 @@ New Features
Parallel Library:
-----------------
- - Changed the default behavior in parallel when reading the same dataset in its entirely
+ - Changed the default behavior in parallel when reading the same dataset in its entirety
(i.e. H5S_ALL dataset selection) which is being read by all the processes collectively.
The dataset must be contiguous, less than 2GB, and of an atomic datatype.
The new behavior is the HDF5 library will use an MPI_Bcast to pass the data read from
@@ -348,7 +348,7 @@ Bug Fixes since HDF5-1.10.4 release
The slow down is due to the search of the "tag_list" to find
out the "corked" status of an object and "uncork" it if so.
- Improve porformance by skipping the search of the "tag_list"
+ Improve performance by skipping the search of the "tag_list"
if there are no "corked" objects when closing an object.
(VC - 2019/02/06)