summaryrefslogtreecommitdiffstats
path: root/release_docs/RELEASE.txt
blob: 05daa53294e1f2ba453ae9bade45a4d7dd73516c (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
HDF5 version 1.10.5-snap1 currently under development
================================================================================


INTRODUCTION

This document describes the differences between this release and the previous
HDF5 release. It contains information on the platforms tested and known
problems in this release. For more details check the HISTORY*.txt files in the
HDF5 source.

Note that documentation in the links below will be updated at the time of each
final release.

Links to HDF5 documentation can be found on The HDF5 web page:

     https://portal.hdfgroup.org/display/HDF5/HDF5

The official HDF5 releases can be obtained from:

     https://www.hdfgroup.org/downloads/hdf5/

Changes from Release to Release and New Features in the HDF5-1.10.x release series
can be found at:

     https://portal.hdfgroup.org/display/HDF5/HDF5+Application+Developer%27s+Guide

If you have any questions or comments, please send them to the HDF Help Desk:

     help@hdfgroup.org


CONTENTS

- New Features
- Support for new platforms and languages
- Bug Fixes since HDF5-1.10.3
- Supported Platforms
- Tested Configuration Features Summary
- More Tested Platforms
- Known Problems
- CMake vs. Autotools installations


New Features
============

    Configuration:
    -------------
    - Keep stderr and stdout separate in tests

      Changed test handling of output capture. Tests now keep the stderr
      output separate from the stdout output. It is up to the test to decide
      which output to check against a reference. Also added the option
      to grep for a string in either output.

      (ADB - 2018/12/12, HDFFV-10632)

    - Add toolchain and cross-compile support

      Added info on using a toolchain file to INSTALL_CMAKE.txt. A
      toolchain file is also used in cross-compiling, which requires
      CMAKE_CROSSCOMPILING_EMULATOR to be set. To help with cross-compiling
      the fortran configure process, the HDF5UseFortran.cmake file macros
      were improved. Fixed a Fortran configure file issue that incorrectly
      used #cmakedefine instead of #define.

      (ADB - 2018/10/04, HDFFV-10594)

    - Add warning flags for Intel compilers

      Identified Intel compiler specific warnings flags that should be used
      instead of GNU flags.

      (ADB - 2018/10/04, TRILABS-21)

    - Add default rpath to targets

      Default rpaths should be set in shared executables and
      libraries to allow the use of loading dependent libraries
      without requiring LD_LIBRARY_PATH to be set. The default
      path should be relative using @rpath on osx and $ORIGIN
      on linux. Windows is not affected.

      (ADB - 2018/09/26, HDFFV-10594)

    - Incorrectly installed private header files were removed from
      CMake installs.

      The CMake build files incorrectly flagged the following header files
      as public and installed them. They are private and will no longer be
      installed.

        HDF5 library private package files (H5Xpkg.h)
        H5Edefin.h
        H5Einit.h
        H5Eterm.h
        H5LTparse.h
        h5diff.h
        h5tools_dump.h
        h5tools.h
        h5tools_ref.h
        h5tools_str.h
        h5tools_utils.h
        h5trav.h

      (DER - 2018/10/26, HDFFV-10614, 10609)
 
    - Autotools installs now install H5FDwindows.h

      This is simply to align the installed header files between the
      autotools and CMake. H5FDwindows.h has no functionality on
      non-Windows systems.

      (DER - 2018/10/26, HDFFV-10614)

 
    Library:
    --------
    - Added new chunk query functions

      The following public functions were added to discover information about
      the chunks in an HDF5 file.
        herr_t H5Dget_num_chunks(dset_id, fspace_id, *nchunks)
        herr_t H5Dget_chunk_info_by_coord(dset_id, *coord, *filter_mask, *addr, *size)
        herr_t H5Dget_chunk_info(dset_id, fspace_id, index, *coord, *filter_mask, *addr, *size)

      (BMR - 2018/11/07, HDFFV-10615)

    - Allow pre-generated H5Tinit.c and H5make_libsettings.c to be used.

      Rather than always running H5detect and generating H5Tinit.c and
      H5make_libsettings.c, supply a location for those files.

      (ADB - 2018/09/18, HDFFV-10332)

    - Several empty public header files where removed from the distribution

      The following files were empty placeholders. They are for internal
      packages that are unlikely to ever have public functionality and have
      thus been removed.

        H5Bpublic.h
        H5B2public.h
        H5FSpublic.h
        H5HFpublic.h
        H5HGpublic.h
        H5HLpublic.h

      They were only installed in CMake builds.

      (DER - 2018/10/26, HDFFV-10614)

    - Add ability to minimze dataset object headers.

      Creation of many, very small datasets resulted in extensive file bloat
      due to extra space in the dataset object headers -- this space is
      allocated by default to allow for the insertion of a small number of
      attributes within the object header and not require a continuation
      block, an unnecessary provision in the target use case.

      Inform the library to expect no attributes on created datasets, and to
      allocate the least space possible for the object headers.
      NOTE: A continuation block is created if attributes are added to a
      'minimized' dataset, which can reduce performance.
      NOTE: Some extra space is allocated for attributes essential to the
      correct behavior of the object header (store creation times, e.g.). This
      does not violate the design principle, as the space is calculated and
      allocated as needed at the time of dataset object header creation --
      unused space is not generated.
      New API calls:
      H5Fget_dset_no_attrs_hint
      H5Fset_dset_no_attrs_hint
      H5Pget_dset_no_attrs_hint
      H5Pset_dset_no_attrs_hint

      (JOS - 2019/01/04, TRILAB-45)


     Parallel Library:
    -----------------
    - All MPI-1 API calls have been replaced with MPI-2 equivalents.

      This was done to better support OpenMPI, as default builds no longer
      include MPI-1 support (as of OpenMPI 4.0).

      (DER - 2018/12/30, HDFFV-10566)


    Fortran Library:
    ----------------
    -

    C++ Library:
    ------------
    - 

    Java Library:
    ----------------
    - Duplicate the data read/write functions of Datasets for Attributes.

      Region references could not be displayed for attributes as they could
      for datasets. Datasets had overloaded read and write functions for different
      datatypes that were not available for attributes. After adding similar
      functions, attribute region references work normally.

      (ADB - 2018/12/12, HDFVIEW-4)


    Tools:
    ------
    - The h5repart -family-to-sec2 argument was changed to -family-to-single

      In order to better support other single-file VFDs which could work with
      h5repart, the -family-to-sec2 argument was renamed to -family-to-single.
      This is just a name change and the functionality of the argument has not
      changed.

      The -family-to-sec2 argument has been kept for backwards-compatibility.
      This argument should be considered deprecated.

      (DER - 2018/11/14, HDFFV-10633)

    High-Level APIs:
    ---------------
    -

    C Packet Table API
    ------------------
    -

    Internal header file
    --------------------
    -

    Documentation
    -------------
    -

Support for new platforms, languages and compilers.
=======================================
    -

Bug Fixes since HDF5-1.10.3 release
==================================

    Library
    -------
    - A bug was discovered in the parallel library where an application
      would eventually consume all of the available MPI communicators
      when continually writing to a compressed dataset in parallel. This
      was due to internal copies of an HDF5 File Access Property List,
      which each contained a copy of the MPI communicator, not being
      closed at the end of each write operation. This problem was
      exacerbated by larger numbers of processors.

      (JTH - 2018/12/05, HDFFV-10629)

    - Allow H5detect and H5make_libsettings to take a file as an argument.

      Rather than only writing to stdout, add a command argument to name
      the file that H5detect and H5make_libsettings will use for output.
      Without an argument, stdout is still used, so backwards compatibility
      is maintained.

      (ADB - 2018/09/05, HDFFV-9059)

    - A bug was discovered in the parallel library where an application
      would hang if a collective read/write of a chunked dataset occurred
      when collective metadata reads were enabled and some of the ranks
      had no selection in the dataset's dataspace. The ranks which had no
      selection in the dataset's dataspace called H5D__chunk_addrmap() to
      retrieve the lowest chunk address in the dataset. This is because we
      require reads/writes to be performed in strictly non-decreasing order
      of chunk address in the file.

      When the chunk index used was a version 1 or 2 B-tree, these
      non-participating ranks would issue a collective MPI_Bcast() call
      that the participating ranks would not issue, causing the hang. Since
      the non-participating ranks are not actually reading/writing anything,
      the H5D__chunk_addrmap() call can be safely removed and the address used
      for the read/write can be set to an arbitrary number (0 was chosen).

      (JTH - 2018/08/25, HDFFV-10501)

    - There was an incorrect protection against division by zero reported
      to The HDF Group as issue #CVE-2018-17233.
      
      Protection against division by zero was added to address the issue
      #CVE-2018-17233.  In addition, several similar occurrences in the same
      file were fixed as well.

      (BMR - 2018/02/26, HDFFV-10577)


    Java Library:
    ----------------
    - JNI native library dependencies

      The build for the hdf5_java native library used the wrong
      hdf5 target library for CMake builds. Correcting the hdf5_java
      library to build with the shared hdf5 library required testing
      paths to change also.

      (ADB - 2018/08/31, HDFFV-10568)
     - Java iterator callbacks

      Change global callback object to a small stack structure in order
      to fix a runtime crash. This crash was discovered when iterating
      through a file with nested group members. The global variable
      visit_callback is overwritten when recursion starts. When recursion
      completes, visit_callback will be pointing to the wrong callback method.

      (ADB - 2018/08/15, HDFFV-10536)

    - Java HDFLibraryException class

      Change parent class from Exception to RuntimeException.

      (ADB - 2018/07/30, HDFFV-10534)

    - JNI Read and Write

      Refactored variable-length functions, H5DreadVL and H5AreadVL,
      to correct dataset and attribute reads. New write functions,
      H5DwriteVL and H5AwriteVL, are under construction.

      (ADB - 2018/06/02, HDFFV-10519)

    Fortran
    --------

    - Made Fortran specific subroutines PRIVATE in generic procedures.

      Effected generic procedures were functions in H5A, H5D, H5P, H5R and H5T.
    
      (MSB, 2018/12/04, HDFFV-10511)

Supported Platforms
===================

    Linux 2.6.32-696.16.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
    #1 SMP ppc64 GNU/Linux        g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
    (ostrich)                     GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
                                  IBM XL C/C++ V13.1
                                  IBM XL Fortran V15.1

    Linux 3.10.0-327.10.1.el7   GNU C (gcc), Fortran (gfortran), C++ (g++)
    #1 SMP x86_64 GNU/Linux       compilers:
    (kituo/moohan)                Version 4.8.5 20150623 (Red Hat 4.8.5-4)
                                    Version 4.9.3, Version 5.2.0
                                  Intel(R) C (icc), C++ (icpc), Fortran (icc)
                                  compilers:
                                     Version 17.0.0.098 Build 20160721
                                  MPICH 3.1.4 compiled with GCC 4.9.3

    SunOS 5.11 32- and 64-bit     Sun C 5.12 SunOS_sparc
    (emu)                         Sun Fortran 95 8.6 SunOS_sparc
                                  Sun C++ 5.12 SunOS_sparc

    Windows 7                     Visual Studio 2015 w/ Intel Fortran 16 (cmake)

    Windows 7 x64                 Visual Studio 2013
                                  Visual Studio 2015 w/ Intel Fortran 16 (cmake)
                                  Visual Studio 2015 w/ Intel C, Fortran 2018 (cmake)
                                  Visual Studio 2015 w/ MSMPI 8 (cmake)

    Windows 10                    Visual Studio 2015 w/ Intel Fortran 18 (cmake)

    Windows 10 x64                Visual Studio 2015 w/ Intel Fortran 18 (cmake)
                                  Visual Studio 2017 w/ Intel Fortran 18 (cmake)

    Mac OS X Yosemite 10.10.5     Apple clang/clang++ version 6.1 from Xcode 7.0
    64-bit                        gfortran GNU Fortran (GCC) 4.9.2
    (osx1010dev/osx1010test)      Intel icc/icpc/ifort version 15.0.3

    Mac OS X El Capitan 10.11.6   Apple clang/clang++ version 7.3.0 from Xcode 7.3
    64-bit                        gfortran GNU Fortran (GCC) 5.2.0
    (osx1011dev/osx1011test)      Intel icc/icpc/ifort version 16.0.2

    Mac OS Sierra 10.12.6         Apple LLVM version 8.1.0 (clang/clang++-802.0.42)
    64-bit                        gfortran GNU Fortran (GCC) 7.1.0
    (swallow/kite)                Intel icc/icpc/ifort version 17.0.2


Tested Configuration Features Summary
=====================================

    In the tables below
          y   = tested
          n   = not tested in this release
          C   = Cluster
          W   = Workstation
          x   = not working in this release
          dna = does not apply
          ( ) = footnote appears below second table
          <blank> = testing incomplete on this feature or platform

Platform                              C         F90/   F90      C++  zlib  SZIP
                                      parallel  F2003  parallel
Solaris2.11 32-bit                      n        y/y    n        y    y     y
Solaris2.11 64-bit                      n        y/n    n        y    y     y
Windows 7                               y        y/y    n        y    y     y
Windows 7 x64                           y        y/y    y        y    y     y
Windows 7 Cygwin                        n        y/n    n        y    y     y
Windows 7 x64 Cygwin                    n        y/n    n        y    y     y
Windows 10                              y        y/y    n        y    y     y
Windows 10 x64                          y        y/y    n        y    y     y
Mac OS X Mavericks 10.9.5 64-bit        n        y/y    n        y    y     y
Mac OS X Yosemite 10.10.5 64-bit        n        y/y    n        y    y     y
Mac OS X El Capitan 10.11.6 64-bit      n        y/y    n        y    y     y
Mac OS Sierra 10.12.6 64-bit            n        y/y    n        y    y     y
CentOS 7.2 Linux 2.6.32 x86_64 PGI      n        y/y    n        y    y     y
CentOS 7.2 Linux 2.6.32 x86_64 GNU      y        y/y    y        y    y     y
CentOS 7.2 Linux 2.6.32 x86_64 Intel    n        y/y    n        y    y     y
Linux 2.6.32-573.18.1.el6.ppc64         n        y/y    n        y    y     y


Platform                                 Shared  Shared    Shared    Thread-
                                         C libs  F90 libs  C++ libs  safe
Solaris2.11 32-bit                         y       y         y         y
Solaris2.11 64-bit                         y       y         y         y
Windows 7                                  y       y         y         y
Windows 7 x64                              y       y         y         y
Windows 7 Cygwin                           n       n         n         y
Windows 7 x64 Cygwin                       n       n         n         y
Windows 10                                 y       y         y         y
Windows 10 x64                             y       y         y         y
Mac OS X Mavericks 10.9.5 64-bit           y       n         y         y
Mac OS X Yosemite 10.10.5 64-bit           y       n         y         y
Mac OS X El Capitan 10.11.6 64-bit         y       n         y         y
Mac OS Sierra 10.12.6 64-bit               y       n         y         y
CentOS 7.2 Linux 2.6.32 x86_64 PGI         y       y         y         n
CentOS 7.2 Linux 2.6.32 x86_64 GNU         y       y         y         y
CentOS 7.2 Linux 2.6.32 x86_64 Intel       y       y         y         n
Linux 2.6.32-573.18.1.el6.ppc64            y       y         y         n

Compiler versions for each platform are listed in the preceding
"Supported Platforms" table.


More Tested Platforms
=====================
The following platforms are not supported but have been tested for this release.

    Linux 2.6.32-573.22.1.el6    GNU C (gcc), Fortran (gfortran), C++ (g++)
    #1 SMP x86_64 GNU/Linux       compilers:
    (mayll/platypus)                 Version 4.4.7 20120313
                                     Version 4.9.3, 5.3.0, 6.2.0
                                  PGI C, Fortran, C++ for 64-bit target on
                                  x86-64;
                                     Version 17.10-0
                                  Intel(R) C (icc), C++ (icpc), Fortran (icc)
                                  compilers:
                                     Version 17.0.4.196 Build 20170411
                                  MPICH 3.1.4 compiled with GCC 4.9.3

    Linux 3.10.0-327.18.2.el7     GNU C (gcc) and C++ (g++) compilers
    #1 SMP x86_64 GNU/Linux          Version 4.8.5 20150623 (Red Hat 4.8.5-4)
    (jelly)                       with NAG Fortran Compiler Release 6.1(Tozai)
                                  GCC Version 7.1.0
                                  OpenMPI 3.0.0-GCC-7.2.0-2.29, 
                                     3.1.0-GCC-7.2.0-2.29
                                  Intel(R) C (icc) and C++ (icpc) compilers
                                     Version 17.0.0.098 Build 20160721
                                  with NAG Fortran Compiler Release 6.1(Tozai)

    Linux 3.10.0-327.10.1.el7     MPICH 3.2 compiled with GCC 5.3.0
    #1 SMP x86_64 GNU/Linux
    (moohan)

    Linux 2.6.32-573.18.1.el6.ppc64  MPICH mpich 3.1.4 compiled with
    #1 SMP ppc64 GNU/Linux           IBM XL C/C++ for Linux, V13.1
    (ostrich)                        and IBM XL Fortran for Linux, V15.1

    Debian 8.4 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1 x86_64 GNU/Linux
                                  gcc, g++ (Debian 4.9.2-10) 4.9.2
                                  GNU Fortran (Debian 4.9.2-10) 4.9.2
                                  (cmake and autotools)

    Fedora 24  4.7.2-201.fc24.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
                                  gcc, g++ (GCC) 6.1.1 20160621
                                      (Red Hat 6.1.1-3)
                                  GNU Fortran (GCC) 6.1.1 20160621
                                      (Red Hat 6.1.1-3)
                                  (cmake and autotools)

    Ubuntu 16.04.1 4.4.0-38-generic #57-Ubuntu SMP x86_64 GNU/Linux
                                  gcc, g++ (Ubuntu 5.4.0-6ubuntu1~16.04.2)
                                      5.4.0 20160609
                                  GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.2)
                                      5.4.0 20160609
                                  (cmake and autotools)


Known Problems
==============

    At present, metadata cache images may not be generated by parallel
    applications.  Parallel applications can read files with metadata cache
    images, but since this is a collective operation, a deadlock is possible
    if one or more processes do not participate.

    Three tests fail with OpenMPI 3.0.0/GCC-7.2.0-2.29:
        testphdf5 (ecdsetw, selnone, cchunk1, cchunk3, cchunk4, and actualio)
        t_shapesame (sscontig2)
        t_pflush1/fails on exit
    The first two tests fail attempting collective writes.

    Known problems in previous releases can be found in the HISTORY*.txt files
    in the HDF5 source. Please report any new problems found to
    help@hdfgroup.org.


CMake vs. Autotools installations
=================================
While both build systems produce similar results, there are differences.
Each system produces the same set of folders on linux (only CMake works
on standard Windows); bin, include, lib and share. Autotools places the
COPYING and RELEASE.txt file in the root folder, CMake places them in
the share folder.

The bin folder contains the tools and the build scripts. Additionally, CMake
creates dynamic versions of the tools with the suffix "-shared". Autotools
installs one set of tools depending on the "--enable-shared" configuration
option.
  build scripts
  -------------
  Autotools: h5c++, h5cc, h5fc
  CMake: h5c++, h5cc, h5hlc++, h5hlcc

The include folder holds the header files and the fortran mod files. CMake
places the fortran mod files into separate shared and static subfolders,
while Autotools places one set of mod files into the include folder. Because
CMake produces a tools library, the header files for tools will appear in
the include folder.

The lib folder contains the library files, and CMake adds the pkgconfig
subfolder with the hdf5*.pc files used by the bin/build scripts created by
the CMake build. CMake separates the C interface code from the fortran code by
creating C-stub libraries for each Fortran library. In addition, only CMake
installs the tools library. The names of the szip libraries are different
between the build systems.

The share folder will have the most differences because CMake builds include
a number of CMake specific files for support of CMake's find_package and support
for the HDF5 Examples CMake project.