1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
|
HDF5 version 1.10.1-pre1 released on 2017-03-23
================================================================================
Note: this document is a work in progress. In particular not all bug fixes have
been documented here, and issues reported in the pre-release versions will appear in
"known problems" in the final 1.10.1 release.
INTRODUCTION
This document describes the differences between HDF5-1.10.0-patch1 and
HDF5 1.10.1, and contains information on the platforms tested and known problems in HDF5-1.10.1.
For more details check the HISTORY*.txt files in the HDF5 source.
Links to HDF5 1.10.1 source code, documentation, and additional materials can be found on The HDF5 web page at:
https://support.hdfgroup.org/HDF5/
The HDF5 1.10.1 release can be obtained from:
https://support.hdfgroup.org/HDF5/release/obtain5110.html
User documentation for the snapshot can be accessed directly at this location:
https://support.hdfgroup.org/HDF5/doc1.10/
New features in the HDF5-1.10.x release series, including brief general
descriptions of some new and modified APIs, are described in the "What's New
in 1.10.1?" document:
https://support.hdfgroup.org/HDF5/doc/ADGuide/WhatsNew1101.html
All new and modified APIs are listed in detail in the "HDF5 Software Changes
from Release to Release" document, in the section "Release 1.8.19 (current
release) versus Release 1.10.1
https://support.hdfgroup.org/HDF5/doc1.10/ADGuide/Changes.html
If you have any questions or comments, please send them to the HDF Help Desk:
help@hdfgroup.org
CONTENTS
- Major New Features Introduced in HDF5 1.10.1
- Other New Features
- Support for new platforms and languages
- Bug Fixes since HDF5-1.10.0-patch1
- Supported Platforms
- Tested Configuration Features Summary
- More Tested Platforms
- Known Problems
Major New Features Introduced in HDF5 1.10.1
============================================
For links to the RFCs and documentation in this section please view
https://support.hdfgroup.org/HDF5/docNewFeatures in a web browser.
________________________________________
Metadata Cache Image
________________________________________
HDF5 metadata is typically small, and scattered throughout the HDF5 file.
This can affect performance, particularly on large HPC systems. The Metadata
Cache Image feature can improve performance by writing the metadata cache in
a single block on file close, and then populating the cache with the contents
of this block on file open, thus avoiding the many small I/O operations that
would otherwise be required on file open and close. See the RFC for complete
details regarding this feature. Also, see the Fine Tuning the Metadata Cache
documentation.
________________________________________
Metadata Cache Evict on Close
________________________________________
The HDF5 library's metadata cache is fairly conservative about holding on to
HDF5 object metadata (object headers, chunk index structures, etc.), which can
cause the cache size to grow, resulting in memory pressure on an application
or system. The "evict on close" property will cause all metadata for an object
to be evicted from the cache as long as metadata is not referenced from any
other open object. See the Fine Tuning the Metadata Cache documentation
for information on the APIs.
________________________________________
Paged Aggregation
________________________________________
The current HDF5 file space allocation accumulates small pieces of metadata
and raw data in aggregator blocks which are not page aligned and vary widely
in sizes. The paged aggregation feature was implemented to provide efficient
paged access of these small pieces of metadata and raw data. See the RFC for
details. Also, see the File Space Management documentation.
________________________________________
Page Buffering
________________________________________
Small and random I/O accesses on parallel file systems result in poor
performance for applications. Page buffering in conjunction with paged
aggregation can improve performance by giving an application control of
minimizing HDF5 I/O requests to a specific granularity and alignment.
See the RFC for details. Also, see the Page Buffering documentation.
New Features
============
Configuration:
-------------
- CMake minimum is now 3.2.2.
(ADB 2017/01/10)
- Tools folder is separated into source and test folders. This
allows autotools to skip the make command and just execute
the make check command.
(HDFFV-9719 ADB 2016/10/27)
Library:
--------
- Paged Aggregation
This is one of the file space management strategies. This strategy
aggregates small metadata and raw data allocations into constant-sized
well-aligned pages, which are suitable for page caching. Paged
aggregation together with the page buffering feature will allow efficient
I/O accesses.
- Page Buffering
The page buffering layer in the HDF5 library absorbs small accesses to
the file system. Each page in memory corresponds to a page allocated in
the file. Access to the file system is then performed as a single page
or multiple of pages, if they are contiguous. This ensures that small
accesses to the file system are avoided while providing another caching
layer for improved I/O performance. This feature works in conjunction
with the paged aggregation feature.
Parallel Library:
-----------------
-
Fortran Library:
----------------
-
C++ Library:
------------
- New wrappers for C APIs:
(HDFFV-10004, HDFFV-10139, HDFFV-10145 - PRs #209, #232, #310, #334, #348)
// Sets/Gets the strategy and the threshold value that the library will
// will employ in managing file space.
FileCreatPropList::setFileSpaceStrategy - H5Pset_file_space_strategy
FileCreatPropList::getFileSpaceStrategy - H5Pget_file_space_strategy
// Sets/Gets the file space page size for paged aggregation.
FileCreatPropList::setFileSpacePagesize - H5Pset_file_space_page_size
FileCreatPropList::getFileSpacePagesize - H5Pget_file_space_page_size
// Checks if the given ID is valid.
IdComponent::isValid - H5Iis_valid
// Sets/Gets the number of soft or user-defined links that can be
// traversed before a failure occurs.
LinkAccPropList::setNumLinks - H5Pset_nlinks
LinkAccPropList::getNumLinks - H5Pget_nlinks
// Returns a copy of the creation property list of a datatype.
DataType::getCreatePlist - H5Tget_create_plist
// Opens/Closes an object within a group or a file, regardless object type
Group::getObjId - H5Oopen
Group::closeObjId - H5Oclose
// Maps elements of a virtual dataset to elements of the source dataset.
DSetCreatPropList::setVirtual - H5Pset_virtual
// Gets general information about this file.
H5File::getFileInfo - H5Fget_info2
// Returns the number of members in a type.
IdComponent::getNumMembers - H5Inmembers
// Determines if an element type exists.
IdComponent::typeExists - H5Itype_exists
// Determines if an object exists.
H5Location::exists - H5Lexists.
// Returns the header version of an HDF5 object.
H5Object::objVersion - H5Oget_info for version
-BMR, 2017/03/20
- New exception: ObjHeaderIException for H5O interface.
(HDFFV-10145 - PR #334) -BMR, 2017/03/15
- New class LinkAccPropList for link access property list, to be used by
wrappers of H5Lexists. (HDFFV-10145 - PR #232) -BMR, 2017/01/04
- New constructors to open datatypes in ArrayType, CompType, DataType,
EnumType, FloatType, IntType, StrType, and VarLenType.
(HDFFV-10056 - PR #209) -BMR, 2016/12/26
- New member functions
DSetCreatPropList::setNbit() to setup N-bit compression for a dataset.
(HDFFV-8623 patch 7) -BMR, 2016/04/25
ArrayType::getArrayNDims() const
ArrayType::getArrayDims() const
both to replace the non-const versions. (HDFFV-9725) -BMR, 2016/04/25
Tools:
------
- Add options to h5clear:
-s: clear the status_flags field in the file's superblock
-m: Remove the metadata cache image from the file
(Pull Request #361 QK 2017/03/22)
High-Level APIs:
---------------
C Packet Table API
------------------
-
Internal header file
--------------------
-
Documentation
-------------
Support for new platforms, languages and compilers.
=======================================
-
Bug Fixes since HDF5-1.10.0-patch1 release
==================================
Library
-------
- Changed the plugins dlopen option from RTLD_NOW to RTLD_LAZY
(PR 201 ADB 2016/12/12)
- Fix error when copying dataset with attribute which is a compound datatype
consisting of a variable length string.
(HDFFV-7991 VC 2016/08/19, 2016/08/21, 2016/08/24)
- H5DOappend will not fail if a dataset has no append callback registered.
(HDFFV-9960 VC 2016/08/05, 2016/08/14)
- Fix the problem where the committed datatype's file location is different
from the file location of an attribute with that committed datatype.
(HDFFV-9940 VC 2016/07/03, 2016/07/06)
- (a) Throw an error instead of assertion when v1 btree level hits the 1 byte limit.
(b) Modifications to better handle error recovery when conversion by
h5format_convert fails.
(HDFFV-9434 VC 2016/05/29)
Configuration
-------------
- Configuration will check for the strtoll and strtoull functions
before using alternatives
(PR 340 ADB 2017/03/17)
- CMake uses a Windows pdb directory variable if available and
will generate both static and shared pdb files.
(HDFFV-9875 ADB 2017/02/06)
- CMake now builds shared versions of tools.
(HDFFV-10123 ADB 2017/02/01)
Performance
-------------
-
Fortran
--------
-
Tools
-----
- h5diff correctly ignores strpad in comparing strings.
(HDFFV-10128 ADB 2017/03/03)
- h5repack now correctly parses the command line filter options.
(HDFFV-10046 ADB 2017/01/24)
High-Level APIs:
------
-
Fortran High-Level APIs:
------
-
Documentation
-------------
F90 APIs
--------
-
C++ APIs
--------
- Due to the change in the C API, the overloaded functions of
PropList::setProperty now need const for some arguments. They are
planned for deprecation and are replaced by new versions with proper
consts. (PR #344) -BMR, 2017/03/17
- The problem where a user-defined function cannot access both, attribute
and dataset, using only one argument is now fixed.
(HDFFV-9920 - PR #45) -BMR, 2016/10/11
- The high-level API Packet Table (PT) did not write data correctly when
the datatype is a compound type that has string type as one of the
members. This problem started in 1.8.15, after the fix of HDFFV-9042
was applied, which caused the Packet Table to use native type to access
the data. It should be up to the application to specify whether the
buffer to be read into memory is in the machine's native architecture.
Thus, the PT is fixed to not use native type but to make a copy of the
user's provided datatype during creation or the packet table's datatype
during opening. If an application wishes to use native type to read the
data, then the application will request that. However, the Packet Table
doesn't provide a way to specify memory datatype in this release. This
feature will be available in future releases, HDFFV-10023.
(HDFFV-9758 - PRs #93 and #108) -BMR, 2016/10/27
- The obsolete macros, H5_NO_NAMESPACE and H5_NO_STD, are removed.
(HDFFV-9532 - PR #92) -BMR, 2016/10/23
- In-memory array information, ArrayType::rank and
ArrayType::dimensions, were removed. This is an implementation
detail and should not affect applications. (HDFFV-9725) -BMR, 2016/04/25
Testing
-------
-
Supported Platforms
===================
Linux 2.6.32-573.18.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
#1 SMP ppc64 GNU/Linux g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
(ostrich) GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
IBM XL C/C++ V13.1
IBM XL Fortran V15.1
Linux 3.10.0-327.10.1.el7 GNU C (gcc), Fortran (gfortran), C++ (g++)
#1 SMP x86_64 GNU/Linux compilers:
(kituo/moohan) Version 4.8.5 20150623 (Red Hat 4.8.5-4)
Version 4.9.3, Version 5.2.0
Intel(R) C (icc), C++ (icpc), Fortran (icc)
compilers:
Version 15.0.3.187 Build 20150407
MPICH 3.1.4 compiled with GCC 4.9.3
SunOS 5.11 32- and 64-bit Sun C 5.12 SunOS_sparc
(emu) Sun Fortran 95 8.6 SunOS_sparc
Sun C++ 5.12 SunOS_sparc
Windows 7 Visual Studio 2012 w/ Intel Fortran 15 (cmake)
Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Visual Studio 2015 w/ Intel Fortran 16 (cmake)
Cygwin(CYGWIN_NT-6.1 2.2.1(0.289/5/3) gcc(4.9.3) compiler and gfortran)
(cmake and autotools)
Windows 7 x64 Visual Studio 2012 w/ Intel Fortran 15 (cmake)
Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Visual Studio 2015 w/ Intel Fortran 16 (cmake)
Visual Studio 2015 w/ Intel Parallel Studio 2017 (cmake)
Windows 10 Visual Studio 2015 w/ Intel Fortran 16 (cmake)
Windows 10 x64 Visual Studio 2015 w/ Intel Fortran 16 (cmake)
Mac OS X Mt. Lion 10.8.5 Apple clang/clang++ version 5.1 from Xcode 5.1
64-bit gfortran GNU Fortran (GCC) 4.8.2
(swallow/kite) Intel icc/icpc/ifort version 15.0.3
Mac OS X Mavericks 10.9.5 Apple clang/clang++ version 6.0 from Xcode 6.2
64-bit gfortran GNU Fortran (GCC) 4.9.2
(wren/quail) Intel icc/icpc/ifort version 15.0.3
Mac OS X Yosemite 10.10.5 Apple clang/clang++ version 6.1 from Xcode 7.0
64-bit gfortran GNU Fortran (GCC) 4.9.2
(osx1010dev/osx1010test) Intel icc/icpc/ifort version 15.0.3
Mac OS X El Capitan 10.11.6 Apple clang/clang++ version 7.3.0 from Xcode 7.3
64-bit gfortran GNU Fortran (GCC) 5.2.0
(osx1010dev/osx1010test) Intel icc/icpc/ifort version 16.0.2
Tested Configuration Features Summary
=====================================
In the tables below
y = tested
n = not tested in this release
C = Cluster
W = Workstation
x = not working in this release
dna = does not apply
( ) = footnote appears below second table
<blank> = testing incomplete on this feature or platform
Platform C F90/ F90 C++ zlib SZIP
parallel F2003 parallel
Solaris2.11 32-bit n y/y n y y y
Solaris2.11 64-bit n y/n n y y y
Windows 7 y y/y n y y y
Windows 7 x64 y y/y y y y y
Windows 7 Cygwin n y/n n y y y
Windows 7 x64 Cygwin n y/n n y y y
Windows 10 y y/y n y y y
Windows 10 x64 y y/y n y y y
Mac OS X Mountain Lion 10.8.5 64-bit n y/y n y y y
Mac OS X Mavericks 10.9.5 64-bit n y/y n y y y
Mac OS X Yosemite 10.10.5 64-bit n y/y n y y y
Mac OS X El Capitan 10.11.6 64-bit n y/y n y y y
CentOS 7.2 Linux 2.6.32 x86_64 PGI n y/y n y y y
CentOS 7.2 Linux 2.6.32 x86_64 GNU y y/y y y y y
CentOS 7.2 Linux 2.6.32 x86_64 Intel n y/y n y y y
Linux 2.6.32-573.18.1.el6.ppc64 n y/y n y y y
Platform Shared Shared Shared Thread-
C libs F90 libs C++ libs safe
Solaris2.11 32-bit y y y y
Solaris2.11 64-bit y y y y
Windows 7 y y y y
Windows 7 x64 y y y y
Windows 7 Cygwin n n n y
Windows 7 x64 Cygwin n n n y
Windows 10 y y y y
Windows 10 x64 y y y y
Mac OS X Mountain Lion 10.8.5 64-bit y n y y
Mac OS X Mavericks 10.9.5 64-bit y n y y
Mac OS X Yosemite 10.10.5 64-bit y n y y
Mac OS X El Capitan 10.11.6 64-bit y n y y
CentOS 7.2 Linux 2.6.32 x86_64 PGI y y y n
CentOS 7.2 Linux 2.6.32 x86_64 GNU y y y y
CentOS 7.2 Linux 2.6.32 x86_64 Intel y y y n
Linux 2.6.32-573.18.1.el6.ppc64 y y y n
Compiler versions for each platform are listed in the preceding
"Supported Platforms" table.
More Tested Platforms
=====================
The following platforms are not supported but have been tested for this release.
Linux 2.6.32-573.22.1.el6 GNU C (gcc), Fortran (gfortran), C++ (g++)
#1 SMP x86_64 GNU/Linux compilers:
(mayll/platypus) Version 4.4.7 20120313
Version 4.8.4
PGI C, Fortran, C++ for 64-bit target on
x86-64;
Version 16.10-0
Intel(R) C (icc), C++ (icpc), Fortran (icc)
compilers:
Version 15.0.3.187 (Build 20150407)
MPICH 3.1.4 compiled with GCC 4.9.3
Linux 2.6.32-573.18.1.el6.ppc64 MPICH mpich 3.1.4 compiled with
#1 SMP ppc64 GNU/Linux IBM XL C/C++ for Linux, V13.1
(ostrich) and IBM XL Fortran for Linux, V15.1
Debian 8.4 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1 x86_64 GNU/Linux
gcc, g++ (Debian 4.9.2-10) 4.9.2
GNU Fortran (Debian 4.9.2-10) 4.9.2
(cmake and autotools)
Fedora 24 4.7.2-201.fc24.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
gcc, g++ (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3)
GNU Fortran (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3)
(cmake and autotools)
Ubuntu 16.04.1 4.4.0-38-generic #57-Ubuntu SMP x86_64 GNU/Linux
gcc, g++ (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609
GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609
(cmake and autotools)
Known Problems
==============
This section will be populated with issues reported to help@hdfgroup.org
during pre-release testing. Please report any issues.
|
