1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
|
HDF5 version 1.10.0-pre1 currently under development
================================================================================
INTRODUCTION
This document describes the differences between HDF5-1.8.* and
HDF5 1.10.0 release, and contains information on the platforms
tested and known problems in HDF5-1.10.0.
Links to HDF5 1.10.0 source code can be found on The HDF Group's
development FTP server at the following location:
http://www.hdfgroup.org/HDF5/release/obtain5110.html
User documentation be accessed directly at this location:
http://www.hdfgroup.org/HDF5/docNewFeatures/
For more information, see the HDF5 home page:
http://www.hdfgroup.org/HDF5/
If you have any questions or comments, please send them to the HDF Help Desk:
help@hdfgroup.org
CONTENTS
- New Features
- Issues Addressed in this Release
- Tested Configuration Features Summary
- More Tested Platforms
- Known Problems
New Features
============
This release supports the following features:
Configuration
-------------
- Autotools configuration has been extensively reworked
The autotools configuration options have been updated to allow more fine-
grained control of the build options and to correct some bugs. See
configure --help for comprehensive information on each option.
Specific changes:
* --enable-debug and --enable-production are no longer accepted. Use
--enable-build-mode=(debug | production) instead. These set appropriate
defaults for the other configuration options (symbols, optimization,
etc.). These defaults can be overridden by the user.
* Extra debug output messages are no longer enabled with
--enable-debug=<package list>. Use --enable-internal-debug=<pkg list>
instead.
* A new --enable-symbols option allows symbols to be generated
independently of the build mode. --disable-symbols can be used to strip
symbols from the binary.
* A new --enable-asserts option sets/unsets NDEBUG. This is independent of
the build mode. This also enables some extra low-overhead debug checks
in the library.
* A new --enable-profiling option sets profiling flags. This is
independent of the build mode.
* A new --enable-optimization option sets the optimization level. This is
independent of the build mode.
* Many of these options can take a flags string that will be used to
build the library. This can be useful for specifying custom optimization
flags (e.g., -Os, -Ofast), etc.
* gnu C++ and Fortran use configure sub-files that update the build flags
and turn on warnings. The increase in warnings when building these
wrapper libraries is due to these flag changes, not a decrease in code
quality.
* The option to clear file buffers has been removed. Any buffer that will
eventually be written to disk will now always be memset to zero. This
prevents the previous contents of the buffer from being written to the
disk if the buffer contents are not completely overwritten, which has
security implications.
- LFS changes
The way the autotools handle large file support (LFS) has been overhauled
in this release.
* We assume ftello and fseeko exist
* We no longer explicitly use the *64 I/O functions. Instead we rely on a
mapping provided by _FILE_OFFSET_BITS or its equivalent.
* _LARGEFILE(64)_SOURCE is no longer exported via AM_CPPFLAGS.
Parallel Library:
-----------------
- Collective Metadata I/O
Calls for HDF5 metadata can result in many small reads and writes. On metadata
reads, collective metadata I/O can improve performance by allowing the library
to perform optimizations when reading the metadata, by having one rank read the
data and broadcasting it to all other ranks.
Collective metadata I/O improves metadata write performance through the
construction of an MPI derived datatype that is then written collectively
in a single call. For more information see
https://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesCollectiveMetadataIoDocs.html
Library:
--------
- Concurrent access to HDF5 file - Single Writer/ Multple Reader (SWMR)
The Single Writer/ Multiple Reader or SWMR feature enables users to read
data concurrently while writing it. Communications between the processes
and file locking are not required. The processes can run on the same or
on different platforms, as long as they share a common file system that
is POSIX compliant.
For more information, see the Single-Writer/Multiple-Reader (SWMR)
Documentation at
https://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesSwmrDocs.html
- Virtual Dataset (VDS)
The VDS feature enables data to be accessed across HDF5 files using standard
HDF5 objects such as groups and datasets without rewriting or rearranging
the data. An HDF5 virtual dataset (VDS) is an HDF5 dataset that is composed of
source HDF5 datasets in a predefined mapping. VDS can be used with the SWMR
feature. For documentation check
https://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesVirtualDatasetDocs.html
- Persistent Free File Space Tracking
Usage patterns when working with an HDF5 file sometimes result in wasted space
within the file. This can also impair access times when working with the
resulting files. The new file space management feature provides strategies
for managing space in a file to improve performance in both of these areas.
For more information see
http://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesFileSpaceMgmtDocs.html
C++ Library:
------------
- New member function added to H5::ArrayType
The assignment operator ArrayType::operator= is added because ArrayType
has pointer data members.
(BMR, 2016/03/07, HDFFV-9562)
Tools:
------
- h5watch
Allows users to output new records appended to a dataset iunder SWMR access
as it grows. The functionality is similar to the Unix user command "tail"
with the follow option, which outputs appended data as the file grows.
For more information see
http://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesSwmrDocs.html#Tools
- h5format_convert
The tool allows users to convert the indexing type of a chunked dataset made
with a 1.10.x version of the HDF5 library when the latest file format is used
to the 1.8.x version 1 B-tree indexing type. For example, datasets created
using SWMR access, can be converted to be accessed by the HDF5 1.18 library
and tools. The tools doesn't rewrite raw data, but HDF5 metadata only.
High-Level APIs:
----------------
- H5DOappend
The function appends data to a dataset along a specified dimension.
C Packet Table API
------------------
- Replacement of a public function
The existing function H5PTcreate_fl limits applications to deflate
compression only. The public function H5PTcreate is added to replace
H5PTcreate_fl. H5PTcreate takes a property list ID to provide
flexibility on creation properties.
hid_t H5PTcreate(hid_t loc_id, const char *dset_name,
hid_t dtype_id, hsize_t chunk_size, hid_t plist_id);
(BMR, 2016/03/04, HDFFV-8623)
- New public functions
Two accessor functions were added per HDFFV-8623/patch 003.
/* Returns the ID of the dataset associated with the packet table */
hid_t H5PTget_dataset(hid_t table_id);
/* Returns the ID of the datatype the packet table uses */
hid_t H5PTget_type(hid_t table_id);
(BMR, 2016/03/04, HDFFV-8623)
- Regarding #ifdef VLPT_REMOVED
The #ifdef VLPT_REMOVED blocks are removed from the PT library source
except the following cases:
+ H5PTis_varlen() is made available again.
+ H5PTfree_vlen_readbuff() now became H5PTfree_vlen_buff()
(BMR, 2016/03/04, HDFFV-442)
C++ Packet Table API
--------------------
- New constructor
An overloaded constructor is added to FL_PacketTable and takes a property
list ID to provide flexibility on creation properties.
FL_PacketTable(hid_t fileID, hid_t plist_id, const char* name, hid_t dtypeID, hsize_t chunkSize);
(BMR, 2016/03/08, HDFFV-8623)
- New public functions
Two accessor wrappers to class PacketTable, per HDFFV-8623/patch 004.
/* Returns the ID of the dataset associated with the packet table */
hid_t PacketTable::GetDataset()
/* Returns the ID of the datatype the packet table uses */
hid_t PacketTable::GetDataset()
(BMR, 2016/03/04, HDFFV-8623)
- Member functions having "char*" as an argument
Overloaded functions were added to provide "const char*" argument, the
existing version will be deprecated.
(BMR, 2016/03/04)
- Regarding #ifdef VLPT_REMOVED
The #ifdef VLPT_REMOVED blocks are removed from the PT library source
except the following cases:
+ VL_PacketTable::IsVariableLength() is moved to PacketTable
+ VL_PacketTable::FreeReadBuff() now became PacketTable::FreeBuff()
(BMR, 2016/03/04, HDFFV-442)
Java wrapper library
--------------------
Java HDF5 JNI library has been integrated into the HDF5 repo, the configure
option is "--enable-java" and the CMake option is HDF5_BUILD_JAVA:BOOL=ON.
The package hierarchy has changed from the HDF5 1.8 JNI, which was
"ncsa.hdf.hdflib.hdf5", to HDF5 1.10, "hdf.hdflib.hdf5".
Also there are a number of new APIs added, including those for VDS and SWMR.
Other important changes:
-----------------------
hid_t type was changed from 32-bit to a 64-bit value.
Documentation
-------------
See http://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesSwmrDocs.html#Tools
Issues Addressed in this Release since alpha1
=============================================
- Wrong settings returned for a dataset access property list.
One couldn't get the correct access property setting with the following functions calls
H5Pget_virtual_printf_gap
H5Pget_virtual_view
H5Pget_efile_prefix
when a dataset access property list was obtained from H5Dget_access_plist(). Fixed.
(DER and NAF 2016-03-14 HDFFV-9716)
- h5dump segfaulted when provided with invalid arguments or when optional aruments are last option
on command line.
When h5dump was provided with the name of a non existing file or when optional arguments were the
last option on the command line, h5dump failed. Fixed.
(ADB 2016-02-28 HDFFV-9639, HDFFV-9684)
- HDF5 library segfaults on a corrupted file.
The HDF5 library didn't propagate an error when encounting a corruped metadata in an HDF5 file.
The issue was fixed for a specific filei porvided by a user. If you still see the problem,
please contact help@hdfgroup.org
(MC 2016-02-18 HDFFV-9670)
- When the latest file format is used, and when a chunked dataset is created with a datatype with the size
bigger than a chunk size, the data cannot be read back.
The issue was reported for the chunked datasets with a string datatype, and was confirmed for other i
datatypes with the sizes bigger than the chunk size in bytes. Fixed.
(JM 2016-02-13 HDFFV-9672)
- Control over the location of the external files
New API H5Pset_efile_prefix was added to specify locations of the external files.
This enhancements was contributed by Steffen Kieß.
(DER 2016-02-04 HDFFV-8740)
Issues Addressed in this Release since alpha0
=============================================
- h5format_convert tool didn't downgrade version of the superblock. Fixed. See
Known Problems section.
(EIP 2016-01-11)
- It was reported that alpha0 crashed when used with multiple threads.
The issue exists in the HDF5 library versions 1.8 and 1.9.
The problem is related to a shared file pointer used in the misc. data
structures. The thread-safe library exposed paths in the library where
file pointer becomes invalid. Alpha1 release contains the fixes for the specific
use case as decribed in HDFFV-9643.
We will keep working on identifying other paths in the library with similar problem
and fixing the issue.
(EIP 2016-01-15)
Supported Platforms
===================
The following platforms are supported and have been tested for this release.
They are built with the configure process unless specified otherwise.
AIX 6.1 xlc/xlc_r 10.1.0.5
(NASA G-ADA) xlC/xlC_r 10.1.0.5
xlf90/xlf90_r 12.1.0.6
Linux 2.6.32-573.18.1.el6 GNU C (gcc), Fortran (gfortran), C++ (g++)
#1 SMP x86_64 GNU/Linux compilers:
(mayll/platypus) Version 4.4.7 20120313 (Red Hat 4.4.7-16)
Version 4.9.3, Version 5.2.0
PGI C, Fortran, C++ for 64-bit target on
x86-64;
Version 15.7-0
Intel(R) C (icc), C++ (icpc), Fortran (icc)
compilers:
Version 15.0.3.187 Build 20150407
MPICH 3.1.4 compiled with GCC 4.9.3
Linux 2.6.32-504.8.1.el6.ppc64 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11)
#1 SMP ppc64 GNU/Linux g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11)
(ostrich) GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11)
IBM XL C/C++ V13.1
IBM XL Fortran V15.1
Linux 3.10.0-327.10.1.el7 GNU C (gcc), Fortran (gfortran), C++ (g++)
#1 SMP x86_64 GNU/Linux compilers:
(kituo/moohan) Version 4.8.5 20150623 (Red Hat 4.8.5-4)
Version 4.9.3, Version 5.2.0
Intel(R) C (icc), C++ (icpc), Fortran (icc)
compilers:
Version 15.0.3.187 Build 20150407
MPICH 3.1.4 compiled with GCC 4.9.3
SunOS 5.11 32- and 64-bit Sun C 5.12 SunOS_sparc
(emu) Sun Fortran 95 8.6 SunOS_sparc
Sun C++ 5.12 SunOS_sparc
Windows 7 Visual Studio 2012 w/ Intel Fortran 15 (cmake)
Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Visual Studio 2015 w/ Intel Fortran 16 (cmake)
Cygwin(CYGWIN_NT-6.1 2.2.1(0.289/5/3) gcc(4.9.3) compiler and gfortran)
(cmake and autotools)
Windows 7 x64 Visual Studio 2012 w/ Intel Fortran 15 (cmake)
Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Visual Studio 2015 w/ Intel Fortran 16 (cmake)
Windows 8.1 Visual Studio 2012 w/ Intel Fortran 15 (cmake)
Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Windows 8.1 x64 Visual Studio 2012 w/ Intel Fortran 15 (cmake)
Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Mac OS X Mt. Lion 10.8.5 Apple clang/clang++ version 5.1 from Xcode 5.1
64-bit gfortran GNU Fortran (GCC) 4.8.2
(swallow/kite) Intel icc/icpc/ifort version 15.0.3
Mac OS X Mavericks 10.9.5 Apple clang/clang++ version 6.0 from Xcode 6.2.0
64-bit gfortran GNU Fortran (GCC) 4.9.2
(wren/quail) Intel icc/icpc/ifort version 15.0.3
Mac OS X Yosemite 10.10.5 Apple clang/clang++ version 6.0 from Xcode 7.0.0
64-bit gfortran GNU Fortran (GCC) 4.9.2
(osx1010dev/osx1010test) Intel icc/icpc/ifort version 15.0.3
Mac OS X El Capitan 10.11.3 Apple clang/clang++ version 7.0.2 from Xcode 7.0.2
64-bit gfortran GNU Fortran (GCC) 5.2.0
(osx1010dev/osx1010test) Intel icc/icpc/ifort version 15.0.3
Tested Configuration Features Summary
=====================================
In the tables below
y = tested
n = not tested in this release
C = Cluster
W = Workstation
x = not working in this release
dna = does not apply
( ) = footnote appears below second table
<blank> = testing incomplete on this feature or platform
Platform C F90/ F90 C++ zlib SZIP
parallel F2003 parallel
SunOS 5.11 32-bit n y/y n y y y
SunOS 5.11 64-bit n y/y n y y y
Windows 7 y y/y n y y y
Windows 7 x64 y y/y n y y y
Windows 7 Cygwin n y/y n y y n
Windows 8.1 n y/y n y y y
Windows 8.1 x64 n y/y n y y y
Mac OS X Mountain Lion 10.8.5 64-bit n y/y n y y y
Mac OS X Mavericks 10.9.5 64-bit n y/y n y y y
Mac OS X Yosemeti 10.10.5 64-bit n y/y n y y y
AIX 6.1 32- and 64-bit n y/n n y y y
CentOS 6.7 Linux 2.6.32 x86_64 GNU y y/y y y y y
CentOS 6.7 Linux 2.6.32 x86_64 Intel n y/y n y y y
CentOS 6.7 Linux 2.6.32 x86_64 PGI n y/y n y y y
CentOS 7.1 Linux 3.10.0 x86_64 GNU y y/y y y y y
CentOS 7.1 Linux 3.10.0 x86_64 Intel n y/y n y y y
Linux 2.6.32-431.11.2.el6.ppc64 n y/n n y y y
Platform Shared Shared Shared Thread-
C libs F90 libs C++ libs safe
SunOS 5.11 32-bit y y y y
SunOS 5.11 64-bit y y y y
Windows 7 y y y y
Windows 7 x64 y y y y
Windows 7 Cygwin n n n y
Windows 8.1 y y y y
Windows 8.1 x64 y y y y
Mac OS X Mountain Lion 10.8.5 64-bit y n y y
Mac OS X Mavericks 10.9.5 64-bit y n y y
Mac OS X Yosemeti 10.10.5 64-bit y n y y
AIX 6.1 32- and 64-bit y n n y
CentOS 6.7 Linux 2.6.32 x86_64 GNU y y y y
CentOS 6.7 Linux 2.6.32 x86_64 Intel y y y y
CentOS 6.7 Linux 2.6.32 x86_64 PGI y y y y
CentOS 7.1 Linux 3.10.0 x86_64 GNU y y y y
CentOS 7.1 Linux 3.10.0 x86_64 Intel y y y y
Linux 2.6.32-431.11.2.el6.ppc64 y y y y
Compiler versions for each platform are listed in the preceding
"Supported Platforms" table.
More Tested Platforms
=====================
The following platforms are not supported but have been tested for this release.
Linux 2.6.18-431.11.2.el6 g95 (GCC 4.0.3 (g95 0.94!)
#1 SMP x86_64 GNU/Linux
(platypus)
Windows 7 Visual Studio 2008 (cmake)
Windows 7 x64 Visual Studio 2008 (cmake)
Windows 7 x64 Visual Studio 2010 (cmake) with SWMR using GPFS
Windows 10 Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Windows 10 x64 Visual Studio 2013 w/ Intel Fortran 15 (cmake)
Debian7.5.0 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux
gcc (Debian 4.7.2-5) 4.7.2
GNU Fortran (Debian 4.7.2-5) 4.7.2
(cmake and autotools)
Fedora20 3.15.3-200.fc20.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
gcc (GCC) 4.8.3 20140624 (Red Hat 4.8.3-1)
GNU Fortran (GCC) 4.8.3 20140624 (Red Hat 4.8.3-1)
(cmake and autotools)
SUSE 13.1 3.11.10-17-desktop #1 SMP PREEMPT x86_64 x86_64 x86_64 GNU/Linux
gcc (SUSE Linux) 4.8.1
GNU Fortran (SUSE Linux) 4.8.1
(cmake and autotools)
Ubuntu 14.04 3.13.0-35-generic #62-Ubuntu SMP x86_64 GNU/Linux
gcc (Ubuntu/Linaro 4.9.1-0ubuntu1) 4.9.1
GNU Fortran (Ubuntu/Linaro 4.9.1-0ubuntu1) 4.9.1
(cmake and autotools)
hopper.nersc.gov PrgEnv-gnu/5.2.40
gcc (GCC) 4.9.2 20141030 (Cray Inc.)
GNU Fortran (GCC) 4.9.2 20141030 (Cray Inc.)
g++ (GCC) 4.9.2 20141030 (Cray Inc.)
Known Problems
==============
WARNINGS: This section is under contsruction. It will be updated for the first HDF5 1.10.0-beta
release.
- h5format_convert only works with the files that contain chunked datasets.
- h5repack tool fails to downgrade object layout message to make file
1.8 compatible.
|
