1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
|
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>Compression</title>
</head>
<body>
<h1>Compression</h1>
<h2>1. Introduction</h2>
<p>HDF5 supports compression of raw data by compression methods
built into the library or defined by an application. A
compression method is associated with a dataset when the dataset
is created and is applied independently to each storage chunk of
the dataset.
The dataset must use the <code>H5D_CHUNKED</code> storage
layout. The library doesn't support compression for contiguous
datasets because of the difficulty of implementing random access
for partial I/O, and compact dataset compression is not
supported because it wouldn't produce significant results.
<h2>2. Supported Compression Methods</h2>
<p>The library identifies compression methods with small
integers, with values less than 16 reserved for use by NCSA and
values between 16 and 255 (inclusive) available for general
use. This range may be extended in the future if it proves to
be too small.
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Method Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td><code>H5Z_NONE</code></td>
<td>The default is to not use compression. Specifying
<code>H5Z_NONE</code> as the compression method results
in better perfomance than writing a function that just
copies data because the library's I/O pipeline
recognizes this method and is able to short circuit
parts of the pipeline.</td>
</tr>
<tr valign=top>
<td><code>H5Z_DEFLATE</code></td>
<td>The <em>deflate</em> method is the algorithm used by
the GNU <code>gzip</code>program. It's a combination of
a Huffman encoding followed by a 1977 Lempel-Ziv (LZ77)
dictionary encoding. The aggressiveness of the
compression can be controlled by passing an integer value
to the compressor with <code>H5Pset_deflate()</code>
(see below). In order for this compression method to be
used, the HDF5 library must be configured and compiled
in the presence of the GNU zlib version 1.1.2 or
later.</td>
</tr>
<tr valign=top>
<td><code>H5Z_RES_<em>N</em></code></td>
<td>These compression methods (where <em>N</em> is in the
range two through 15, inclusive) are reserved by NCSA
for future use.</td>
</tr>
<tr valign=top>
<td>Values of <em>N</em> between 16 and 255, inclusive</td>
<td>These values can be used to represent application-defined
compression methods. We recommend that methods under
testing should be in the high range and when a method is
about to be published it should be given a number near
the low end of the range (or even below 16). Publishing
the compression method and its numeric ID will make a
file sharable.</td>
</tr>
</table>
</center>
<p>Setting the compression for a dataset to a method which was
not compiled into the library and/or not registered by the
application is allowed, but writing to such a dataset will
silently <em>not</em> compress the data. Reading a compressed
dataset for a method which is not available will result in
errors (specifically, <code>H5Dread()</code> will return a
negative value). The errors will be displayed in the
compression statistics if the library was compiled with
debugging turned on for the "z" package. See the
section on diagnostics below for more details.
<h2>3. Application-Defined Methods</h2>
<p>Compression methods 16 through 255 can be defined by an
application. As mentioned above, methods that have not been
released should use high numbers in that range while methods
that have been published will be assigned an official number in
the low region of the range (possibly less than 16). Users
should be aware that using unpublished compression methods
results in unsharable files.
<p>A compression method has two halves: one have handles
compression and the other half handles uncompression. The
halves are implemented as functions
<code><em>method</em>_c</code> and
<code><em>method</em>_u</code> respectively. One should not use
the names <code>compress</code> or <code>uncompress</code> since
they are likely to conflict with other compression libraries
(like the GNU zlib).
<p>Both the <code><em>method</em>_c</code> and
<code><em>method</em>_u</code> functions take the same arguments
and return the same values. They are defined with the type:
<dl>
<dt><code>typedef size_t (*H5Z_func_t)(unsigned int
<em>flags</em>, size_t <em>cd_size</em>, const void
*<em>client_data</em>, size_t <em>src_nbytes</em>, const
void *<em>src</em>, size_t <em>dst_nbytes</em>, void
*<em>dst</em>/*out*/)</code>
<dd>The <em>flags</em> are an 8-bit vector which is stored in
the file and which is defined when the compression method is
defined. The <em>client_data</em> is a pointer to
<em>cd_size</em> bytes of configuration data which is also
stored in the file. The function compresses or uncompresses
<em>src_nbytes</em> from the source buffer <em>src</em> into
at most <em>dst_nbytes</em> of the result buffer <em>dst</em>.
The function returns the number of bytes written to the result
buffer or zero if an error occurs. But if a result buffer
overrun occurs the function should return a value at least as
large as <em>dst_size</em> (the uncompressor will see an
overrun only for corrupt data).
</dl>
<p>The application associates the pair of functions with a name
and a method number by calling <code>H5Zregister()</code>. This
function can also be used to remove a compression method from
the library by supplying null pointers for the functions.
<dl>
<dt><code>herr_t H5Zregister (H5Z_method_t <em>method</em>,
const char *<em>name</em>, H5Z_func_t <em>method_c</em>,
H5Z_func_t <em>method_u</em>)</code>
<dd>The pair of functions to be used for compression
(<em>method_c</em>) and uncompression (<em>method_u</em>) are
associated with a short <em>name</em> used for debugging and a
<em>method</em> number in the range 16 through 255. This
function can be called as often as desired for a particular
compression method with each call replacing the information
stored by the previous call. Sometimes it's convenient to
supply only one half of the compression, for instance in an
application that opens files for read-only. Compression
statistics for the method are accumulated across calls to this
function.
</dl>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Registering an
Application-Defined Compression Method</h4></caption>
<tr>
<td>
<p>Here's a simple-minded "compression" method
that just copies the input value to the output. It's
similar to the <code>H5Z_NONE</code> method but
slower. Compression and uncompression are performed
by the same function.
<p><code><pre>
size_t
bogus (unsigned int flags,
size_t cd_size, const void *client_data,
size_t src_nbytes, const void *src,
size_t dst_nbytes, void *dst/*out*/)
{
memcpy (dst, src, src_nbytes);
return src_nbytes;
}
</pre></code>
<p>The function could be registered as method 250 as
follows:
<p><code><pre>
#define H5Z_BOGUS 250
H5Zregister (H5Z_BOGUS, "bogus", bogus, bogus);
</pre></code>
<p>The function can be unregistered by saying:
<p><code><pre>
H5Zregister (H5Z_BUGUS, "bogus", NULL, NULL);
</pre></code>
<p>Notice that we kept the name "bogus" even
though we unregistered the functions that perform the
compression and uncompression. This makes compression
statistics more understandable when they're printed.
</td>
</tr>
</table>
</center>
<h2>4. Enabling Compression for a Dataset</h2>
<p>If a dataset is to be compressed then the compression
information must be specified when the dataset is created since
once a dataset is created compression parameters cannot be
adjusted. The compression is specified through the dataset
creation property list (see <code>H5Pcreate()</code>).
<dl>
<dt><code>herr_t H5Pset_deflate (hid_t <em>plist</em>, int
<em>level</em>)</code>
<dd>The compression method for dataset creation property list
<em>plist</em> is set to <code>H5Z_DEFLATE</code> and the
aggression level is set to <em>level</em>. The <em>level</em>
must be a value between one and nine, inclusive, where one
indicates no (but fast) compression and nine is aggressive
compression.
<br><br>
<dt><code>int H5Pget_deflate (hid_t <em>plist</em>)</code>
<dd>If dataset creation property list <em>plist</em> is set to
use <code>H5Z_DEFLATE</code> compression then this function
will return the aggression level, an integer between one and
nine inclusive. If <em>plist</em> isn't a valid dataset
creation property list or it isn't set to use the deflate
method then a negative value is returned.
<br><br>
<dt><code>herr_t H5Pset_compression (hid_t <em>plist</em>,
H5Z_method_t <em>method</em>, unsigned int <em>flags</em>,
size_t <em>cd_size</em>, const void *<em>client_data</em>)</code>
<dd>This is a catch-all function for defining compresion methods
and is intended to be called from a wrapper such as
<code>H5Pset_deflate()</code>. The dataset creation property
list <em>plist</em> is adjusted to use the specified
compression method. The <em>flags</em> is an 8-bit vector
which is stored in the file as part of the compression message
and passed to the compress and uncompress functions. The
<em>client_data</em> is a byte array of length
<em>cd_size</em> which is copied to the file and passed to the
compress and uncompress methods.
<br><br>
<dt><code>H5Z_method_t H5Pget_compression (hid_t <em>plist</em>,
unsigned int *<em>flags</em>, size_t *<em>cd_size</em>, void
*<em>client_data</em>)</code>
<dd>This is a catch-all function for querying the compression
method associated with dataset creation property list
<em>plist</em> and is intended to be called from a wrapper
function such as <code>H5Pget_deflate()</code>. The
compression method (or a negative value on error) is returned
by value, and compression flags and client data is returned by
argument. The application should allocate the
<em>client_data</em> and pass its size as the
<em>cd_size</em>. On return, <em>cd_size</em> will contain
the actual size of the client data. If <em>client_data</em>
is not large enough to hold the entire client data then
<em>cd_size</em> bytes are copied into <em>client_data</em>
and <em>cd_size</em> is set to the total size of the client
data, a value larger than the original.
</dl>
<p>It is possible to set the compression to a method which hasn't
been defined with <code>H5Zregister()</code> and which isn't
supported as a predefined method (for instance, setting the
method to <code>H5Z_DEFLATE</code> when the GNU zlib isn't
available). If that happens then data will be written to the
file in its uncompressed form and the compression statistics
will show failures for the compression.
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Statistics for an
Unsupported Compression Method</h4></caption>
<tr>
<td>
<p>If an application attempts to use an unsupported
method then the compression statistics will show large
numbers of compression errors and no data
uncompressed.
<p><code><pre>
H5Z: compression statistics accumulated over life of library:
Method Total Overrun Errors User System Elapsed Bandwidth
------ ----- ------- ------ ---- ------ ------- ---------
deflate-c 160000 0 160000 0.00 0.01 0.01 1.884e+07
deflate-u 0 0 0 0.00 0.00 0.00 NaN
</pre></code>
<p>This example is from a program that tried to use
<code>H5Z_DEFLATE</code> on a system that didn't have
the GNU zlib to write to a dataset and then read the
result. The read and write both succeeded but the
data was not compressed.
</td>
</tr>
</table>
</center>
<h2>5. Compression Diagnostics</h2>
<p>If the library is compiled with debugging turned on for the H5Z
layer (usually as a result of <code>configure --enable-debug=z</code>)
then statistics about data compression are printed when the
application exits normally or the library is closed. The
statistics are written to the standard error stream and include
two lines for each compression method that was used: the first
line shows compression statistics while the second shows
uncompression statistics. The following fields are displayed:
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Method</td>
<td>This is the name of the method as defined with
<code>H5Zregister()</code> with the letters
"-c" or "-u" appended to indicate
compression or uncompression.</td>
</tr>
<tr valign=top>
<td>Total</td>
<td>The total number of bytes compressed or decompressed
including buffer overruns and errors. Bytes of
non-compressed data are counted.</td>
</tr>
<tr valign=top>
<td>Overrun</td>
<td>During compression, if the algorithm causes the result
to be at least as large as the input then a buffer
overrun error occurs. This field shows the total number
of bytes from the Total column which can be attributed to
overruns. Overruns for decompression can only happen if
the data has been corrupted in some way and will result
in failure of <code>H5Dread()</code>.</td>
</tr>
<tr valign=top>
<td>Errors</td>
<td>If an error occurs during compression the data is
stored in it's uncompressed form; and an error during
uncompression causes <code>H5Dread()</code> to return
failure. This field shows the number of bytes of the
Total column which can be attributed to errors.</td>
</tr>
<tr valign=top>
<td>User, System, Elapsed</td>
<td>These are the amount of user time, system time, and
elapsed time in seconds spent by the library to perform
compression. Elapsed time is sensitive to system
load. These times may be zero on operating systems that
don't support the required operations.</td>
</tr>
<tr valign=top>
<td>Bandwidth</td>
<td>This is the compression bandwidth which is the total
number of bytes divided by elapsed time. Since elapsed
time is subject to system load the bandwidth numbers
cannot always be trusted. Furthermore, the bandwidth
includes overrun and error bytes which may significanly
taint the value.</td>
</tr>
</table>
</center>
<p>
<center>
<table border align=center width="100%">
<caption align=bottom><h4>Example: Compression
Statistics</h4></caption>
<tr>
<td>
<p><code><pre>
H5Z: compression statistics accumulated over life of library:
Method Total Overrun Errors User System Elapsed Bandwidth
------ ----- ------- ------ ---- ------ ------- ---------
deflate-c 160000 200 0 0.62 0.74 1.33 1.204e+05
deflate-u 120000 0 0 0.11 0.00 0.12 9.885e+05
</pre></code>
</td>
</tr>
</table>
</center>
<hr>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- Created: Fri Apr 17 13:39:35 EDT 1998 -->
<!-- hhmts start -->
Last modified: Fri Apr 17 16:15:21 EDT 1998
<!-- hhmts end -->
</body>
</html>
|
