summaryrefslogtreecommitdiffstats
path: root/doc/html/H5.format.html
diff options
context:
space:
mode:
authorRobb Matzke <matzke@llnl.gov>1998-08-05 22:22:59 (GMT)
committerRobb Matzke <matzke@llnl.gov>1998-08-05 22:22:59 (GMT)
commit002b1494b79e2fd638a0676745c340a9a9e9d8e7 (patch)
tree9b98efcd614acd02fd3de309d5d9f1d12048230e /doc/html/H5.format.html
parent99506091b3a72fa05f364ea5fb2c51fad3b176ec (diff)
downloadhdf5-002b1494b79e2fd638a0676745c340a9a9e9d8e7.zip
hdf5-002b1494b79e2fd638a0676745c340a9a9e9d8e7.tar.gz
hdf5-002b1494b79e2fd638a0676745c340a9a9e9d8e7.tar.bz2
[svn-r569] Changes since 19980731
---------------------- ./bin/release Added ./Makefile to the distribution again -- it got lost in the changes last week although it isn't all that important a file since it gets clobbered by configure anyway. ./bin/trace ./doc/html/Filters.html ./doc/html/H5.format.html ./doc/html/H5.user.html ./src/H5.c ./src/H5D.c ./src/H5Dprivate.h ./src/H5E.c ./src/H5Epublic.h ./src/H5Farray.c ./src/H5Fistore.c ./src/H5Fprivate.h ./src/H5O.c ./src/H5Ocomp.c ./src/H5Oprivate.h ./src/H5P.c ./src/H5Ppublic.h ./src/H5Sall.c ./src/H5Shyper.c ./src/H5Spoint.c ./src/H5Sprivate.h ./src/H5Ssimp.c ./src/H5Z.c ./src/H5Zprivate.h ./src/H5Zpublic.h ./src/hdf5.h ./test/dsets.c ./tools/h5ls.c Added the data filter pipeline, a generalization of the compression stuff which allows things like checksums, encryption, compression, performance monitoring, etc. See ./doc/html/Filters.html for details -- it replaces the Compression.html doc. ./src/H5T.c Cleaned up debugging output. ./config/linux Added checks for egcs and pgcc and changed optimization flags for the compilers. ./src/H5G.c ./tools/h5dump.c Fixed compiler warnings in these files and others. ./configure.in ./src/H5private.h ./test/mtime.c Added a check for difftime() and defined HDdifftime() to do something else on systems that don't have difftime().
Diffstat (limited to 'doc/html/H5.format.html')
-rw-r--r--doc/html/H5.format.html205
1 files changed, 121 insertions, 84 deletions
diff --git a/doc/html/H5.format.html b/doc/html/H5.format.html
index 7813856..fff60db 100644
--- a/doc/html/H5.format.html
+++ b/doc/html/H5.format.html
@@ -54,8 +54,8 @@
Name: Reserved - not assigned yet</a>
<li><a href="#ReservedMessage_000A"> <!-- 0x000a -->
Name: Reserved - not assigned yet</a>
- <li><a href="#CompressionMessage"> <!-- 0x000b -->
- Name: Data Storage - Compressed</a>
+ <li><a href="#FilterMessage"> <!-- 0x000b -->
+ Name: Data Storage - Filter Pipeline</a>
<li><a href="#AttributeMessage"> <!-- 0x000c -->
Name: Attribute</a>
<li><a href="#NameMessage"> <!-- 0x000d -->
@@ -2433,18 +2433,20 @@
<ol>
<li>The array can be stored in one contiguous area of the file.
The layout requires that the size of the array be constant and
- does not permit chunking or compression. The message stores
- the total size of the array and the offset of an element from
- the beginning of the storage area is computed as in C.
+ does not permit chunking, compression, checksums, encryption,
+ etc. The message stores the total size of the array and the
+ offset of an element from the beginning of the storage area is
+ computed as in C.
<li>The array domain can be regularly decomposed into chunks and
each chunk is allocated separately. This layout supports
- arbitrary element traversals and compression and the chunks
- can be distributed across external raw data files (these
- features are described in other messages). The message stores
- the size of a chunk instead of the size of the entire array;
- the size of the entire array can be calculated by traversing
- the B-tree that stores the chunk addresses.
+ arbitrary element traversals, compression, encryption, and
+ checksums, and the chunks can be distributed across external
+ raw data files (these features are described in other
+ messages). The message stores the size of a chunk instead of
+ the size of the entire array; the size of the entire array can
+ be calculated by traversing the B-tree that stores the chunk
+ addresses.
</ol>
<p>
@@ -2555,22 +2557,21 @@
<b>Format of Data:</b> N/A
<hr>
- <h3><a name="CompressionMessage">Name: Data Storage - Compressed</a></h3>
+ <h3><a name="FilterMessage">Name: Data Storage - Filter Pipeline</a></h3>
<b>Type:</b> 0x000B<BR>
<b>Length:</b> varies<BR>
<b>Status:</b> Optional, may not be repeated.
-
- <p><b>Purpose and Description:</b> Compressed objects are
- datasets which are stored in an HDF file after they have been
- compressed. The encoding algorithm and its parameters are
- stored in a Compression Message in the object header of the
- dataset.
+
+ <p><b>Purpose and Description:</b> This message describes the
+ filter pipeline which should be applied to the data stream by
+ providing filter identification numbers, flags, a name, an
+ client data.
<p>
<center>
<table border align=center cellpadding=4 witdh="80%">
<caption align=top>
- <b>Compression Message</b>
+ <b>Filter Pipeline Message</b>
</caption>
<tr align=center>
@@ -2581,13 +2582,17 @@
</tr>
<tr align=center>
- <td>Method</td>
- <td>Flags</td>
- <td colspan=2>Client Data Size</td>
+ <td>Version</td>
+ <td>Number of Filters</td>
+ <td colspan=2>Reserved</td>
</tr>
<tr align=center>
- <td colspan=4><br>Client Data<br><br></td>
+ <td colspan=4>Reserved</td>
+ </tr>
+
+ <tr align=center>
+ <td colspan=4><br>Filter List<br><br></td>
</tr>
</table>
</center>
@@ -2601,106 +2606,138 @@
</tr>
<tr valign=top>
- <td>Method</td>
- <td>The compression method is a value between zero and 255,
- inclusive, that is used as a index into a compression
- method lookup table. The value zero indicates no
- compression. The values one through 15, inclusive, are
- reserved for methods defined by NCSA. All other values
- are user-defined compression methods.</td>
+ <td>Version</td>
+ <td>The version number for this message. This document
+ describes version one.</td>
</tr>
<tr valign=top>
- <td>Flags</td>
- <td>Eight bits of flags which are passed to the compression
- algorithm. There meaning depends on the compression
- method.</td>
+ <td>Number of Filters</td>
+ <td>The total number of filters described by this
+ message. The maximum possible number of filters in a
+ message is 32.</td>
</tr>
<tr valign=top>
- <td>Client Data Size</td>
- <td>The size in bytes of the optional Client Data
- field.</td>
+ <td>Filter List</td>
+ <td>A description of each filter. A filter description
+ appears in the next table.</td>
</tr>
+ </table>
+ </center>
- <tr valign=top>
- <td>Client Data</td>
- <td>Additional information needed by the compression method
- can be stored in this field. The data will be passed to
- the compression algorithm as a void pointer.</td>
+ <p>
+ <center>
+ <table border align=center cellpadding=4 witdh="80%">
+ <caption align=top>
+ <b>Filter Pipeline Message</b>
+ </caption>
+
+ <tr align=center>
+ <th width="25%">byte</th>
+ <th width="25%">byte</th>
+ <th width="25%">byte</th>
+ <th width="25%">byte</th>
+ </tr>
+
+ <tr align=center>
+ <td colspan=2>Filter Identification</td>
+ <td colspan=2>Name Length</td>
+ </tr>
+
+ <tr align=center>
+ <td colspan=2>Flags</td>
+ <td colspan=2>Client Data Number of Values</td>
+ </tr>
+
+ <tr align=center>
+ <td colspan=4><br>Name<br><br></td>
+ </tr>
+
+ <tr align=center>
+ <td colspan=4><br>Client Data<br><br></td>
+ </tr>
+
+ <tr align=center>
+ <td colspan=4>Padding</td>
</tr>
</table>
</center>
- <p>Sometimes additional redundancy can be added to the data before
- it's compressed to result in a better compression ratio. The
- library doesn't specifically support modeling methods to add
- redundancy, but the effect can be achieved through the use of
- user-defined data types.
-
- <p>The library uses the following compression methods.
+ <p>
<center>
<table align=center width="80%">
- <tr valign=top>
- <td><code>0</code></td>
- <td>No compression: The blocks of data are stored in
- their raw format.</td>
+ <tr>
+ <th width="30%">Field Name</th>
+ <th width="70%">Description</th>
</tr>
<tr valign=top>
- <td><code>1</code></td>
- <td>Deflation: This is the same algorithm used by
- GNU gzip which is a combination Huffman and LZ77
- dictionary encoder. The <code>libz</code> library version
- 1.1.2 or later must be available.</td>
+ <td>Filter Identification</td>
+ <td>This is a unique (except in the case of testing)
+ identifier for the filter. Values from zero through 255
+ are reserved for filters defined by the NCSA HDF5
+ library. Values 256 through 511 have been set aside for
+ use when developing/testing new filters. The remaining
+ values are allocated to specific filters by contacting the
+ <a href="mailto:hdf5dev@ncsa.uiuc.edu">HDF5 Development
+ Team</a>.</td>
</tr>
<tr valign=top>
- <td><code>2</code></td>
- <td>Run length encoding: Not implemented yet.</td>
+ <td>Name Length</td>
+ <td>Each filter has an optional null-terminated ASCII name
+ and this field holds the length of the name including the
+ null termination padded with nulls to be a multiple of
+ eight. If the filter has no name then a value of zero is
+ stored in this field.</td>
</tr>
<tr valign=top>
- <td><code>3</code></td>
- <td>Adaptive Huffman: Not implemented yet.</td>
- </tr>
+ <td>Flags</td>
+ <td>The flags indicate certain properties for a filter. The
+ bit values defined so far are:
- <tr valign=top>
- <td><code>4</code></td>
- <td>Adaptive Arithmetic: Not implemented yet.</td>
+ <dl>
+ <dt><code>bit 1</code>
+ <dd>If set then the filter is an optional filter.
+ During output, if an optional filter fails it will be
+ silently removed from the pipeline.
+ </dl>
</tr>
<tr valign=top>
- <td><code>5</code></td>
- <td>LZ78 Dictionary Encoding: Not implemented yet.</td>
+ <td>Client Data Number of Values</td>
+ <td>Each filter can store a few integer values to control
+ how the filter operates. The number of entries in the
+ Client Data array is stored in this field.</td>
</tr>
<tr valign=top>
- <td><code>6</code></td>
- <td>Adaptive Lempel-Ziv: Similar to Unix
- <code>compress</code>. Not implemented yet.</td>
+ <td>Name</td>
+ <td>If the Name Length field is non-zero then it will
+ contain the size of this field, a multiple of eight. This
+ field contains a null-terminated, ASCII character
+ string to serve as a comment/name for the filter.</td>
</tr>
<tr valign=top>
- <td><code>7-15</code></td>
- <td>Reserved for future use.</td>
+ <td>Client Data</td>
+ <td>This is an array of four-byte integers which will be
+ passed to the filter function. The Client Data Number of
+ Values determines the number of elements in the
+ array.</td>
</tr>
<tr valign=top>
- <td><code>16-255</code></td>
- <td>User-defined.</td>
+ <td>Padding</td>
+ <td>Four bytes of zeros are added to the message at this
+ point if the Client Data Number of Values field contains
+ an odd number.</td>
</tr>
</table>
</center>
- <p>The compression is applied independently to each chunk of
- storage (after data space and data type conversions). If the
- compression is unable to make the chunk smaller than it would
- normally be, the chunk is stored without compression. At the
- library's discretion, chunks which fail the compression can also
- be stored in their raw format.
-
-
<hr>
<h3><a name="AttributeMessage">Name: Attribute</a></h3>
<b>Type:</b> 0x000C<BR>
@@ -3255,7 +3292,7 @@ data-type.
<address><a href="mailto:koziol@ncsa.uiuc.edu">Quincey Koziol</a></address>
<address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address>
<!-- hhmts start -->
-Last modified: Fri Jul 24 15:10:57 EDT 1998
+Last modified: Tue Aug 4 10:04:40 EDT 1998
<!-- hhmts end -->
</body>
</html>