diff options
Diffstat (limited to 'doc/html/storage.html')
| -rw-r--r-- | doc/html/storage.html | 274 |
1 files changed, 274 insertions, 0 deletions
diff --git a/doc/html/storage.html b/doc/html/storage.html new file mode 100644 index 0000000..87ea54d --- /dev/null +++ b/doc/html/storage.html @@ -0,0 +1,274 @@ +<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> +<html> + <head> + <title>Raw Data Storage in HDF5</title> + </head> + + <body> + <h1>Raw Data Storage in HDF5</h1> + + <p>This document describes the various ways that raw data is + stored in an HDF5 file and the object header messages which + contain the parameters for the storage. + + <p>Raw data storage has three components: the mapping from some + logical multi-dimensional element space to the linear address + space of a file, compression of the raw data on disk, and + striping of raw data across multiple files. These components + are orthogonal. + + <p>Some goals of the storage mechanism are to be able to + efficently store data which is: + + <dl> + <dt>Small + <dd>Small pieces of raw data can be treated as meta data and + stored in the object header. This will be achieved by storing + the raw data in the object header with message 0x0006. + Compression and striping are not supported in this case. + + <dt>Complete Large + <dd>The library should be able to store large arrays + contiguously in the file provided the user knows the final + array size a priori. The array can then be read/written in a + single I/O request. This is accomplished by describing the + storage with object header message 0x0005. Compression and + striping are not supported in this case. + + <dt>Sparse Large + <dd>A large sparse raw data array should be stored in a manner + that is space-efficient but one in which any element can still + be accessed in a reasonable amount of time. Implementation + details are below. + + <dt>Dynamic Size + <dd>One often doesn't have prior knowledge of the size of an + array. It would be nice to allow arrays to grow dynamically in + any dimension. It might also be nice to allow the array to + grow in the negative dimension directions if convenient to + implement. Implementation details are below. + + <dt>Subslab Access + <dd>Some multi-dimensional arrays are almost always accessed by + subslabs. For instance, a 2-d array of pixels might always be + accessed as smaller 1k-by-1k 2-d arrays always aligned on 1k + index values. We should be able to store the array in such a + way that striding though the entire array is not necessary. + Subslab access might also be useful with compression + algorithms where each storage slab can be compressed + independently of the others. Implementation details are below. + + <dt>Compressed + <dd>Various compression algorithms can be applied to the entire + array. We're not planning to support separate algorithms (or a + single algorithm with separate parameters) for each chunk + although it would be possible to implement that in a manner + similar to the way striping across files is + implemented. + + <dt>Striped Across Files + <dd>The array access functions should support arrays stored + discontiguously across a set of files. + </dl> + + <h1>Implementation of Indexed Storage</h1> + + <p>The Sparse Large, Dynamic Size, and Subslab Access methods + share so much code that they can be described with a single + message. The new Indexed Storage Message (<code>0x0008</code>) + will replace the old Chunked Object (<code>0x0009</code>) and + Sparse Object (<code>0x000A</code>) Messages. + + <p> + <center> + <table border cellpadding=4 width="60%"> + <caption align=bottom> + <b>The Format of the Indexed Storage Message</b> + </caption> + <tr align=center> + <th width="25%">byte</th> + <th width="25%">byte</th> + <th width="25%">byte</th> + <th width="25%">byte</th> + </tr> + + <tr align=center> + <td colspan=4><br>Address of B-tree<br><br></td> + </tr> + <tr align=center> + <td>Number of Dimensions</td> + <td>Reserved</td> + <td>Reserved</td> + <td>Reserved</td> + </tr> + <tr align=center> + <td colspan=4>Reserved (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Alignment for Dimension 0 (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Alignment for Dimension 1 (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>...</td> + </tr> + <tr align=center> + <td colspan=4>Alignment for Dimension N (4 bytes)</td> + </tr> + </table> + </center> + + <p>The alignment fields indicate the alignment in logical space to + use when allocating new storage areas on disk. For instance, + writing every other element of a 100-element one-dimensional + array (using one HDF5 I/O partial write operation per element) + that has unit storage alignment would result in 50 + single-element, discontiguous storage segments. However, using + an alignment of 25 would result in only four discontiguous + segments. The size of the message varies with the number of + dimensions. + + <p>A B-tree is used to point to the discontiguous portions of + storage which has been allocated for the object. All keys of a + particular B-tree are the same size and are a function of the + number of dimensions. It is therefore not possible to change the + dimensionality of an indexed storage array after its B-tree is + created. + + <p> + <center> + <table border cellpadding=4 width="60%"> + <caption align=bottom> + <b>The Format of a B-Tree Key</b> + </caption> + <tr align=center> + <th width="25%">byte</th> + <th width="25%">byte</th> + <th width="25%">byte</th> + <th width="25%">byte</th> + </tr> + + <tr align=center> + <td colspan=4>External File Number or Zero (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Chunk Offset in Dimension 0 (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Chunk Offset in Dimension 1 (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>...</td> + </tr> + <tr align=center> + <td colspan=4>Chunk Offset in Dimension N (4 bytes)</td> + </tr> + </table> + </center> + + <p>The keys within a B-tree obey an ordering based on the chunk + offsets. If the offsets in dimension-0 are equal, then + dimension-1 is used, etc. The External File Number field + contains a 1-origin offset into the External File List message + which contains the name of the external file in which that chunk + is stored. + + <h1>Implementation of Striping</h1> + + <p>The indexed storage will support arbitrary striping at the + chunk level; each chunk can be stored in any file. This is + accomplished by using the External File Number field of an + indexed storage B-tree key as a 1-origin offset into an External + File List Message (0x0009) which takes the form: + + <p> + <center> + <table border cellpadding=4 width="60%"> + <caption align=bottom> + <b>The Format of the External File List Message</b> + </caption> + <tr align=center> + <th width="25%">byte</th> + <th width="25%">byte</th> + <th width="25%">byte</th> + <th width="25%">byte</th> + </tr> + + <tr align=center> + <td colspan=4><br>Name Heap Address<br><br></td> + </tr> + <tr align=center> + <td colspan=4>Number of Slots Allocated (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Number of File Names (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Byte Offset of Name 1 in Heap (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>Byte Offset of Name 2 in Heap (4 bytes)</td> + </tr> + <tr align=center> + <td colspan=4>...</td> + </tr> + <tr align=center> + <td colspan=4><br>Unused Slot(s)<br><br></td> + </tr> + </table> + </center> + + <p>Each indexed storage array that has all or part of its data + stored in external files will contain a single external file + list message. The size of the messages is determined when the + message is created, but it may be possible to enlarge the + message on demand by moving it. At this time, it's not possible + for multiple arrays to share a single external file list + message. + + <dl> + <dt><code> + H5O_efl_t *H5O_efl_new (H5G_entry_t *object, intn + nslots_hint, intn heap_size_hint) + </code> + <dd>Adds a new, empty external file list message to an object + header and returns a pointer to that message. The message + acts as a cache for file descriptors of external files that + are open. + + <p><dt><code> + intn H5O_efl_index (H5O_efl_t *efl, const char *filename) + </code> + <dd>Gets the external file index number for a particular file name. + If the name isn't in the external file list then it's added to + the H5O_efl_t struct and immediately written to the object + header to which the external file list message belongs. Name + comparison is textual. Each name should be relative to the + directory which contains the HDF5 file. + + <p><dt><code> + H5F_low_t *H5O_efl_open (H5O_efl_t *efl, intn index, uintn mode) + </code> + <dd>Gets a low-level file descriptor for an external file. The + external file list caches file descriptors because we might + have many more external files than there are file descriptors + available to this process. The caller should not close this file. + + <p><dt><code> + herr_t H5O_efl_release (H5O_efl_t *efl) + </code> + <dd>Releases an external file list, closes all files + associated with that list, and if the list has been modified + since the call to <code>H5O_efl_new</code> flushes the message + to disk. + </dl> + + <hr> + <address><a href="mailto:robb@arborea.spizella.com">Robb Matzke</a></address> +<!-- Created: Fri Oct 3 09:52:32 EST 1997 --> +<!-- hhmts start --> +Last modified: Tue Nov 25 12:36:50 EST 1997 +<!-- hhmts end --> + </body> +</html> |