HDF5 File Format Specification

Introduction

	Figure 1: Relationships among the HDF5 root group, other groups, and objects
	Figure 2: HDF5 objects -- datasets, datatypes, or dataspaces

The format of an HDF5 file on disk encompasses several key ideas of the HDF4 and AIO file formats as well as addressing some shortcomings therein. The new format is more self-describing than the HDF4 format and is more uniformly applied to data objects in the file.

An HDF5 file appears to the user as a directed graph. The nodes of this graph are the higher-level HDF5 objects that are exposed by the HDF5 APIs:

• Groups
• Datasets
• Datatypes
• Dataspaces

At the lowest level, as information is actually written to the disk, an HDF5 file is made up of the following objects:

• A super block
• B-tree nodes (containing either symbol nodes or raw data chunks)
• Object headers
• Collections
• Local heaps
• Free space

This Document

This document describes the lower-level data objects; the higher-level objects and their properties are described in the HDF5 User's Guide.

Three levels of information comprise the file format. Level 0 contains basic information for identifying and defining information about the file. Level 1 information contains the group information (stored as a B-tree) and is used as the index for all the objects in the file. Level 2 is the rest of the file and contains all of the data objects, with each object partitioned into header information, also known as meta information, and data.

The sizes of various fields in the following layout tables are determined by looking at the number of columns the field spans in the table. There are three exceptions: (1) The size may be overridden by specifying a size in parentheses, (2) the size of addresses is determined by the Size of Offsets field in the super block, and (3) the size of size fields is determined by the Size of Lengths field in the super block.

Disk Format: Level 0 - File Signature and Super Block

The super block may begin at certain predefined offsets within the HDF5 file, allowing a block of unspecified content for users to place additional information at the beginning (and end) of the HDF5 file without limiting the HDF5 library's ability to manage the objects within the file itself. This feature was designed to accommodate wrapping an HDF5 file in another file format or adding descriptive information to the file without requiring the modification of the actual file's information. The super block is located by searching for the HDF5 file signature at byte offset 0, byte offset 512 and at successive locations in the file, each a multiple of two of the previous location, i.e. 0, 512, 1024, 2048, etc.

The super block is composed of a file signature, followed by super block and group version numbers, information about the sizes of offset and length values used to describe items within the file, the size of each group page, and a group entry for the root object in the file.

Disk Format: Level 1 - File Infrastructure

Disk Format: Level 1A - B-link Trees and B-tree Nodes

B-link trees allow flexible storage for objects which tend to grow in ways that cause the object to be stored discontiguously. B-trees are described in various algorithms books including "Introduction to Algorithms" by Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest. The B-link tree, in which the sibling nodes at a particular level in the tree are stored in a doubly-linked list, is described in the "Efficient Locking for Concurrent Operations on B-trees" paper by Phillip Lehman and S. Bing Yao as published in the ACM Transactions on Database Systems, Vol. 6, No. 4, December 1981.

The B-link trees implemented by the file format contain one more key than the number of children. In other words, each child pointer out of a B-tree node has a left key and a right key. The pointers out of internal nodes point to sub-trees while the pointers out of leaf nodes point to symbol nodes and raw data chunks. Aside from that difference, internal nodes and leaf nodes are identical.

Each B-tree node looks like this:

The following question must next be answered: "Is the value described by key[i] contained in child[i-1] or in child[i]?" The answer depends on the type of tree. In trees for groups (node type 0) the object described by key[i] is the greatest object contained in child[i-1] while in chunk trees (node type 1) the chunk described by key[i] is the least chunk in child[i].

That means that key[0] for group trees is sometimes unused; it points to offset zero in the heap, which is always the empty string and compares as "less-than" any valid object name.

And key[N] for chunk trees is sometimes unused; it contains a chunk offset which compares as "greater-than" any other chunk offset and has a chunk byte size of zero to indicate that it is not actually allocated.

Disk Format: Level 1B - Group and Symbol Nodes

A group is an object internal to the file that allows arbitrary nesting of objects (including other groups). A group maps a set of names to a set of file address relative to the base address. Certain meta data for an object to which the group points can be duplicated in the group symbol table in addition to the object header.

An HDF5 object name space can be stored hierarchically by partitioning the name into components and storing each component in a group. The group entry for a non-ultimate component points to the group containing the next component. The group entry for the last component points to the object being named.

A group is a collection of group nodes pointed to by a B-link tree. Each group node contains entries for one or more symbols. If an attempt is made to add a symbol to an already full group node containing 2K entries, then the node is split and one node contains K symbols and the other contains K+1 symbols.

Disk Format: Level 1C - Group Entry

Each group entry in a group node is designed to allow for very fast browsing of stored objects. Toward that design goal, the group entries include space for caching certain constant meta data from the object header.

Format of the Scratch-pad Space

The group entry scratch-pad space is formatted according to the value in the Cache Type field.

If the Cache Type field contains the value zero (0) then no information is stored in the scratch-pad space.

If the Cache Type field contains the value one (1), then the scratch-pad space contains cached meta data for another object header in the following format:

If the Cache Type field contains the value two (2), then the scratch-pad space contains cached meta data for another symbolic link in the following format:

Disk Format: Level 1D - Local Heaps

A heap is a collection of small heap objects. Objects can be inserted and removed from the heap at any time. The address of a heap does not change once the heap is created. References to objects are stored in the group table; the names of those objects are stored in the local heap.

Objects within the heap should be aligned on an 8-byte boundary.

Disk Format: Level 1E - Global Heap

Each HDF5 file has a global heap which stores various types of information which is typically shared between datasets. The global heap was designed to satisfy these goals:

1. Repeated access to a heap object must be efficient without resulting in repeated file I/O requests. Since global heap objects will typically be shared among several datasets, it is probable that the object will be accessed repeatedly.
   
   
2. Collections of related global heap objects should result in fewer and larger I/O requests. For instance, a dataset of void pointers will have a global heap object for each pointer. Reading the entire set of void pointer objects should result in a few large I/O requests instead of one small I/O request for each object.
   
   
3. It should be possible to remove objects from the global heap and the resulting file hole should be eligible to be reclaimed for other uses.
   
   

The implementation of the heap makes use of the memory management already available at the file level and combines that with a new top-level object called a collection to achieve Goal B. The global heap is the set of all collections. Each global heap object belongs to exactly one collection and each collection contains one or more global heap objects. For the purposes of disk I/O and caching, a collection is treated as an atomic object.

Disk Format: Level 1F - Free-space Heap

The Free-space Index is a collection of blocks of data, dispersed throughout the file, which are currently not used by any file objects.

The super block contains a pointer to root of the free-space description; that pointer is currently (i.e., in HDF5 Release 1.2.x) required to be the undefined address 0xfff...ff.

The free-sapce index is not otherwise publicly defined at this time.

Disk Format: Level 2 - Data Objects

Data objects contain the real information in the file. These objects compose the scientific data and other information which are generally thought of as "data" by the end-user. All the other information in the file is provided as a framework for these data objects.

A data object is composed of header information and data information. The header information contains the information needed to interpret the data information for the data object as well as additional "meta-data" or pointers to additional "meta-data" used to describe or annotate each data object.

Disk Format: Level 2a - Data Object Headers

The header information of an object is designed to encompass all the information about an object which would be desired to be known, except for the data itself. This information includes the dimensionality, number-type, information about how the data is stored on disk (in external files, compressed, broken up in blocks, etc.), as well as other information used by the library to speed up access to the data objects or maintain a file's integrity. The header of each object is not necessarily located immediately prior to the object's data in the file and in fact may be located in any position in the file.

The header message types and the message data associated with them compose the critical "meta-data" about each object. Some header messages are required for each object while others are optional. Some optional header messages may also be repeated several times in the header itself, the requirements and number of times allowed in the header will be noted in each header message description below.

The following is a list of currently defined header messages:

Name: NIL

Name: Simple Dataspace

The Simple Dataspace message describes the number of dimensions and size of each dimension that the data object has. This message is only used for datasets which have a simple, rectilinear grid layout; datasets requiring a more complex layout (irregularly structured or unstructured grids, etc.) must use the Complex Dataspace message for expressing the space the dataset inhabits. (Note: The Complex Dataspace functionality is not yet implemented (as of HDF5 Release 1.2.x). It is not described in this document.)

Name: Datatype

The datatype message defines the datatype for each data point of a dataset. A datatype can describe an atomic type like a fixed- or floating-point type or a compound type like a C struct. A datatype does not, however, describe how data points are combined to produce a dataset. Datatypes are stored on disk as a datatype message, which is a list of datatype classes and their associated properties.

The Class Bit Field and Properties fields vary depending on the Type Class, which is the low-order four bits of the Type Class and Version field (the high-order four byte are the version which should be set to the value one). The type class is one of 0 (fixed-point number), 1 (floating-point number), 2 (date and time), 3 (text string), 4 (bit field), 5 (opaque), 6 (compound), 7 (reference), 8 (enumeration), or 9 (variable-length). The Class Bit Field is zero and the size of the Properties field is zero except for the cases noted here.

The Properties field of a compound datatype is a list of the member definitions of the compound datatype. The member definitions appear one after another with no intervening bytes. The member types are described with a recursive datatype message.

Name: Data Storage - Fill Value

The fill value message stores a single data point value which is returned to the application when an uninitialized data point is read from the dataset. The fill value is interpretted with the same datatype as the dataset. If no fill value message is present then a fill value of all zero is assumed.

Name: Reserved - Not Assigned Yet

Name: Data Storage - Compact

This message indicates that the data for the data object is stored within the current HDF file by including the actual data as the header data for this message. The data is stored internally in the normal format, i.e. in one chunk, uncompressed, etc.

Note that one and only one of the Data Storage headers can be stored for each data object.

Format of Data: The message data is actually composed of dataset data, so the format will be determined by the dataset format.

Name: Data Storage - External Data Files

Purpose and Description: The external object message indicates that the data for an object is stored outside the HDF5 file. The filename of the object is stored as a Universal Resource Location (URL) of the actual filename containing the data. An external file list record also contains the byte offset of the start of the data within the file and the amount of space reserved in the file for that data.

Name: Data Storage - Layout

Purpose and Description: Data layout describes how the elements of a multi-dimensional array are arranged in the linear address space of the file. Two types of data layout are supported:

1. The array can be stored in one contiguous area of the file. The layout requires that the size of the array be constant and does not permit chunking, compression, checksums, encryption, etc. The message stores the total size of the array and the offset of an element from the beginning of the storage area is computed as in C.
2. The array domain can be regularly decomposed into chunks and each chunk is allocated separately. This layout supports arbitrary element traversals, compression, encryption, and checksums, and the chunks can be distributed across external raw data files (these features are described in other messages). The message stores the size of a chunk instead of the size of the entire array; the size of the entire array can be calculated by traversing the B-tree that stores the chunk addresses.

Name: Reserved - Not Assigned Yet

Name: Reserved - Not Assigned Yet

Name: Data Storage - Filter Pipeline

Purpose and Description: This message describes the filter pipeline which should be applied to the data stream by providing filter identification numbers, flags, a name, an client data.

Name: Attribute

Purpose and Description: The Attribute message is used to list objects in the HDF file which are used as attributes, or "meta-data" about the current object. An attribute is a small dataset; it has a name, a datatype, a data space, and raw data. Since attributes are stored in the object header they must be relatively small (<64kb) and can be associated with any type of object which has an object header (groups, datasets, named types and spaces, etc.).

Name: Object Name

Type: 0x000D
Length: varies
Status: Optional, may not be repeated.

Purpose and Description: The object name or comment is designed to be a short description of an object. An object name is a sequence of non-zero (\0) ASCII characters with no other formatting included by the library.

Name: Object Modification Date & Time

Type: 0x000E
Length: fixed
Status: Optional, may not be repeated.

Purpose and Description: The object modification date and time is a timestamp which indicates (using ISO-8601 date and time format) the last modification of an object. The time is updated when any object header message changes according to the system clock where the change was posted.

Name: Shared Object Message

A constant message can be shared among several object headers by writing that message in the global heap and having the object headers all point to it. The pointing is accomplished with a Shared Object message which is understood directly by the object header layer of the library. It is also possible to have a message of one object header point to a message in some other object header, but care must be exercised to prevent cycles.

If a message is shared, then the message appears in the global heap and its message ID appears in the Header Message Type field of the object header. Also, the Flags field in the object header for that message will have bit two set (the H5O_FLAG_SHARED bit). The message body in the object header will be that of a Shared Object message defined here and not that of the pointed-to message.

Name: Object Header Continuation

The object header continuation is formatted as follows (assuming a 4-byte length & offset are being used in the current file):

The elements of the Header Continuation Message are described below:

Header Continuation Offset: (<offset> bytes)

This value is the offset in bytes from the beginning of the file where the header continuation information is located.

Header Continuation Length: (<length> bytes)

This value is the length in bytes of the header continuation information in the file.

Name: Group Message

The group message is formatted as follows:

The elements of the Group Message are described below:

B-tree Address (<offset> bytes)

This value is the offset in bytes from the beginning of the file where the B-tree is located.

Heap Address (<offset> bytes)

This value is the offset in bytes from the beginning of the file where the group name heap is located.

Disk Format: Level 2b - Shared Data Object Headers

In order to share header messages between several dataset objects, object header messages may be placed into the global heap. Since these messages require additional information beyond the basic object header message information, the format of the shared message is detailed below.

The elements of the shared object header message are described below:

Reference Count of Shared Header Message: (32-bit unsigned integer)

This value is used to keep a count of the number of dataset objects which refer to this message from their dataset headers. When this count reaches zero, the shared message header may be removed from the global heap.

Shared Object Header Message: (various lengths)

The data stored for the shared object header message is formatted in the same way as the private object header messages described in the object header description earlier in this document and begins with the header message Type.

Disk Format: Level 2c - Data Object Data Storage

The data for an object is stored separately from the header information in the file and may not actually be located in the HDF5 file itself if the header indicates that the data is stored externally. The information for each record in the object is stored according to the dimensionality of the object (indicated in the dimensionality header message). Multi-dimensional data is stored in C order [same as current scheme], i.e. the "last" dimension changes fastest.

Data whose elements are composed of simple number-types are stored in native-endian IEEE format, unless they are specifically defined as being stored in a different machine format with the architecture-type information from the number-type header message. This means that each architecture will need to [potentially] byte-swap data values into the internal representation for that particular machine.

Data with a "variable" sized number-type is stored in a data heap internal to the HDF5 file. Global heap identifiers are stored in the data object storage.

Data whose elements are composed of pointer number-types are stored in several different ways depending on the particular pointer type involved. Simple pointers are just stored as the dataset offset of the object being pointed to with the size of the pointer being the same number of bytes as offsets in the file. Partial-object pointers are stored as a heap-ID which points to the following information within the file-heap: an offset of the object pointed to, number-type information (same format as header message), dimensionality information (same format as header message), sub-set start and end information (i.e. a coordinate location for each), and field start and end names (i.e. a [pointer to the] string indicating the first field included and a [pointer to the] string name for the last field).

Data of a compound datatype is stored as a contiguous stream of the items in the structure, with each item formatted according to its datatype.