HDF5 File Format Discussion

Quincey Koziol
koziol@ncsa.uiuc.edu
May 15, 2003

  1. Document's Audience:

  2. Background Reading:

    HDF5 File Format Specification
    This describes the current HDF5 file format.
  3. Introduction:

    What is this document about?
    This document attempts to explain the HDF5 file format specification with a few examples and describes some potential improvements to the format specification.

  4. File Format Examples:

    This section has several small programs and describes the format of a file created with each of them.

    Example program one - Create an empty file:

     
    #include "hdf5.h"
    #include 
    
    int main()
    {
        hid_t fid;      /* File ID */
        herr_t ret;     /* Generic return value */
    
        /* Create the file */
        fid=H5Fcreate("example1.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
        assert(fid>=0);
    
        /* Close the file */
        ret=H5Fclose(fid);
        assert(ret>=0);
    
        return(0);
    }
     
    Super Block
    byte byte byte byte
    \211 'H' 'D' 'F'
    \r \n \032 \n
    0 0 0 0
    0 8 8 0
    4 16
    0x00000003
    0

    0xffffffffffffffff

    ?

    0xffffffffffffffff

    0

    928

    H5G_CACHED_STAB (1)
    0

    384


    96


     
    %h5debug example1.h5
    
    Reading signature at address 0 (rel)
    File Super Block...
    File name:                                         example1.h5
    File access flags                                  0x00000000
    File open reference count:                         1
    Address of super block:                            0 (abs)
    Size of user block:                                0 bytes
    Super block version number:                        0
    Free list version number:                          0
    Root group symbol table entry version number:      0
    Shared header version number:                      0
    Size of file offsets (haddr_t type):               8 bytes
    Size of file lengths (hsize_t type):               8 bytes
    Symbol table leaf node 1/2 rank:                   4
    Symbol table internal node 1/2 rank:               16
    File consistency flags:                            0x00000003
    Base address:                                      0 (abs)
    Free list address:                                 UNDEF (rel)
    Address of driver information block:               UNDEF (rel)
    Root group symbol table entry:
       Name offset into private heap:                  0
       Object header address:                          928
       Dirty:                                          Yes
       Cache info type:                                Symbol Table
       Cached information:
          B-tree address:                              384
          Heap address:                                96
     
    Root Group Object Header
    byte byte byte byte
    1 0 2
    1
    32
    0x0011 16
    0x01 0

    384


    96

    0 0
    0x00 0

     
    %h5debug example1.h5 928
    
    New address: 928
    Reading signature at address 928 (rel)
    Object Header...
    Dirty:                                             0
    Version:                                           1
    Header size (in bytes):                            16
    Number of links:                                   1
    Number of messages (allocated):                    2 (32)
    Number of chunks (allocated):                      1 (8)
    Chunk 0...
       Dirty:                                          0
       Address:                                        944
       Size in bytes:                                  32
    Message 0...
       Message ID (sequence number):                   0x0011 stab(0)
       Shared message:                                 No
       Constant:                                       Yes
       Raw size in obj header:                         16 bytes
       Chunk number:                                   0
       Message Information:
          B-tree address:                              384
          Name heap address:                           96
    Message 1...
       Message ID (sequence number):                   0x0000 null(0)
       Shared message:                                 No
       Constant:                                       No
       Raw size in obj header:                         0 bytes
       Chunk number:                                   0
       Message Information:
          
     
    Root Group Local Heap
    byte byte byte byte
    'H' 'E' 'A' 'P'
    0
    256
    8
    128

     
    %h5debug example1.h5 96
    
    New address: 96
    Reading signature at address 96 (rel)
    Local Heap...
    Dirty:                                             0
    Header size (in bytes):                            32
    Address of heap data:                              128
    Data bytes allocated on disk:                      256
    Data bytes allocated in core:                      256
    Free Blocks (offset, size):
       Block #0:                                        8,      248
    Percent of heap used:                              3.12%
    Data follows (`__' indicates free region)...
         0: 00 00 00 00 00 00 00 00  __ __ __ __ __ __ __ __ ........
        16: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        32: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        48: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        64: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        80: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        96: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       112: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       128: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       144: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       160: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       176: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       192: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       208: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       224: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
       240: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
    
     
    Root Group B-tree
    byte byte byte byte
    'T' 'R' 'E' 'E'
    0 0 0

    0xffffffffffffffff


    0xffffffffffffffff


     
    %h5debug example1.h5 384 96
    
    New address: 384
    Reading signature at address 384 (rel)
    Tree type ID:                                      H5B_SNODE_ID
    Size of node:                                      544
    Size of raw (disk) key:                            8
    Dirty flag:                                        False
    Number of initial dirty children:                  0
    Level:                                             0
    Address of left sibling:                           UNDEF
    Address of right sibling:                          UNDEF
    Number of children (max):                          0 (32)
    
     

    Example program two - Create a file with a single dataset in it:

     
    #include "hdf5.h"
    #include 
    
    int main()
    {
        hid_t fid;      /* File ID */
        hid_t sid;      /* Dataspace ID */
        hid_t did;      /* Dataset ID */
        herr_t ret;     /* Generic return value */
    
        /* Create the file */
        fid=H5Fcreate("example2.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
        assert(fid>=0);
    
        /* Create a scalar dataspace for the dataset */
        sid=H5Screate(H5S_SCALAR);
        assert(sid>=0);
    
        /* Create a trivial dataset */
        did=H5Dcreate(fid, "Dataset", H5T_NATIVE_INT, sid, H5P_DEFAULT);
        assert(did>=0);
    
        /* Close the dataset */
        ret=H5Dclose(did);
        assert(ret>=0);
    
        /* Close the dataspace */
        ret=H5Sclose(sid);
        assert(ret>=0);
    
        /* Close the file */
        ret=H5Fclose(fid);
        assert(ret>=0);
    
        return(0);
    }
     
    Super Block
    byte byte byte byte
    \211 'H' 'D' 'F'
    \r \n \032 \n
    0 0 0 0
    0 8 8 0
    4 16
    0x00000003
    0

    0xffffffffffffffff

    ?

    0xffffffffffffffff

    0

    928

    H5G_CACHED_STAB (1)
    0

    384


    96


     
    %h5debug example2.h5
    
    Reading signature at address 0 (rel)
    File Super Block...
    File name:                                         example2.h5
    File access flags                                  0x00000000
    File open reference count:                         1
    Address of super block:                            0 (abs)
    Size of user block:                                0 bytes
    Super block version number:                        0
    Free list version number:                          0
    Root group symbol table entry version number:      0
    Shared header version number:                      0
    Size of file offsets (haddr_t type):               8 bytes
    Size of file lengths (hsize_t type):               8 bytes
    Symbol table leaf node 1/2 rank:                   4
    Symbol table internal node 1/2 rank:               16
    File consistency flags:                            0x00000003
    Base address:                                      0 (abs)
    Free list address:                                 UNDEF (rel)
    Address of driver information block:               UNDEF (rel)
    Root group symbol table entry:
       Name offset into private heap:                  0
       Object header address:                          928
       Dirty:                                          Yes
       Cache info type:                                Symbol Table
       Cached entry information:
          B-tree address:                              384
          Heap address:                                96
     
    Root Group Object Header
    byte byte byte byte
    1 0 2
    1
    32
    0x0011 16
    0x01 0

    384


    96

    0 0
    0x00 0

     
    %h5debug example2.h5 928
    
    New address: 928
    Reading signature at address 928 (rel)
    Object Header...
    Dirty:                                             0
    Version:                                           1
    Header size (in bytes):                            16
    Number of links:                                   1
    Number of messages (allocated):                    2 (32)
    Number of chunks (allocated):                      1 (8)
    Chunk 0...
       Dirty:                                          0
       Address:                                        944
       Size in bytes:                                  32
    Message 0...
       Message ID:                                     0x0011 stab(0)
       Shared message:                                 No
       Constant:                                       Yes
       Raw size in obj header:                         16 bytes
       Chunk number:                                   0
       Message Information:
          B-tree address:                              384
          Name heap address:                           96
    Message 1...
       Message ID:                                     0x0000 null(0)
       Shared message:                                 No
       Constant:                                       No
       Raw size in obj header:                         0 bytes
       Chunk number:                                   0
       Message Information:
          
     
    Root Group Local Heap
    byte byte byte byte
    'H' 'E' 'A' 'P'
    0
    256
    16
    128

     
    %h5debug example2.h5 96
    
    New address: 96
    Reading signature at address 96 (rel)
    Local Heap...
    Dirty:                                             0
    Header size (in bytes):                            32
    Address of heap data:                              128
    Data bytes allocated on disk:                      256
    Data bytes allocated in core:                      256
    Free Blocks (offset, size):
       Block #0:                                       16,      240
    Percent of heap used:                              6.25%
    Data follows (`__' indicates free region)...
          0: 00 00 00 00 00 00 00 00  44 61 74 61 73 65 74 00 ........Dataset.
         16: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         32: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         48: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         64: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         80: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
         96: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        112: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        128: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        144: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        160: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        176: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        192: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        208: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        224: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
        240: __ __ __ __ __ __ __ __  __ __ __ __ __ __ __ __
     
    Root Group B-tree
    byte byte byte byte
    'T' 'R' 'E' 'E'
    0 0 1

    0xffffffffffffffff


    0xffffffffffffffff


    0


    1248


    8


     
    %h5debug example2.h5 384 96
    
    New address: 384
    Reading signature at address 384 (rel)
    Tree type ID:                                      H5B_SNODE_ID
    Size of node:                                      544
    Size of raw (disk) key:                            8
    Dirty flag:                                        False
    Number of initial dirty children:                  0
    Level:                                             0
    Address of left sibling:                           UNDEF
    Address of right sibling:                          UNDEF
    Number of children (max):                          1 (32)
    Child 0...
       Address:                                        1248
       Left Key:
          Heap offset:                                 0
          Name :
       Right Key:
          Heap offset:                                 8
          Name :                                       Dataset
     
    Root Group B-tree Symbol Table Node
    byte byte byte byte
    'S' 'N' 'O' 'D'
    1 0 1
    8

    976

    0
    0


    0



     
    %h5debug example2.h5 1248 96
    
    New address: 1248
    Reading signature at address 1248 (rel)
    Symbol Table Node...
    Dirty:                                             No
    Size of Node (in bytes):                           328
    Number of Symbols:                                 1 of 8
    Symbol 0:
       Name:                                           `Dataset'
       Name offset into private heap:                  8
       Object header address:                          976
       Dirty:                                          No
       Cache info type:                                Nothing Cached
     
    '/Dataset' Object Header
    byte byte byte byte
    Version: 1 Reserved: 0 Number of Header Messages: 6
    Object Reference Count: 1
    Total Object Header Size: 256
    Fill Value Header Message
    Message Type: 0x0005 Message Data Size: 8
    Flags: 0x01 Reserved: 0
    Version: 1 Space Allocation Time: 2 (Late) Fill Value Writing Time: 0 (At allocation) Fill Value Defined: 0 (Undefined)
    Fill Value Datatype Size: 0 (Use dataset's datatype for fill-value datatype)
    Datatype Header Message
    Message Type: 0x0003 Message Data Size: 16
    Flags: 0x01 Reserved: 0
    Version: 0x1 Class: 0x0 (Fixed-Point)
    Fixed-Point Bit-Field: 0x08 (Little-endian, No padding, Signed)
    Size: 4
    Bit Offset: 0 Bit Precision: 32
    Message Alignment Filler: -
    Dataspace Header Message
    Message Type: 0x0001 Message Data Size: 8
    Flags: 0x00 Reserved: 0
    Version: 1 Rank: 0 (Scalar) Flags: 0x00 (No maximum dimensions, no permutation information) Reserved: 0
    Reserved: 0
    Layout Header Message
    Message Type: 0x0008 Message Data Size: 24
    Flags: 0x00 Reserved: 0
    Version: 1 Rank: 1 (Dataspace rank+1) Class: 1 (Contiguous) Reserved: 0
    Reserved: 0

    Address: 0xffffffffffffffff (Undefined)

    Dimension 0 Size: 4 (Datatype size)
    Message Alignment Filler: -
    Modification Date & Time Header Message
    Message Type: 0x0012 Message Data Size: 8
    Flags: 0x00 Reserved: 0
    Version: 1 Reserved: 0
    Seconds Since Epoch: 1052401700 (2003-05-08 08:48:20 CDT)
    Null Header Message
    Message Type: 0x0000 Message Data Size: 144
    Flags: 0x00 Reserved: 0

     
    %h5debug example2.h5 976
    
    New address: 976
    Reading signature at address 976 (rel)
    Object Header...
    Dirty:                                             0
    Version:                                           1
    Header size (in bytes):                            16
    Number of links:                                   1
    Number of messages (allocated):                    6 (32)
    Number of chunks (allocated):                      1 (8)
    Chunk 0...
       Dirty:                                          0
       Address:                                        992
       Size in bytes:                                  256
    Message 0...
       Message ID (sequence number):                   0x0005 `fill_new' (0)
       Shared:                                         No
       Constant:                                       Yes
       Raw size in obj header:                         8 bytes
       Chunk number:                                   0
       Message Information:
          Version:                                     1
          Space Allocation Time:                       Late
          Fill Time:                                   On Allocation
          Fill Value Defined:                          Undefined
          Size:                                        0
          Data type:                                   
    Message 1...
       Message ID (sequence number):                   0x0003 data_type(0)
       Shared message:                                 No
       Constant:                                       Yes
       Raw size in obj header:                         16 bytes
       Chunk number:                                   0
       Message Information:
          Type class:                                  integer
          Size:                                        4 bytes
          Byte order:                                  little endian
          Precision:                                   32 bits
          Offset:                                      0 bits
          Low pad type:                                zero
          High pad type:                               zero
          Sign scheme:                                 2's comp
    Message 2...
       Message ID (sequence number):                   0x0001 simple_dspace(0)
       Shared message:                                 No
       Constant:                                       No
       Raw size in obj header:                         8 bytes
       Chunk number:                                   0
       Message Information:
          Rank:                                        0
    Message 3...
       Message ID (sequence number):                   0x0008 layout(0)
       Shared message:                                 No
       Constant:                                       No
       Raw size in obj header:                         24 bytes
       Chunk number:                                   0
       Message Information:
          Data address:                                UNDEF
          Number of dimensions:                        1
          Size:                                        {4}
    Message 4...
       Message ID (sequence number):                   0x0012 mtime_new(0)
       Shared message:                                 No
       Constant:                                       No
       Raw size in obj header:                         8 bytes
       Chunk number:                                   0
       Message Information:
          Time:                                        2003-03-05 14:52:00 CST
    Message 5...
       Message ID (sequence number):                   0x0000 null(0)
       Shared message:                                 No
       Constant:                                       No
       Raw size in obj header:                         144 bytes
       Chunk number:                                   0
       Message Information: