Chunking and Extendible Datasets

Creating an Extendible Dataset

HDF5 requires you to use chunking to define extendible datasets. This makes it possible to extend datasets efficiently without having to excessively reorganize storage.

The following operations are required in order to write an extendible dataset:

1. Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended.
2. Set dataset creation properties to enable chunking.
3. Create the dataset.
4. Extend the size of the dataset.

Programming Example

Description

Remarks

• The routine H5Pcreate / h5pcreate_f creates a new property as an instance of a property list. The signature is as follows:

  C:

      hid_t H5Pcreate (H5P_class_t classtype)
  

  FORTRAN:

      h5pcreate_f (classtype, prp_id, hdferr) 
  
              classtype  IN: INTEGER 
              prp_id    OUT: INTEGER(HID_T)
              hdferr    OUT: INTEGER 
  

  • The parameter classtype is the type of property list to create. Valid class types are as follows:

    C	FORTRAN
    H5P_FILE_CREATE H5P_FILE_ACCESS H5P_DATASET_CREATE H5P_DATASET_XFER H5P_MOUNT	H5P_FILE_CREATE_F H5P_FILE_ACCESS_F H5P_DATASET_CREATE_F H5P_DATASET_XFER_F H5P_MOUNT_F

  • In C, the property list identifier is returned if successful; otherwise a negative value is returned, if not. In FORTRAN, the property list identifier is returned in prp_id and the return value for the call is returned in hdferr.

• The routine H5Pset_chunk / h5pset_chunk_f sets the size of the chunks used to store a chunked layout dataset. The signature of this routine is as follows:

  C:

      herr_t H5Pset_chunk (hid_t prp_id, int ndims, 
                           const hsize_t * dims) 
  

  FORTRAN:

      h5pset_chunk_f (prp_id, ndims, dims, hdferr) 
  
              prp_id    IN: INTEGER(HID_T)
              ndims     IN: INTEGER
              dims      IN: INTEGER(HSIZE_T), DIMENSION(ndims) 
              hdferr   OUT: INTEGER
  
  

  • The prp_id parameter is the identifier for the property list to query.
  • The ndims parameter is the number of dimensions of each chunk.
  • The dims parameter is an array containing the size of each chunk.
  • In C, a non-negative value is returned if successful; otherwise a negative value is returned. In FORTRAN, the return value is returned in hdferr: 0 if successful and -1 otherwise.

• The H5Dextend / h5dextend_f routine extends a dataset that has an unlimited dimension. The signature is as follows:

  C:

      herr_t H5Dextend (hid_t dset_id, const hsize_t * size) 
  

  FORTRAN:

      h5dextend_f (dset_id, size, hdferr) 
  
              dset_id   IN: INTEGER(HID_T) 
              size         IN: INTEGER(HSIZE_T), DIMENSION(*)  
              hdferr      OUT: INTEGER
   
  

  • The dset_id parameter is the dataset identifier.
  • The size parameter, is an array containing the new magnitude of each dimension.
  • In C, this function returns a non-negative value if successful and a negative value otherwise. In FORTRAN, the return value is returned in hdferr: 0 if successful and -1 otherwise.

• The H5Dget_create_plist / h5dget_create_plist_f routine returns an identifier for a copy of the dataset creation property list for a dataset.

• The C function, H5Pget_layout, returns the layout of the raw data for a dataset. Valid types are H5D_CONTIGUOUS and H5D_CHUNKED. A FORTRAN routine for H5Pget_layout does not yet exist.

• The H5Pget_chunk / h5pget_chunk_f routine retrieves the size of chunks for the raw data of a chunked layout dataset. The signature is as follows:

  C:

      int H5Pget_chunk (hid_t prp_id, int ndims, hsize_t * dims) 
  

  FORTRAN:

      h5pget_chunk_f (prp_id, ndims, dims, hdferr)
  
              prp_id    IN: INTEGER(HID_T) 
              ndims     IN: INTEGER
              dims     OUT: INTEGER(HSIZE_T), DIMENSION(ndims) 
              hdferr   OUT: INTEGER 
  

  • The prp_id parameter is the identifier of the property list to query.
  • The ndims parameter is the size of the dims array.
  • The dims parameter is the array in which to store the chunk dimensions.
  • In C, this function returns the chunk dimensionality if successful and a negative value otherwise. In FORTRAN, the return value is returned in hdferr: the chunked rank if successful and -1 otherwise.

• The H5Pclose / h5pclose_f routine terminates access to a property list. The signature is as follows:

  C:

      herr_t H5Pclose (hid_t prp_id) 
  

  FORTRAN:

      h5pclose_f (prp_id, hdferr) 
  
              prp_id    IN: INTEGER(HID_T) 
              hdferr   OUT: INTEGER 
  

  • The prp_id parameter is the identifier of the property list to terminate access to.