Introduction to HDF5 1.0 Alpha2.0

This is a brief introduction to the HDF5 data model and programming model. Being a Getting Started or QuickStart document, this Introduction to HDF5 is intended to provide enough information for you to develop a basic understanding of how HDF5 works and is meant to be used. Knowledge of the current version of HDF, will make it easier to follow the text, but it is not required. More complete information, of the sort you will need to actually use HDF5, is available in the HDF5 documentation. . Available documents include the following:

• HDF5 User’s Guide. Where appropriate, this Introduction will refer to specific sections of the User’s Guide.
• HDF5 Reference Manual

• The directory hdf5/examples contains the examples used in this document.
• The directory hdf5/test contains the development tests used by the HDF5 developers. Since these codes are intended to fully exercise the system, they provide more diverse and sophisticated examples of what HDF5 can do.

What is the HDF5 prototype?

• A single file cannot store more than 20,000 complex objects, and a single file cannot be larger than 2 gigabytes
• The data models are less consistent than they should be, there are more object types than necessary, and datatypes are too restricted.
• The library source is old and overly complex, does not support parallel I/O effectively, and is difficult to use in threaded applications.

• A new file format designed to address some of the deficiencies of HDF4.x, particularly the need to store larger files and more objects per file.
• A simpler, more comprehensive data model that includes only two basic structures: a multidimensional array of record structures, and a grouping structure.
• A simpler, better-engineered library and API, with improved support for parallel i/o, threads, and other requirements imposed by modern systems and applications.

Limitations of the current prototype

• Data compression is supported, though only GZIP is implemented. GZIP, or GNU Zip, is a compression function from the GNU Project.
• Some functions for manipulating datasets, dataspaces, and groups have not been implemented.
• Some number types, including user-defined number types are not supported. Also number type conversion is limited.

HDF5 file organization and data model.

• HDF5 group: a grouping structure containing instances of zero or more groups or datasets, together with supporting metadata
• HD5F dataset: a multidimensional array of data elements, together with supporting metadata.

/ signifies the root group.

/foo signifies a member of the root group called foo.

/foo/zoo signifies a member of the group foo, which in turn is a member of the root group.

HDF5 Groups

• A group header, which contains a group name and a list of group attributes.
• A group symbol table, which is a list of the HDF5 objects that belong to the group.

HDF5 Datasets

Name. A dataset name is a sequence of alphanumeric ASCII characters.

Datatype. HDF5 allows one to define many different kinds of datatypes. There are two basic categories of data types: "atomic" types and "compound" types. Atomic types are those that are not decomposed at the data type interface level, such as integers and floats. Compound types are made up of atomic types.

• Integer datatypes: 8-bit, 16-bit, 32-bit, and 64-bit integers in both little and big-endian format.
• Floating-point numbers: IEEE 32-bit and 64-bit floating-point numbers in both little and big-endian format.

Properties of a dataspace consist of the rank (number of dimensions) of the data array, and the actual sizes of the dimensions of the array, and the maximum sizes of the dimensions of the array. For a fixed-dimension dataset, the actual size is the same as the maximum size of a dimension. When a dimension is unlimited, the maximum size is set to the value H5P_UNLIMITED. (An example below shows how to create extendible datasets.)

See Dataspaces in the HDF User’s Guide for further information.

Storage layout. The HDF5 format makes it possible to store data in a variety of ways. The default storage layout format is contiguous, meaning that data is stored in the same linear way that it is organized in memory. Two other storage layout formats are currently defined for HDF5: compact, and chunked. In the future, other storage layouts may be added.

See Datasets in the HDF User’s Guide for further information.

HDF5 attributes

See Attributes in the HDF User’s Guide for further information.

The HDF5 Applications Programming Interface (API)

Naming conventions

• H5F: File-level access routines.
  Example: H5Fopen, which opens an HDF5 file.
• H5G: Group functions, for creating and operating on physical groups of objects.
  Example: H5Gset,which sets the working group to the specified group.
• H5T: DataType functions, for creating and operating on simple and compound datatypes to be used as the elements in data arrays.
  Example: H5Tcopy,which creates a copy of an existing data type.
• H5S: DataSpace functions, which create and manipulate the dataspace in which the elements of a data array are stored.
  Example: H5Screate_simple, which creates simple dataspaces.
• H5D: Dataset functions, which manipulate the data within datasets and determine how the data is to be stored in the file.
  Example: H5Dread, which reads all or part of a dataset into a buffer in memory.
• H5P: Property list functions, for manipulating object creation and access properties.
  Example: H5Pset_chunk, which sets the number of dimensions and the size of a chunk.
• H5A: Attribute access and manipulating routines.
  Example: H5Aget_name, which retrieves name of an attribute.
• H5SZ: Compression registration routine.
  Example: H5Zregister, which registers new compression and uncompression functions for use with the HDF5 library.
• H5E: Error handling routines.
  Example: H5Eprint, which prints the current error stack.

Include files

There are a number definitions and declarations that should be included with any HDF5 program. These definitions and declarations are contained in several include files. The main include file is hdf5.h. This file includes all of the other files that your program is likely to need. Be sure to include hdf5.h in any program that accesses HDF5.

Predefined atomic datatypes

The datatype interface provides a mechanism to describe the storage format of individual data points of a data set and is designed to allow new features to be easily added without disrupting applications that use the datatype interface. A dataset (the H5D interface) is composed of a collection or raw data points of homogeneous type organized according to the dataspace (the H5S interface).

A datatype is a collection of data type properties, all of which can be stored on disk, and which when taken as a whole, provide complete information for data conversion to or from that data type. The interface provides functions to set and query properties of a data type.

A data point is an instance of a data type, which is an instance of a type class. We have defined a set of type classes and properties which can be extended at a later time. The atomic type classes are those which describe types which cannot be decomposed at the data type interface level; all other classes are compound.

To illustrate, let us consider a set of predefined atomic datatypes. The library predefines a modest number of data types having names like H5T_arch_base where arch is an architecture name and base is a programming type name. New types can be derived from the predifined types by copying the predefined type (see H5Tcopy()) and then modifying the result.

The NATIVE architecture, for example, contains C-like data types for the machine on which the library was compiled. The types were actually defined by running the H5detect program when the library was compiled. In order to be portable, applications should almost always use this architecture to describe things in memory.

The NATIVE architecture has base names which do not follow the same rules as the others. Instead, native type names are similar to the C type names. Here are some examples:

signed char

float

See Datatypes at in the HDF User’s Guide for further information.

Programming models

• create a file
• create and initialize a dataset
• discard objects when they are no longer needed
• write a dataset to a new file
• obtain information about a dataset
• read a portion of a dataset
• create and write compound datatypes
• create and write extendible datasets
• create and populate groups

How to create an HDF5 file

This programming model shows how to create a file and also how to close the file.

1. Create the file using H5Fcreate. Obtain a file identifier.
2. Close the file with H5Fclose.

The following code fragment implements the specified model. If there is a possibility that the file already exists, the user must add the flag H5ACC_TRUNC to the access mode to overwrite the previous file's information.

hid_t       file;                          /* handle */

/*

 * Create a new file using H5ACC_TRUNC access,

 * default file creation properties, and default file

 * access properties.

 * Then close the file.

 */

file = H5Fcreate(FILE, H5ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

status = H5Fclose(file); 

How to create and initialize the essential components of a dataset for writing to a file.

Recall that datatypes and dimensionality (dataspace) are independent objects, which are created separately from any dataset that they might be attached to. Because of this the creation of a dataset requires, at a minimum, separate definitions of datatype, dimensionality, and dataset. Hence, to create a dataset the following steps need to be taken:

1. Create and initialize a dataspace for the dataset to be written.
2. Define the datatype for the dataset to be written.
3. Create and initialize the dataset itself.

hid_t    dataset, datatype, dataspace;   /* declare handles */



/* 

 * 1. Create dataspace: Describe the size of the array and 

 * create the data space for fixed size dataset. 

 */

dimsf[0] = NX;

dimsf[1] = NY;

dataspace = H5Pcreate_simple(RANK, dimsf, NULL); 

/*

/* 

 * 2. Define datatype for the data in the file.

 * We will store little endian integer numbers.

 */

datatype = H5Tcopy(H5T_NATIVE_INT);

status = H5Tset_order(datatype, H5T_ORDER_LE);

/*

 * 3. Create a new dataset within the file using defined 

 * dataspace and datatype and default dataset creation

 * properties.

 * NOTE: H5T_NATIVE_INT can be used as datatype if conversion

 * to little endian is not needed.

 */

dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace, H5P_DEFAULT);

How to discard objects when they are no longer needed

How to write a dataset to a new file

Having defined the datatype, dataset, and dataspace parameters, you write out the data with a call to H5Dwrite.

/*

 * Write the data to the dataset using default transfer

 * properties.

 */

status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,

                  H5P_DEFAULT, data);

The third and fourth parameters of H5Dwrite in the example describe the dataspaces in memory and in the file, respectively. They are set to the value H5S_ALL to indicate that an entire dataset is to be written. In a later section we look at how we would access a portion of a dataset.

Example 1 contains a program that creates a file and a dataset, and writes the dataset to the file.

Reading is analogous to writing. If, in the previous example, we wish to read an entire dataset, we would use the same basic calls with the same parameters. Of course, the routine H5Dread would replace H5Dwrite.

Getting information about a dataset

/*

 * Get datatype and dataspace handles and then query

 * dataset class, order, size, rank and dimensions.

 */



datatype  = H5Dget_type(dataset);     /* datatype handle */ 

class     = H5Tget_class(datatype);

if (class == H5T_INTEGER) printf("Data set has INTEGER type \n");

order     = H5Tget_order(datatype);

if (order == H5T_ORDER_LE) printf("Little endian order \n");



size  = H5Tget_size(datatype);

printf(" Data size is %d \n", size);



dataspace = H5Dget_space(dataset);    /* dataspace handle */

rank      = H5Sextent_ndims(dataspace);

status_n  = H5Sextent_dims(dataspace, dims_out);

printf("rank %d, dimensions %d x %d \n", rank, dims_out[0], dims_out[1]);

Reading a portion of a dataset

Consider the following example. Suppose there is 500x600 dataset in a file, and we wish to read from the dataset a 100x200 hyperslab located beginning at element <200,200>. In addition, suppose we wish to read the hyperslab into an 200x400 array in memory beginning at element <0,0> in memory. Visually, the transfer looks something like this:

As the example illustrates, whenever we read part of a dataset from a file we must provide two dataspaces: the dataspace of the object in the file as well as the dataspace of the object in memory into which we read. There are dataspace routines (H5S... ) for doing this.

For example, suppose we want to read a 3x4 hyperslab from a dataset in a file beginning at the element <1,2> in the dataset. In order to do this, we must create a dataspace that describes the overall rank and dimensions of the dataset in the file, as well as the position and size of the hyperslab that we are extracting from that dataset. The following code illustrates how this would be done.

/* 

 * Get overall rank and dimensions of dataspace.

 */

dataspace = H5Dget_space(dataset);    /* get dataspace handle */

rank      = H5Pextent_ndims(dataspace);

status_n  = H5Pextent_dims(dataspace, dims_out);



/* 

 * Define hyperslab in the dataset. 

 */

offset[0] = 1;

offset[1] = 2;

count[0]  = 3;

count[1]  = 4;

status = H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, offset, NULL, count, NULL);

This describes the dataspace from which we wish to read. We need to define the dataspace in memory analogously. Suppose, for instance, that we have in memory a 3 dimensional 7x7x3 array into which we wish to read the 3x4 hyperslab described above beginning at the element <3,0,0>. Since the in-memory dataspace has three dimensions, we have to describe the hyperslab as an array with three dimensions, with the last dimension being 1: <3,4,1>.

/*

 * Define the memory dataspace.

 */

dimsm[0] = 7;

dimsm[1] = 7;

dimsm[2] = 3;

memspace = H5Screate_simple(RANK_OUT,dimsm,NULL);   



/* 

 * Define memory hyperslab. 

 */

offset_out[0] = 3;

offset_out[1] = 0;

offset_out[2] = 0;

count_out[0]  = 3;

count_out[1]  = 4;

count_out[2]  = 1;

status = H5Sselect_hyperslab(memspace, H5S_SELECT_SET, offset_out, NULL, count_out, NULL);



/*

Example 2 contains a complete program that performs these operations.

Creating compound datatypes

Properties of compound datatypes. A compound datatype is similar to a struct in C or a common block in Fortran. It is a collection of one or more atomic types or small arrays of such types. To create and use of a compound datatype requires you need to refer to various properties of the data compound datatype:

• It is of class compound.
• It has a fixed total size, in bytes.
• It consists of zero or more members (defined in any order) with unique names and which occupy non-overlapping regions within the datum.
• Each member has its own datatype.
• Each member is referenced by an index number between zero and N-1, where N is the number of members in the compound datatype.
• Each member has a name which is unique among its siblings in a compound data type.
• Each member has a fixed byte offset, which is the first byte (smallest byte address) of that member in a compound datatype.
• Each member can be a small array of up to four dimensions.

HOFFSET(s,m). This macro computes the offset of member m within a struct variable s.

Here is an example in which a compound data type is created to describe complex numbers whose type is defined by the complex_t struct.

typedef struct {

   double re;   /*real part */

   double im;   /*imaginary part */

} complex_t;



complex_t tmp;  /*used only to compute offsets */

hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);

H5Tinsert (complex_id, "real", HOFFSET(tmp,re),

           H5T_NATIVE_DOUBLE);

H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im),

           H5T_NATIVE_DOUBLE);

Example 3 shows how to create a compound data type, write an array that has the compound data type to the file, and read back subsets of the members.

Creating and writing extendible datasets

     1 1 1

     1 1 1 

     1 1 1 

     1 1 1 

     1 1 1 

     1 1 1 

     2 2 2

     2 2 2

     2 2 2

     2 2 2

     2 2 2

     2 2 2

     2 2 2

     1 1 1 3 3 

     1 1 1 3 3 

     1 1 1 3 3 

     2 2 2 3 3

     2 2 2 3 3

     2 2 2 3 3

     2 2 2 3 3

     2 2 2 3 3

     2 2 2 3 3

     2 2 2 3 3

Declaring unlimited dimensions. We could declare the dataspace to have unlimited dimensions with the following code, which uses the predefined constant H5S_UNLIMITED to specify unlimited dimensions.

hsize_t dims[2] = { 3, 3}; /* dataset dimensions

at the creation time */ 

hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};

/*

* 1. Create the data space with unlimited dimensions. 

*/

dataspace = H5Screate_simple(RANK, dims, maxdims); 

Enabling chunking. We can then modify the dataset storage layout properties to enable chunking. We do this using the routine H5Pset_chunk:

hid_t cparms; 

hsize_t chunk_dims[2] ={2, 5};

/* 

* 2. Modify dataset creation properties to enable chunking.

*/

cparms = H5Pcreate (H5P_DATASET_CREATE);

status = H5Pset_chunk( cparms, RANK, chunk_dims);

Extending dataset size. Finally, when we want to extend the size of the dataset, we invoke H5Dextend to extend the size of the dataset. In the following example, we extend the dataset along the first dimension, by seven rows, so that the new dimensions are <10,3>:

/*

* Extend the dataset. Dataset becomes 10 x 3.

*/

dims[0] = dims[0] + 7;

size[0] = dims[0]; 

size[1] = dims[1]; 

status = H5Dextend (dataset, size);

Example 4 shows how to create a 3x3 extendible dataset, write the dataset, extend the dataset to 10x3, write the dataset again, extend it again to 10x5, write the dataset again.

Example 5 shows how to read the data written by Example 4.

Working with groups in a file

Groups provide a mechanism for organizing datasets in an HDF5 file extendable meaningful ways. The H5G API contains routines for working with groups.

To create a group, use H5Gcreate. For example, the following code creates two groups that are members of the root group. They are called /IntData and /FloatData. The return value (dir) is the group identifier.

/*

* Create two groups in a file.

*/

dir = H5Gcreate(file, "/IntData", 0);

status = H5Gclose(dir);

dir = H5Gcreate(file,"/FloatData", 0);

status = H5Gclose(dir);

The third parameter in H5Gcreate optionally specifies how much file space to reserve to store the names that will appear in this group. If a non-positive value is supplied then a default size is chosen.

H5Gclose closes the group and releases the group identifier.

Creating an object in a particular group. Except for single-object HDF5 files, every object in an HDF5 file must belong to a group, and hence has a path name. Hence, we put an object in a particular group by giving its path name when we create it. For example, the following code creates a dataset IntArray in the group /IntData:

/*

 * Create dataset in the /IntData group by specifying full path.

 */

dims[0] = 2;

dims[1] = 3;

dataspace = H5Pcreate_simple(2, dims, NULL);

dataset = H5Dcreate(file, "/IntData/IntArray", H5T_NATIVE_INT, dataspace, H5C_DEFAULT); 

Changing the current group. The HDF5 Group API supports the idea of a current group. This is analogous to the current working directory idea in UNIX. You can set the current group in HDF5 with the routine H5Gset. The following code shows how to set a current group, then create a certain dataset (FloatData) in that group.

/*

 * Set current group to /FloatData.

 */

status = H5Gset (file, "/FloatData");



/* 

 * Create two datasets

 */



dims[0] = 5;

dims[1] = 10;

dataspace = H5Screate_simple(2, dims, NULL);

dataset = H5Dcreate(file, "FloatArray", H5T_NATIVE_FLOAT, dataspace, H5P_DEFAULT); 

Example 6 shows how to create an HDF5 file with two group, and to place some datasets within those groups.

Example code

Example 1: How to create a homogeneous multi-dimensional dataset and write it to a file.

This example creates a 2-dimensional HDF 5 dataset of little endian 32-bit integers.

/*  

 *  This example writes data to the HDF5 file.

 *  Data conversion is performed during write operation.  

 */

 

#include <hdf5.h>



#define FILE        "SDS.h5"

#define DATASETNAME "IntArray" 

#define NX     5                      /* dataset dimensions */

#define NY     6

#define RANK   2



main ()

{

   hid_t       file, dataset;         /* file and dataset handles */

   hid_t       datatype, dataspace;   /* handles */

   hsize_t     dimsf[2];              /* dataset dimensions */

   herr_t      status;                             

   int         data[NX][NY];          /* data to write */

   int         i, j;



/* 

 * Data  and output buffer initialization. 

 */



for (j = 0; j < NX; j++) {

    for (i = 0; i < NY; i++)

        data[j][i] = i + j;

}     

                                       /*  0 1 2 3 4 5 

                                           1 2 3 4 5 6

                                           2 3 4 5 6 7

                                           3 4 5 6 7 8

                                           4 5 6 7 8 9   */



/*

 * Create a new file using H5F_ACC_TRUNC access,

 * default file creation properties, and default file

 * access properties.

 */

file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);



/*

 * Describe the size of the array and create the data space for fixed

 * size dataset. 

 */

dimsf[0] = NX;

dimsf[1] = NY;

dataspace = H5Screate_simple(RANK, dimsf, NULL); 



/* 

 * Define datatype for the data in the file.

 * We will store little endian INT numbers.

 */

datatype = H5Tcopy(H5T_NATIVE_INT);

status = H5Tset_order(datatype, H5T_ORDER_LE);

/*

 * Create a new dataset within the file using defined dataspace and

 * datatype and default dataset creation properties.

 */

dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace,

                    H5P_DEFAULT);



/*

 * Write the data to the dataset using default transfer properties.

 */

status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,

                  H5P_DEFAULT, data);



/*

 * Close/release resources.

 */

H5Sclose(dataspace);

H5Tclose(datatype);

H5Dclose(dataset);

H5Fclose(file);

 

}     

Example 2. How to read a hyperslab from file into memory.

This example reads a hyperslab from a 2-d HDF5 dataset into a 3-d dataset in memory.

/*  

 *   This example reads hyperslab from the SDS.h5 file 

 *   created by h5_write.c program into two-dimensional

 *   plane of the tree-dimensional array. 

 *   Information about dataset in the SDS.h5 file is obtained. 

 */

 

#include "hdf5.h"



#define FILE        "SDS.h5"

#define DATASETNAME "IntArray" 

#define NX_SUB  3           /* hyperslab dimensions */ 

#define NY_SUB  4 

#define NX 7           /* output buffer dimensions */ 

#define NY 7 

#define NZ  3 

#define RANK         2

#define RANK_OUT     3



main ()

{

   hid_t       file, dataset;         /* handles */

   hid_t       datatype, dataspace;   

   hid_t       memspace; 

   H5T_class_t class;                 /* data type class */

   H5T_order_t order;                 /* data order */

   size_t      size;                  /* size of the data element

                                         stored in file */ 

   hsize_t     dimsm[3];              /* memory space dimensions */

   hsize_t     dims_out[2];           /* dataset dimensions */      

   herr_t      status;                             



   int         data_out[NX][NY][NZ ]; /* output buffer */

   

   hsize_t      count[2];              /* size of the hyperslab in the file */

   hsize_t      offset[2];             /* hyperslab offset in the file */

   hsize_t      count_out[3];          /* size of the hyperslab in memory */

   hsize_t      offset_out[3];         /* hyperslab offset in memory */

   int          i, j, k, status_n, rank;



for (j = 0; j < NX; j++) {

    for (i = 0; i < NY; i++) {

        for (k = 0; k < NZ ; k++)

            data_out[j][i][k] = 0;

    }

} 

 

/*

 * Open the file and the dataset.

 */

file = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);

dataset = H5Dopen(file, DATASETNAME);



/*

 * Get datatype and dataspace handles and then query

 * dataset class, order, size, rank and dimensions.

 */



datatype  = H5Dget_type(dataset);     /* datatype handle */ 

class     = H5Tget_class(datatype);

if (class == H5T_INTEGER) printf("Data set has INTEGER type \n");

order     = H5Tget_order(datatype);

if (order == H5T_ORDER_LE) printf("Little endian order \n");



size  = H5Tget_size(datatype);

printf(" Data size is %d \n", size);



dataspace = H5Dget_space(dataset);    /* dataspace handle */

rank      = H5Sextent_ndims(dataspace);

status_n  = H5Sextent_dims(dataspace, dims_out, NULL);

printf("rank %d, dimensions %d x %d \n", rank, dims_out[0], dims_out[1]);



/* 

 * Define hyperslab in the datatset. 

 */

offset[0] = 1;

offset[1] = 2;

count[0]  = NX_SUB;

count[1]  = NY_SUB;

status = H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, offset, NULL, 

                             count, NULL);



/*

 * Define the memory dataspace.

 */

dimsm[0] = NX;

dimsm[1] = NY;

dimsm[2] = NZ ;

memspace = H5Screate_simple(RANK_OUT,dimsm,NULL);   



/* 

 * Define memory hyperslab. 

 */

offset_out[0] = 3;

offset_out[1] = 0;

offset_out[2] = 0;

count_out[0]  = NX_SUB;

count_out[1]  = NY_SUB;

count_out[2]  = 1;

status = H5Sselect_hyperslab(memspace, H5S_SELECT_SET, offset_out, NULL, 

                             count_out, NULL);



/*

 * Read data from hyperslab in the file into the hyperslab in 

 * memory and display.

 */

status = H5Dread(dataset, H5T_NATIVE_INT, memspace, dataspace,

                 H5P_DEFAULT, data_out);

for (j = 0; j < NX; j++) {

    for (i = 0; i < NY; i++) printf("%d ", data_out[j][i][0]);

    printf("\n");

}

                                         /*  0 0 0 0 0 0 0

                                             0 0 0 0 0 0 0

                                             0 0 0 0 0 0 0

                                             3 4 5 6 0 0 0  

                                             4 5 6 7 0 0 0

                                             5 6 7 8 0 0 0

                                             0 0 0 0 0 0 0 */



/*

 * Close/release resources.

 */

H5Tclose(datatype);

H5Dclose(dataset);

H5Sclose(dataspace);

H5Sclose(memspace);

H5Fclose(file);



}     

Example 3. Working with compound datatypes.

This example shows how to create a compound data type, write an array which has the compound data type to the file, and read back subsets of fields.

/*

 * This example shows how to create a compound data type,

 * write an array which has the compound data type to the file,

 * and read back fields' subsets.

 */



#include "hdf5.h"



#define FILE          "SDScompound.h5"

#define DATASETNAME   "ArrayOfStructures"

#define LENGTH        10

#define RANK          1



main()



{



   

/* First structure  and dataset*/

typedef struct s1_t {

    int    a;

    float  b;

    double c; 

} s1_t;

s1_t       s1[LENGTH];

hid_t      s1_tid;     /* File datatype hadle */



/* Second structure (subset of s1_t)  and dataset*/

typedef struct s2_t {

    double c;

    int    a;

} s2_t;

s2_t       s2[LENGTH];

hid_t      s2_tid;    /* Memory datatype handle */



/* Third "structure" ( will be used to read float field of s1) */

hid_t      s3_tid;   /* Memory datatype handle */

float      s3[LENGTH];



int        i;

hid_t      file, datatype, dataset, space; /* Handles */

herr_t     status;

hsize_t    dim[] = {LENGTH};   /* Dataspace dimensions */





/*

 * Initialize the data

 */

   for (i = 0; i< LENGTH; i++) {

        s1[i].a = i;

        s1[i].b = i*i;

        s1[i].c = 1./(i+1);

}



/*

 * Create the data space.

 */

space = H5Screate_simple(RANK, dim, NULL);



/*

 * Create the file.

 */

file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);



/*

 * Create the memory data type. 

 */

s1_tid = H5Tcreate (H5T_COMPOUND, sizeof(s1_t));

H5Tinsert(s1_tid, "a_name", HOFFSET(s1_t, a), H5T_NATIVE_INT);

H5Tinsert(s1_tid, "c_name", HOFFSET(s1_t, c), H5T_NATIVE_DOUBLE);

H5Tinsert(s1_tid, "b_name", HOFFSET(s1_t, b), H5T_NATIVE_FLOAT);



/* 

 * Create the dataset.

 */

dataset = H5Dcreate(file, DATASETNAME, s1_tid, space, H5P_DEFAULT);



/*

 * Wtite data to the dataset; 

 */

status = H5Dwrite(dataset, s1_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s1);



/*

 * Release resources

 */

H5Tclose(s1_tid);

H5Sclose(space);

H5Dclose(dataset);

H5Fclose(file);

 

/*

 * Open the file and the dataset.

 */

file = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);

 

dataset = H5Dopen(file, DATASETNAME);



/* 

 * Create a data type for s2

 */

s2_tid = H5Tcreate(H5T_COMPOUND, sizeof(s2_t));



H5Tinsert(s2_tid, "c_name", HOFFSET(s2_t, c), H5T_NATIVE_DOUBLE);

H5Tinsert(s2_tid, "a_name", HOFFSET(s2_t, a), H5T_NATIVE_INT);



/*

 * Read two fields c and a from s1 dataset. Fields in the file

 * are found by their names "c_name" and "a_name".

 */

status = H5Dread(dataset, s2_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s2);



/*

 * Display the fields

 */

printf("\n");

printf("Field c : \n");

for( i = 0; i < LENGTH; i++) printf("%.4f ", s2[i].c);

printf("\n");



printf("\n");

printf("Field a : \n");

for( i = 0; i < LENGTH; i++) printf("%d ", s2[i].a);

printf("\n");



/* 

 * Create a data type for s3.

 */

s3_tid = H5Tcreate(H5T_COMPOUND, sizeof(float));



status = H5Tinsert(s3_tid, "b_name", 0, H5T_NATIVE_FLOAT);



/*

 * Read field b from s1 dataset. Field in the file is found by its name.

 */

status = H5Dread(dataset, s3_tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, s3);



/*

 * Display the field

 */

printf("\n");

printf("Field b : \n");

for( i = 0; i < LENGTH; i++) printf("%.4f ", s3[i]);

printf("\n");



/*

 * Release resources

 */

H5Tclose(s2_tid);

H5Tclose(s3_tid);

H5Dclose(dataset);

H5Fclose(file);

}

Example 4. Creating and writing an extendible dataset.

This example shows how to create a 3x3 extendible dataset, to extend the dataset to 10x3, then to extend it again to 10x5.

/*  

 *   This example shows how to work with extendible dataset.

 *   In the current version of the library dataset MUST be

 *   chunked.

 *   

 */

 

#include "hdf5.h"



#define FILE        "SDSextendible.h5"

#define DATASETNAME "ExtendibleArray" 

#define RANK         2

#define NX     10

#define NY     5 



main ()

{

   hid_t       file;                          /* handles */

   hid_t       datatype, dataspace, dataset;  

   hid_t       filespace;                   

   hid_t       cparms;                     

   hsize_t      dims[2]  = { 3, 3};            /* dataset dimensions

                                                 at the creation time  */ 

   hsize_t      dims1[2] = { 3, 3};            /* data1 dimensions */ 

   hsize_t      dims2[2] = { 7, 1};            /* data2 dimensions */  

   hsize_t      dims3[2] = { 2, 2};            /* data3 dimensions */ 



   hsize_t      maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};

   hsize_t      chunk_dims[2] ={2, 5};

   hsize_t      size[2];

   hssize_t     offset[2];



   herr_t      status;                             



   int         data1[3][3] = { 1, 1, 1,       /* data to write */

                               1, 1, 1,

                               1, 1, 1 };      



   int         data2[7]    = { 2, 2, 2, 2, 2, 2, 2};



   int         data3[2][2] = { 3, 3,

                               3, 3};



/*

 * Create the data space with ulimited dimensions. 

 */

dataspace = H5Screate_simple(RANK, dims, maxdims); 



/*

 * Create a new file. If file exists its contents will be overwritten.

 */

file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);



/* 

 * Modify dataset creation properties, i.e. enable chunking.

 */

cparms = H5Pcreate (H5P_DATASET_CREATE);

status = H5Pset_chunk( cparms, RANK, chunk_dims);



/*

 * Create a new dataset within the file using cparms

 * creation properties.

 */

dataset = H5Dcreate(file, DATASETNAME, H5T_NATIVE_INT, dataspace,

                 cparms);



/*

 * Extend the dataset. This call assures that dataset is at least 3 x 3.

 */

size[0]   = 3; 

size[1]   = 3; 

status = H5Dextend (dataset, size);



/*

 * Select a hyperslab.

 */

filespace = H5Dget_space (dataset);

offset[0] = 0;

offset[1] = 0;

status = H5Sselect_hyperslab(filespace, H5S_SELECT_SET, offset, NULL,

                             dims1, NULL);  



/*

 * Write the data to the hyperslab.

 */

status = H5Dwrite(dataset, H5T_NATIVE_INT, dataspace, filespace,

                  H5P_DEFAULT, data1);



/*

 * Extend the dataset. Dataset becomes 10 x 3.

 */

dims[0]   = dims1[0] + dims2[0];

size[0]   = dims[0];  

size[1]   = dims[1]; 

status = H5Dextend (dataset, size);



/*

 * Select a hyperslab.

 */

filespace = H5Dget_space (dataset);

offset[0] = 3;

offset[1] = 0;

status = H5Sselect_hyperslab(filespace, H5S_SELECT_SET, offset, NULL,

                              dims2, NULL);  



/*

 * Define memory space

 */

dataspace = H5Screate_simple(RANK, dims2, NULL); 



/*

 * Write the data to the hyperslab.

 */

status = H5Dwrite(dataset, H5T_NATIVE_INT, dataspace, filespace,

                  H5P_DEFAULT, data2);



/*

 * Extend the dataset. Dataset becomes 10 x 5.

 */

dims[1]   = dims1[1] + dims3[1];

size[0]   = dims[0];  

size[1]   = dims[1]; 

status = H5Dextend (dataset, size);



/*

 * Select a hyperslab

 */

filespace = H5Dget_space (dataset);

offset[0] = 0;

offset[1] = 3;

status = H5Sselect_hyperslab(filespace, H5S_SELECT_SET, offset, NULL, 

                             dims3, NULL);  



/*

 * Define memory space.

 */

dataspace = H5Screate_simple(RANK, dims3, NULL); 



/*

 * Write the data to the hyperslab.

 */

status = H5Dwrite(dataset, H5T_NATIVE_INT, dataspace, filespace,

                  H5P_DEFAULT, data3);



/*

 * Resulting dataset

 *                 

         3 3 3 2 2

         3 3 3 2 2

         3 3 3 0 0

         2 0 0 0 0

         2 0 0 0 0

         2 0 0 0 0

         2 0 0 0 0

         2 0 0 0 0

         2 0 0 0 0

         2 0 0 0 0

 */ 

/*

 * Close/release resources.

 */

H5Dclose(dataset);

H5Sclose(dataspace);

H5Sclose(filespace);

H5Fclose(file);



}     

Example 5. Reading data.

This example shows how to read information the chunked dataset written by Example 4.

/*  

 *   This example shows how to read data from a chunked dataset.

 *   We will read from the file created by h5_extend_write.c 

 */

 

#include "hdf5.h"



#define FILE        "SDSextendible.h5"

#define DATASETNAME "ExtendibleArray" 

#define RANK         2

#define RANKC        1

#define NX           10

#define NY           5 



main ()

{

   hid_t       file;                        /* handles */

   hid_t       datatype, dataset;  

   hid_t       filespace;                   

   hid_t       memspace;                  

   hid_t       cparms;                   

   H5T_class_t class;                       /* data type class */

   size_t      elem_size;                   /* size of the data element

                                               stored in file */ 

   hsize_t     dims[2];                     /* dataset and chunk dimensions */ 

   hsize_t     chunk_dims[2];

   hsize_t     col_dims[1];

   size_t      size[2];

   hsize_t     count[2];

   hsize_t     offset[2];



   herr_t      status, status_n;                             



   int         data_out[NX][NY];  /* buffer for dataset to be read */

   int         chunk_out[2][5];   /* buffer for chunk to be read */

   int         column[10];        /* buffer for column to be read */

   int         i, j, rank, rank_chunk;



 

/*

 * Open the file and the dataset.

 */

file = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);

dataset = H5Dopen(file, DATASETNAME);

 

/*

 * Get dataset rank and dimension.

 */

 

filespace = H5Dget_space(dataset);    /* Get filespace handle first. */

rank      = H5Sextent_ndims(filespace);

status_n  = H5Sextent_dims(filespace, dims, NULL);

printf("dataset rank %d, dimensions %d x %d \n", rank, dims[0], dims[1]);



/*

 * Get creation properties list.

 */

cparms = H5Dget_create_plist(dataset); /* Get properties handle first. */



/* 

 * Check if dataset is chunked.

 */

 if (H5D_CHUNKED == H5Pget_layout(cparms))  {



/*

 * Get chunking information: rank and dimensions

 */

rank_chunk = H5Pget_chunk(cparms, 2, chunk_dims);

printf("chunk rank %d, dimensions %d x %d \n", rank_chunk,

        chunk_dims[0], chunk_dims[1]);

} 

 

/*

 * Define the memory space to read dataset.

 */

memspace = H5Screate_simple(RANK,dims,NULL);

 

/*

 * Read dataset back and display.

 */

status = H5Dread(dataset, H5T_NATIVE_INT, memspace, filespace,

                 H5P_DEFAULT, data_out);

    printf("\n");

    printf("Dataset: \n");

for (j = 0; j < dims[0]; j++) {

    for (i = 0; i < dims[1]; i++) printf("%d ", data_out[j][i]);

    printf("\n");

}     



/*

            dataset rank 2, dimensions 10 x 5 

            chunk rank 2, dimensions 2 x 5 



            Dataset:

            1 1 1 3 3 

            1 1 1 3 3 

            1 1 1 0 0 

            2 0 0 0 0 

            2 0 0 0 0 

            2 0 0 0 0 

            2 0 0 0 0 

            2 0 0 0 0 

            2 0 0 0 0 

            2 0 0 0 0 

*/



/*

 * Read the third column from the dataset.

 * First define memory dataspace, then define hyperslab

 * and read it into column array.

 */

col_dims[0] = 10;

memspace =  H5Screate_simple(RANKC, col_dims, NULL);



/*

 * Define the column (hyperslab) to read.

 */

offset[0] = 0;

offset[1] = 2;

count[0]  = 10;

count[1]  = 1;

status = H5Sselect_hyperslab(filespace, H5S_SELECT_SET, offset, NULL,

                             count, NULL);

status = H5Dread(dataset, H5T_NATIVE_INT, memspace, filespace,

                 H5P_DEFAULT, column);

printf("\n");

printf("Third column: \n");

for (i = 0; i < 10; i++) {

     printf("%d \n", column[i]);

}



/*



            Third column: 

            1 

            1 

            1 

            0 

            0 

            0 

            0 

            0 

            0 

            0 

*/



/*

 * Define the memory space to read a chunk.

 */

memspace = H5Screate_simple(rank_chunk,chunk_dims,NULL);



/*

 * Define chunk in the file (hyperslab) to read.

 */

offset[0] = 2;

offset[1] = 0;

count[0]  = chunk_dims[0];

count[1]  = chunk_dims[1];

status = H5Sselect_hyperslab(filespace, H5S_SELECT_SET, offset, NULL, 

                              count, NULL);



/*

 * Read chunk back and display.

 */

status = H5Dread(dataset, H5T_NATIVE_INT, memspace, filespace,

                 H5P_DEFAULT, chunk_out);

    printf("\n");

    printf("Chunk: \n");

for (j = 0; j < chunk_dims[0]; j++) {

    for (i = 0; i < chunk_dims[1]; i++) printf("%d ", chunk_out[j][i]);

    printf("\n");

}     

/*

         Chunk: 

         1 1 1 0 0 

         2 0 0 0 0 

*/



/*

 * Close/release resources.

 */

H5Pclose(cparms);

H5Dclose(dataset);

H5Sclose(filespace);

H5Sclose(memspace);

H5Fclose(file);



}     

Example 6. Creating groups.

This example shows how to create an HDF5 file with two groups, and to place some datasets within those groups.

/*

 * This example shows how to create groups within the file and    

 * datasets within the file and groups.

 */ 





#include "hdf5.h"





#define FILE    "DIR.h5"

#define RANK    2



main()

{



   hid_t    file, dir;

   hid_t    dataset, dataspace;



   herr_t   status;

   hsize_t  dims[2];

   hsize_t  size[1];



/*

 * Create a file.

 */

file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);



/*

 * Create two groups in a file.

 */

dir = H5Gcreate(file, "/IntData", 0);

status = H5Gclose(dir);



dir = H5Gcreate(file,"/FloatData", 0);

status = H5Gclose(dir);



/* 

 * Create dataspace for the character string

 */

size[0] = 80;

dataspace = H5Screate_simple(1, size, NULL);



/*

 * Create dataset "String" in the root group.  

 */

dataset = H5Dcreate(file, "String", H5T_NATIVE_CHAR, dataspace, H5P_DEFAULT);

H5Dclose(dataset);



/*

 * Create dataset "String" in the /IntData group.  

 */

dataset = H5Dcreate(file, "/IntData/String", H5T_NATIVE_CHAR, dataspace,

                    H5P_DEFAULT);

H5Dclose(dataset);



/*

 * Create dataset "String" in the /FloatData group.  

 */

dataset = H5Dcreate(file, "/FloatData/String", H5T_NATIVE_CHAR, dataspace,

                    H5P_DEFAULT);

H5Sclose(dataspace);

H5Dclose(dataset);



/*

 * Create IntArray dataset in the /IntData group by specifying full path.

 */

dims[0] = 2;

dims[1] = 3;

dataspace = H5Screate_simple(RANK, dims, NULL);

dataset = H5Dcreate(file, "/IntData/IntArray", H5T_NATIVE_INT, dataspace,

                    H5P_DEFAULT); 

H5Sclose(dataspace);

H5Dclose(dataset);



/*

 * Set current group to /IntData and attach to the dataset String.

 */



status = H5Gset (file, "/IntData");

dataset = H5Dopen(file, "String");

if (dataset > 0) printf("String dataset in /IntData group is found\n"); 

H5Dclose(dataset);



/*

 * Set current group to /FloatData.

 */

status = H5Gset (file, "/FloatData");



/* 

 * Create two datasets FlatArray and DoubleArray.

 */



dims[0] = 5;

dims[1] = 10;

dataspace = H5Screate_simple(RANK, dims, NULL);

dataset = H5Dcreate(file, "FloatArray", H5T_NATIVE_FLOAT, dataspace, H5P_DEFAULT); 

H5Sclose(dataspace);

H5Dclose(dataset);



dims[0] = 4;

dims[1] = 6;

dataspace = H5Screate_simple(RANK, dims, NULL);

dataset = H5Dcreate(file, "DoubleArray", H5T_NATIVE_DOUBLE, dataspace,

                    H5P_DEFAULT); 

H5Sclose(dataspace);

H5Dclose(dataset);



/* 

 * Attach to /FloatData/String dataset.

 */



dataset = H5Dopen(file, "/FloatData/String");

if (dataset > 0) printf("/FloatData/String dataset is found\n"); 

H5Dclose(dataset);

H5Fclose(file);



}