/** @page LBGrpCreate Creating an Group Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
\section secLBGrpCreate Creating an group An HDF5 group is a structure containing zero or more HDF5 objects. The two primary HDF5 objects are groups and datasets. To create a group, the calling program must:
  1. Obtain the location identifier where the group is to be created.
  2. Create the group.
  3. Close the group.
To create a group, the calling program must call #H5Gcreate. To close the group, #H5Gclose must be called. The close call is mandatory. For example: C \code group_id = H5Gcreate(file_id, "/MyGroup", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); status = H5Gclose (group_id); \endcode Fortran \code CALL h5gcreate_f (loc_id, name, group_id, error) CALL h5gclose_f (group_id, error) \endcode \section secLBGrpCreateRWEx Programming Example \subsection secLBGrpCreateRWExDesc Description See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. The example shows how to create and close a group. It creates a file called group.h5 in C (groupf.h5 for FORTRAN), creates a group called MyGroup in the root group, and then closes the group and file. For details on compiling an HDF5 application: [ \ref LBCompiling ] \subsection secLBGrpCreateRWExCont File Contents Shown below is the contents and the definition of the group of group.h5 (created by the C program). (The FORTRAN program creates the HDF5 file groupf.h5 and the resulting DDL shows the filename groupf.h5 in the first line.)
The Contents of group.h5.
\image html imggrpcreate.gif
group.h5 in DDL \code HDF5 "group.h5" { GROUP "/" { GROUP "MyGroup" { } } } \endcode
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics @page LBGrpCreateNames Creating Groups using Absolute and Relative Names Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
Recall that to create an HDF5 object, we have to specify the location where the object is to be created. This location is determined by the identifier of an HDF5 object and the name of the object to be created. The name of the created object can be either an absolute name or a name relative to the specified identifier. In the previous example, we used the file identifier and the absolute name /MyGroup to create a group. In this section, we discuss HDF5 names and show how to use absolute and relative names. \section secLBGrpCreateNames Names HDF5 object names are a slash-separated list of components. There are few restrictions on names: component names may be any length except zero and may contain any character except slash (/) and the null terminator. A full name may be composed of any number of component names separated by slashes, with any of the component names being the special name . (a dot or period). A name which begins with a slash is an absolute name which is accessed beginning with the root group of the file; all other names are relative names and and the named object is accessed beginning with the specified group. A special case is the name / (or equivalent) which refers to the root group. Functions which operate on names generally take a location identifier, which can be either a file identifier or a group identifier, and perform the lookup with respect to that location. Several possibilities are described in the following table:
Location Type Object Name Description
File identifier /foo/bar The object bar in group foo in the root group.
Group identifier /foo/bar The object bar in group foo in the root group of the file containing the specified group. In other words, the group identifier's only purpose is to specify a file.
File identifier / The root group of the specified file.
Group identifier / The root group of the file containing the specified group.
Group identifier foo/bar The object bar in group foo in the specified group.
File identifier . The root group of the file.
Group identifier . The specified group.
Other identifier . The specified object.
\section secLBGrpCreateNamesEx Programming Example \subsection secLBGrpCreateNamesExDesc Description See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. The example code shows how to create groups using absolute and relative names. It creates three groups: the first two groups are created using the file identifier and the group absolute names while the third group is created using a group identifier and a name relative to the specified group. For details on compiling an HDF5 application: [ \ref LBCompiling ] \subsection secLBGrpCreateNamesExRem Remarks #H5Gcreate creates a group at the location specified by a location identifier and a name. The location identifier can be a file identifier or a group identifier and the name can be relative or absolute. The first #H5Gcreate/h5gcreate_f creates the group MyGroup in the root group of the specified file. The second #H5Gcreate/h5gcreate_f creates the group Group_A in the group MyGroup in the root group of the specified file. Note that the parent group (MyGroup) already exists. The third #H5Gcreate/h5gcreate_f creates the group Group_B in the specified group. \subsection secLBGrpCreateNamesExCont File Contents Shown below is the contents and the definition of the group of groups.h5 (created by the C program). (The FORTRAN program creates the HDF5 file groupsf.h5 and the resulting DDL shows the filename groupsf.h5 in the first line.)
The Contents of groups.h5.
\image html imggrps.gif
groups.h5 in DDL \code HDF5 "groups.h5" { GROUP "/" { GROUP "MyGroup" { GROUP "Group_A" { } GROUP "Group_B" { } } } } \endcode
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics @page LBGrpDset Creating Datasets in Groups Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
\section secLBGrpDset Datasets in Groups We have shown how to create groups, datasets, and attributes. In this section, we show how to create datasets in groups. Recall that #H5Dcreate creates a dataset at the location specified by a location identifier and a name. Similar to #H5Gcreate, the location identifier can be a file identifier or a group identifier and the name can be relative or absolute. The location identifier and the name together determine the location where the dataset is to be created. If the location identifier and name refer to a group, then the dataset is created in that group. \section secLBGrpDsetEx Programming Example \subsection secLBGrpDsetExDesc Description See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. The example shows how to create a dataset in a particular group. It opens the file created in the previous example and creates two datasets: For details on compiling an HDF5 application: [ \ref LBCompiling ] \subsection secLBGrpDsetExCont File Contents Shown below is the contents and the definition of the group of groups.h5 (created by the C program). (The FORTRAN program creates the HDF5 file groupsf.h5 and the resulting DDL shows the filename groupsf.h5 in the first line.)
The contents of the file groups.h5 (groupsf.h5 for FORTRAN)
\image html imggrpdsets.gif
groups.h5 in DDL \code HDF5 "groups.h5" { GROUP "/" { GROUP "MyGroup" { GROUP "Group_A" { DATASET "dset2" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 2, 10 ) / ( 2, 10 ) } DATA { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 } } } GROUP "Group_B" { } DATASET "dset1" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 3, 3 ) / ( 3, 3 ) } DATA { 1, 2, 3, 1, 2, 3, 1, 2, 3 } } } } } \endcode groupsf.h5 in DDL \code HDF5 "groupsf.h5" { GROUP "/" { GROUP "MyGroup" { GROUP "Group_A" { DATASET "dset2" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 10, 2 ) / ( 10, 2 ) } DATA { 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10 } } } GROUP "Group_B" { } DATASET "dset1" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 3, 3 ) / ( 3, 3 ) } DATA { 1, 1, 1, 2, 2, 2, 3, 3, 3 } } } } } \endcode
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics @page LBDsetSubRW Reading From or Writing To a Subset of a Dataset Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
\section secLBDsetSubRW Dataset Subsets There are two ways that you can select a subset in an HDF5 dataset and read or write to it: HDF5 allows you to read from or write to a portion or subset of a dataset by: \li Selecting a Subset of the Dataset's Dataspace, \li Selecting a Memory Dataspace, \li Reading From or Writing to a Dataset Subset. \section secLBDsetSubRWSel Selecting a Subset of the Dataset's Dataspace First you must obtain the dataspace of a dataset in a file by calling #H5Dget_space. Then select a subset of that dataspace by calling #H5Sselect_hyperslab. The offset, count, stride and block parameters of this API define the shape and size of the selection. They must be arrays with the same number of dimensions as the rank of the dataset's dataspace. These arrays ALL work together to define a selection. A change to one of these arrays can affect the others. \li \em offset: An array that specifies the offset of the starting element of the specified hyperslab. \li \em count: An array that determines how many blocks to select from the dataspace in each dimension. If the block size for a dimension is one then the count is the number of elements along that dimension. \li \em stride: An array that allows you to sample elements along a dimension. For example, a stride of one (or NULL) will select every element along a dimension, a stride of two will select every other element, and a stride of three will select an element after every two elements. \li \em block: An array that determines the size of the element block selected from a dataspace. If the block size is one or NULL then the block size is a single element in that dimension. \section secLBDsetSubRWMem Selecting a Memory Dataspace You must select a memory dataspace in addition to a file dataspace before you can read a subset from or write a subset to a dataset. A memory dataspace can be specified by calling #H5Screate_simple. The memory dataspace passed to the read or write call must contain the same number of elements as the file dataspace. The number of elements in a dataspace selection can be determined with the #H5Sget_select_npoints API. \section secLBDsetSubRWSub Reading From or Writing To a Dataset Subset To read from or write to a dataset subset, the #H5Dread and #H5Dwrite routines are used. The memory and file dataspace identifiers from the selections that were made are passed into the read or write call. For example (C): \code status = H5Dwrite (.., .., memspace_id, dataspace_id, .., ..); \endcode \section secLBDsetSubRWProg Programming Example \subsection subsecLBDsetSubRWProgDesc Description See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. The example creates an 8 x 10 integer dataset in an HDF5 file. It then selects and writes to a 3 x 4 subset of the dataset created with the dimensions offset by 1 x 2. (If using Fortran, the dimensions will be swapped. The dataset will be 10 x 8, the subset will be 4 x 3, and the offset will be 2 x 1.) PLEASE NOTE that the examples and images below were created using C. The following image shows the dataset that gets written originally, and the subset of data that gets modified afterwards. Dimension 0 is vertical and Dimension 1 is horizontal as shown below:
\image html LBDsetSubRWProg.png
The subset on the right above is created using these values for offset, count stride, and block: \code offset = {1, 2} count = {3, 4} stride = {1, 1} block = {1, 1} \endcode \subsection subsecLBDsetSubRWProgExper Experiments with Different Selections Following are examples of changes that can be made to the example code provided to better understand how to make selections. \subsubsection subsubsecLBDsetSubRWProgExperOne Example 1 By default the example code will select and write to a 3 x 4 subset. You can modify the count parameter in the example code to select a different subset, by changing the value of DIM0_SUB (C, C++) / dim0_sub (Fortran) near the top. Change its value to 7 to create a 7 x 4 subset:
\image html imgLBDsetSubRW11.png
If you were to change the subset to 8 x 4, the selection would be beyond the extent of the dimension:
\image html imgLBDsetSubRW12.png
The write will fail with the error: "file selection+offset not within extent" \subsubsection subsubsecLBDsetSubRWProgExperTwo Example 2 In the example code provided, the memory and file dataspaces passed to the H5Dwrite call have the same size, 3 x 4 (DIM0_SUB x DIM1_SUB). Change the size of the memory dataspace to be 4 x 4 so that they do not match, and then compile: \code dimsm[0] = DIM0_SUB + 1; dimsm[1] = DIM1_SUB; memspace_id = H5Screate_simple (RANK, dimsm, NULL); \endcode The code will fail with the error: "src and dest data spaces have different sizes" How many elements are in the memory and file dataspaces that were specified above? Add these lines: \code hssize_t size; /* Just before H5Dwrite call the following */ size = H5Sget_select_npoints (memspace_id); printf ("\nmemspace_id size: %i\n", size); size = H5Sget_select_npoints (dataspace_id); printf ("dataspace_id size: %i\n", size); \endcode You should see these lines followed by the error: \code memspace_id size: 16 dataspace_id size: 12 \endcode \subsubsection subsubsecLBDsetSubRWProgExperThree Example 3 This example shows the selection that occurs if changing the values of the offset, count, stride and block parameters in the example code. This will select two blocks. The count array specifies the number of blocks. The block array specifies the size of a block. The stride must be modified to accommodate the block size.
\image html imgLBDsetSubRW31.png
Now try modifying the count as shown below. The write will fail because the selection goes beyond the extent of the dimension:
\image html imgLBDsetSubRW32.png
If the offset were 1x1 (instead of 1x2), then the selection can be made:
\image html imgLBDsetSubRW33.png
The selections above were tested with the h5_subsetbk.c example code. The memory dataspace was defined as one-dimensional. \subsection subsecLBDsetSubRWProgRem Remarks \li In addition to #H5Sselect_hyperslab, this example introduces the #H5Dget_space call to obtain the dataspace of a dataset. \li If using the default values for the stride and block parameters of #H5Sselect_hyperslab, then, for C you can specify NULL for these parameters, rather than passing in an array for each, and for Fortran 90 you can omit these parameters.
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics @page LBDatatypes Datatype Basics Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
\section secLBDtype What is a Datatype? A datatype is a collection of datatype properties which provide complete information for data conversion to or from that datatype. Datatypes in HDF5 can be grouped as follows: \li Pre-Defined Datatypes: These are datatypes that are created by HDF5. They are actually opened (and closed) by HDF5, and can have a different value from one HDF5 session to the next. \li Derived Datatypes: These are datatypes that are created or derived from the pre-defined datatypes. Although created from pre-defined types, they represent a category unto themselves. An example of a commonly used derived datatype is a string of more than one character. \section secLBDtypePre Pre-defined Datatypes The properties of pre-defined datatypes are: \li Pre-defined datatypes are opened and closed by HDF5. \li A pre-defined datatype is a handle and is NOT PERSISTENT. Its value can be different from one HDF5 session to the next. \li Pre-defined datatypes are Read-Only. \li As mentioned, other datatypes can be derived from pre-defined datatypes. There are two types of pre-defined datatypes, standard (file) and native.

Standard

A standard (or file) datatype can be:
Notes
\li Standard pre-defined datatypes are the SAME on all platforms. \li They are the datatypes that you see in an HDF5 file. \li They are typically used when creating a dataset.

Native

Native pre-defined datatypes are used for memory operations, such as reading and writing. They are NOT THE SAME on different platforms. They are similar to C type names, and are aliased to the appropriate HDF5 standard pre-defined datatype for a given platform. For example, when on an Intel based PC, #H5T_NATIVE_INT is aliased to the standard pre-defined type, #H5T_STD_I32LE. On a MIPS machine, it is aliased to #H5T_STD_I32BE.
Notes
\li Native datatypes are NOT THE SAME on all platforms. \li Native datatypes simplify memory operations (read/write). The HDF5 library automatically converts as needed. \li Native datatypes are NOT in an HDF5 File. The standard pre-defined datatype that a native datatype corresponds to is what you will see in the file.

Pre-Defined

The following table shows the native types and the standard pre-defined datatypes they correspond to. (Keep in mind that HDF5 can convert between datatypes, so you can specify a buffer of a larger type for a dataset of a given type. For example, you can read a dataset that has a short datatype into a long integer buffer.)
Some HDF5 pre-defined native datatypes and corresponding standard (file) type
C Type HDF5 Memory Type HDF5 File Type*
Integer
int #H5T_NATIVE_INT #H5T_STD_I32BE or #H5T_STD_I32LE
short #H5T_NATIVE_SHORT #H5T_STD_I16BE or #H5T_STD_I16LE
long #H5T_NATIVE_LONG #H5T_STD_I32BE, #H5T_STD_I32LE, #H5T_STD_I64BE or #H5T_STD_I64LE
long long #H5T_NATIVE_LLONG #H5T_STD_I64BE or #H5T_STD_I64LE
unsigned int #H5T_NATIVE_UINT #H5T_STD_U32BE or #H5T_STD_U32LE
unsigned short #H5T_NATIVE_USHORT #H5T_STD_U16BE or #H5T_STD_U16LE
unsigned long #H5T_NATIVE_ULONG #H5T_STD_U32BE, #H5T_STD_U32LE, #H5T_STD_U64BE or #H5T_STD_U64LE
unsigned long long #H5T_NATIVE_ULLONG #H5T_STD_U64BE or #H5T_STD_U64LE
Float
float #H5T_NATIVE_FLOAT #H5T_IEEE_F32BE or #H5T_IEEE_F32LE
double #H5T_NATIVE_DOUBLE #H5T_IEEE_F64BE or #H5T_IEEE_F64LE
Some HDF5 pre-defined native datatypes and corresponding standard (file) type
F90 Type HDF5 Memory Type HDF5 File Type*
integer H5T_NATIVE_INTEGER #H5T_STD_I32BE(8,16) or #H5T_STD_I32LE(8,16)
real H5T_NATIVE_REAL #H5T_IEEE_F32BE or #H5T_IEEE_F32LE
double-precision #H5T_NATIVE_DOUBLE #H5T_IEEE_F64BE or #H5T_IEEE_F64LE
* Note that the HDF5 File Types listed are those that are most commonly created. The file type created depends on the compiler switches and platforms being used. For example, on the Cray an integer is 64-bit, and using #H5T_NATIVE_INT (C) or H5T_NATIVE_INTEGER (F90) would result in an #H5T_STD_I64BE file type.
The following code is an example of when you would use standard pre-defined datatypes vs. native types: \code #include "hdf5.h" main() { hid_t file_id, dataset_id, dataspace_id; herr_t status; hsize_t dims[2]={4,6}; int i, j, dset_data[4][6]; for (i = 0; i < 4; i++) for (j = 0; j < 6; j++) dset_data[i][j] = i * 6 + j + 1; file_id = H5Fcreate ("dtypes.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); dataspace_id = H5Screate_simple (2, dims, NULL); dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); status = H5Dclose (dataset_id); status = H5Fclose (file_id); } \endcode By using the native types when reading and writing, the code that reads from or writes to a dataset can be the same for different platforms. Can native types also be used when creating a dataset? Yes. However, just be aware that the resulting datatype in the file will be one of the standard pre-defined types and may be different than expected. What happens if you do not use the correct native datatype for a standard (file) datatype? Your data may be incorrect or not what you expect. \section secLBDtypeDer Derived Datatypes ANY pre-defined datatype can be used to derive user-defined datatypes. To create a datatype derived from a pre-defined type:
  1. Make a copy of the pre-defined datatype: \code tid = H5Tcopy (H5T_STD_I32BE); \endcode
  2. Change the datatype.
There are numerous datatype functions that allow a user to alter a pre-defined datatype. See \ref subsecLBDtypeSpecStr below for a simple example. Refer to the \ref H5T in the \ref RM. Example functions are #H5Tset_size and #H5Tset_precision. \section secLBDtypeSpec Specific Datatypes On the Examples by API page under Datatypes you will find many example programs for creating and reading datasets with different datatypes. Below is additional information on some of the datatypes. See the Examples by API page for examples of these datatypes. \subsection subsecLBDtypeSpec Array Datatype vs Array Dataspace #H5T_ARRAY is a datatype, and it should not be confused with the dataspace of a dataset. The dataspace of a dataset can consist of a regular array of elements. For example, the datatype for a dataset could be an atomic datatype like integer, and the dataset could be an N-dimensional appendable array, as specified by the dataspace. See #H5Screate and #H5Screate_simple for details. Unlimited dimensions and subsetting are not supported when using the #H5T_ARRAY datatype. The #H5T_ARRAY datatype was primarily created to address the simple case of a compound datatype when all members of the compound datatype are of the same type and there is no need to subset by compound datatype members. Creation of such a datatype is more efficient and I/O also requires less work, because there is no alignment involved. \subsection subsecLBDtypeSpecArr Array Datatype The array class of datatypes, #H5T_ARRAY, allows the construction of true, homogeneous, multi-dimensional arrays. Since these are homogeneous arrays, each element of the array will be of the same datatype, designated at the time the array is created. Users may be confused by this datatype, as opposed to a dataset with a simple atomic datatype (eg. integer) that is an array. See subsecLBDtypeSpec for more information. Arrays can be nested. Not only is an array datatype used as an element of an HDF5 dataset, but the elements of an array datatype may be of any datatype, including another array datatype. Array datatypes cannot be subdivided for I/O; the entire array must be transferred from one dataset to another. Within certain limitations, outlined in the next paragraph, array datatypes may be N-dimensional and of any dimension size. Unlimited dimensions, however, are not supported. Functionality similar to unlimited dimension arrays is available through the use of variable-length datatypes. The maximum number of dimensions, i.e., the maximum rank, of an array datatype is specified by the HDF5 library constant #H5S_MAX_RANK. The minimum rank is 1 (one). All dimension sizes must be greater than 0 (zero). One array datatype may only be converted to another array datatype if the number of dimensions and the sizes of the dimensions are equal and the datatype of the first array's elements can be converted to the datatype of the second array's elements. \subsubsection subsubsecLBDtypeSpecArrAPI Array Datatype APIs There are three functions that are specific to array datatypes: one, #H5Tarray_create, for creating an array datatype, and two, #H5Tget_array_ndims and #H5Tget_array_dims for working with existing array datatypes.

Creating

The function #H5Tarray_create creates a new array datatype object. Parameters specify \li the base datatype of each element of the array, \li the rank of the array, i.e., the number of dimensions, \li the size of each dimension, and \li the dimension permutation of the array, i.e., whether the elements of the array are listed in C or FORTRAN order.

Working with existing array datatypes

When working with existing arrays, one must first determine the rank, or number of dimensions, of the array. The function #H5Tget_array_dims returns the rank of a specified array datatype. In many instances, one needs further information. The function #H5Tget_array_dims retrieves the permutation of the array and the size of each dimension. \subsection subsecLBDtypeSpecCmpd Compound \subsubsection subsubsecLBDtypeSpecCmpdProp Properties of compound datatypes A compound datatype is similar to a struct in C or a common block in Fortran. It is a collection of one or more atomic types or small arrays of such types. To create and use of a compound datatype you need to refer to various properties of the data compound datatype: \li It is of class compound. \li It has a fixed total size, in bytes. \li It consists of zero or more members (defined in any order) with unique names and which occupy non-overlapping regions within the datum. \li Each member has its own datatype. \li Each member is referenced by an index number between zero and N-1, where N is the number of members in the compound datatype. \li Each member has a name which is unique among its siblings in a compound datatype. \li Each member has a fixed byte offset, which is the first byte (smallest byte address) of that member in a compound datatype. \li Each member can be a small array of up to four dimensions. Properties of members of a compound datatype are defined when the member is added to the compound type and cannot be subsequently modified. \subsubsection subsubsecLBDtypeSpecCmpdDef Defining compound datatypes Compound datatypes must be built out of other datatypes. First, one creates an empty compound datatype and specifies its total size. Then members are added to the compound datatype in any order. Member names. Each member must have a descriptive name, which is the key used to uniquely identify the member within the compound datatype. A member name in an HDF5 datatype does not necessarily have to be the same as the name of the corresponding member in the C struct in memory, although this is often the case. Nor does one need to define all members of the C struct in the HDF5 compound datatype (or vice versa). Offsets. Usually a C struct will be defined to hold a data point in memory, and the offsets of the members in memory will be the offsets of the struct members from the beginning of an instance of the struct. The library defines the macro to compute the offset of a member within a struct: \code HOFFSET(s,m) \endcode This macro computes the offset of member m within a struct variable s. Here is an example in which a compound datatype is created to describe complex numbers whose type is defined by the complex_t struct. \code typedef struct { double re; /*real part */ double im; /*imaginary part */ } complex_t; complex_t tmp; /*used only to compute offsets */ hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp); H5Tinsert (complex_id, "real", HOFFSET(tmp,re), H5T_NATIVE_DOUBLE); H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im), H5T_NATIVE_DOUBLE); \endcode \subsection subsecLBDtypeSpecRef Reference There are two types of Reference datatypes in HDF5: \li \ref subsubsecLBDtypeSpecRefObj \li \ref subsubsecLBDtypeSpecRefDset \subsubsection subsubsecLBDtypeSpecRefObj Reference to objects In HDF5, objects (i.e. groups, datasets, and named datatypes) are usually accessed by name. There is another way to access stored objects -- by reference. An object reference is based on the relative file address of the object header in the file and is constant for the life of the object. Once a reference to an object is created and stored in a dataset in the file, it can be used to dereference the object it points to. References are handy for creating a file index or for grouping related objects by storing references to them in one dataset.

Creating and storing references to objects

The following steps are involved in creating and storing file references to objects:
  1. Create the objects or open them if they already exist in the file.
  2. Create a dataset to store the objects' references, by specifying #H5T_STD_REF_OBJ as the datatype
  3. Create and store references to the objects in a buffer, using #H5Rcreate.
  4. Write a buffer with the references to the dataset, using #H5Dwrite with the #H5T_STD_REF_OBJ datatype.

Reading references and accessing objects using references

The following steps are involved:
  1. Open the dataset with the references and read them. The #H5T_STD_REF_OBJ datatype must be used to describe the memory datatype.
  2. Use the read reference to obtain the identifier of the object the reference points to using #H5Rdereference.
  3. Open the dereferenced object and perform the desired operations.
  4. Close all objects when the task is complete.
\subsubsection subsubsecLBDtypeSpecRefDset Reference to a dataset region A dataset region reference points to a dataset selection in another dataset. A reference to the dataset selection (region) is constant for the life of the dataset.

Creating and storing references to dataset regions

The following steps are involved in creating and storing references to a dataset region: \li Create a dataset to store the dataset region (selection), by passing in #H5T_STD_REF_DSETREG for the datatype when calling #H5Dcreate. \li Create selection(s) in existing dataset(s) using #H5Sselect_hyperslab and/or #H5Sselect_elements. \li Create reference(s) to the selection(s) using #H5Rcreate and store them in a buffer. \li Write the references to the dataset regions in the file. \li Close all objects.

Reading references to dataset regions

The following steps are involved in reading references to dataset regions and referenced dataset regions (selections).
  1. Open and read the dataset containing references to the dataset regions. The datatype #H5T_STD_REF_DSETREG must be used during read operation.
  2. Use #H5Rdereference to obtain the dataset identifier from the read dataset region reference. OR Use #H5Rget_region to obtain the dataspace identifier for the dataset containing the selection from the read dataset region reference.
  3. With the dataspace identifier, the \ref H5S interface functions, H5Sget_select_*, can be used to obtain information about the selection.
  4. Close all objects when they are no longer needed.
The dataset with the region references was read by #H5Dread with the #H5T_STD_REF_DSETREG datatype specified. The read reference can be used to obtain the dataset identifier by calling #H5Rdereference or by obtaining obtain spatial information (dataspace and selection) with the call to #H5Rget_region. The reference to the dataset region has information for both the dataset itself and its selection. In both functions: \li The first parameter is an identifier of the dataset with the region references. \li The second parameter specifies the type of reference stored. In this example, a reference to the dataset region is stored. \li The third parameter is a buffer containing the reference of the specified type. This example introduces several H5Sget_select_* functions used to obtain information about selections:
Examples of HDF5 predefined datatypes
Function Description
#H5Sget_select_npoints Returns the number of elements in the hyperslab
#H5Sget_select_hyper_nblocks Returns the number of blocks in the hyperslab
#H5Sget_select_hyper_blocklist Returns the "lower left" and "upper right" coordinates of the blocks in the hyperslab selection
#H5Sget_select_bounds Returns the coordinates of the "minimal" block containing a hyperslab selection
#H5Sget_select_elem_npoints Returns the number of points in the element selection
#H5Sget_select_elem_pointlist Returns the coordinates of points in the element selection
\subsection subsecLBDtypeSpecStr String A simple example of creating a derived datatype is using the string datatype, #H5T_C_S1 (#H5T_FORTRAN_S1) to create strings of more than one character. Strings can be stored as either fixed or variable length, and may have different rules for padding of unused storage. \subsubsection subsecLBDtypeSpecStrFix Fixed Length 5-character String Datatype \code hid_t strtype; /* Datatype ID */ herr_t status; strtype = H5Tcopy (H5T_C_S1); status = H5Tset_size (strtype, 5); /* create string of length 5 */ \endcode \subsubsection subsecLBDtypeSpecStrVar Variable Length String Datatype \code strtype = H5Tcopy (H5T_C_S1); status = H5Tset_size (strtype, H5T_VARIABLE); \endcode The ability to derive datatypes from pre-defined types allows users to create any number of datatypes, from simple to very complex. As the term implies, variable length strings are strings of varying lengths. They are stored internally in a heap, potentially impacting efficiency in the following ways: \li Heap storage requires more space than regular raw data storage. \li Heap access generally reduces I/O efficiency because it requires individual read or write operations for each data element rather than one read or write per dataset or per data selection. \li A variable length dataset consists of pointers to the heaps of data, not the actual data. Chunking and filters, including compression, are not available for heaps. See \ref subsubsec_datatype_other_strings in the \ref UG, for more information on how fixed and variable length strings are stored. \subsection subsecLBDtypeSpecVL Variable Length Variable-length (VL) datatypes are sequences of an existing datatype (atomic, VL, or compound) which are not fixed in length from one dataset location to another. In essence, they are similar to C character strings -- a sequence of a type which is pointed to by a particular type of pointer -- although they are implemented more closely to FORTRAN strings by including an explicit length in the pointer instead of using a particular value to terminate the sequence. VL datatypes are useful to the scientific community in many different ways, some of which are listed below: \subsubsection subsubsecLBDtypeSpecVLMem Variable-length datatype memory management With each element possibly being of different sequence lengths for a dataset with a VL datatype, the memory for the VL datatype must be dynamically allocated. Currently there are two methods of managing the memory for VL datatypes: the standard C malloc/free memory allocation routines or a method of calling user-defined memory management routines to allocate or free memory. Since the memory allocated when reading (or writing) may be complicated to release, an HDF5 routine is provided to traverse a memory buffer and free the VL datatype information without leaking memory. \subsubsection subsubsecLBDtypeSpecVLDiv Variable-length datatypes cannot be divided VL datatypes are designed so that they cannot be subdivided by the library with selections, etc. This design was chosen due to the complexities in specifying selections on each VL element of a dataset through a selection API that is easy to understand. Also, the selection APIs work on dataspaces, not on datatypes. At some point in time, we may want to create a way for dataspaces to have VL components to them and we would need to allow selections of those VL regions, but that is beyond the scope of this document. \subsubsection subsubsecLBDtypeSpecVLErr What happens if the library runs out of memory while reading? It is possible for a call to #H5Dread to fail while reading in VL datatype information if the memory required exceeds that which is available. In this case, the #H5Dread call will fail gracefully and any VL data which has been allocated prior to the memory shortage will be returned to the system via the memory management routines detailed below. It may be possible to design a partial read API function at a later date, if demand for such a function warrants. \subsubsection subsubsecLBDtypeSpecVLStr Strings as variable-length datatypes Since character strings are a special case of VL data that is implemented in many different ways on different machines and in different programming languages, they are handled somewhat differently from other VL datatypes in HDF5. HDF5 has native VL strings for each language API, which are stored the same way on disk, but are exported through each language API in a natural way for that language. When retrieving VL strings from a dataset, users may choose to have them stored in memory as a native VL string or in HDF5's #hvl_t struct for VL datatypes. VL strings may be created in one of two ways: by creating a VL datatype with a base type of #H5T_C_S1 and setting its length to #H5T_VARIABLE. The second method is used to access native VL strings in memory. The library will convert between the two types, but they are stored on disk using different datatypes and have different memory representations. Multi-byte character representations, such as \em UNICODE or \em wide characters in C/C++, will need the appropriate character and string datatypes created so that they can be described properly through the datatype API. Additional conversions between these types and the current ASCII characters will also be required. Variable-width character strings (which might be compressed data or some other encoding) are not currently handled by this design. We will evaluate how to implement them based on user feedback. \subsubsection subsubsecLBDtypeSpecVLAPIs Variable-length datatype APIs

Creation

VL datatypes are created with the #H5Tvlen_create function as follows: \code type_id = H5Tvlen_create(hid_t base_type_id); \endcode The base datatype will be the datatype that the sequence is composed of, characters for character strings, vertex coordinates for polygon lists, etc. The base datatype specified for the VL datatype can be of any HDF5 datatype, including another VL datatype, a compound datatype, or an atomic datatype.

Querying base datatype of VL datatype

It may be necessary to know the base datatype of a VL datatype before memory is allocated, etc. The base datatype is queried with the #H5Tget_super function, described in the \ref H5T documentation.

Querying minimum memory required for VL information

It order to predict the memory usage that #H5Dread may need to allocate to store VL data while reading the data, the #H5Dvlen_get_buf_size function is provided: \code herr_t H5Dvlen_get_buf_size(hid_t dataset_id, hid_t type_id, hid_t space_id, hsize_t *size) \endcode This routine checks the number of bytes required to store the VL data from the dataset, using the \em space_id for the selection in the dataset on disk and the \em type_id for the memory representation of the VL data in memory. The *\em size value is modified according to how many bytes are required to store the VL data in memory.

Specifying how to manage memory for the VL datatype

The memory management method is determined by dataset transfer properties passed into the #H5Dread and #H5Dwrite functions with the dataset transfer property list. Default memory management is set by using #H5P_DEFAULT for the dataset transfer property list identifier. If #H5P_DEFAULT is used with #H5Dread, the system \em malloc and \em free calls will be used for allocating and freeing memory. In such a case, #H5P_DEFAULT should also be passed as the property list identifier to #H5Dvlen_reclaim. The rest of this subsection is relevant only to those who choose not to use default memory management. The user can choose whether to use the system \em malloc and \em free calls or user-defined, or custom, memory management functions. If user-defined memory management functions are to be used, the memory allocation and free routines must be defined via #H5Pset_vlen_mem_manager(), as follows: \code herr_t H5Pset_vlen_mem_manager(hid_t plist_id, H5MM_allocate_t alloc, void *alloc_info, H5MM_free_t free, void *free_info) \endcode The \em alloc and \em free parameters identify the memory management routines to be used. If the user has defined custom memory management routines, \em alloc and/or \em free should be set to make those routine calls (i.e., the name of the routine is used as the value of the parameter); if the user prefers to use the system's \em malloc and/or \em free, the \em alloc and \em free parameters, respectively, should be set to \em NULL The prototypes for the user-defined functions would appear as follows: \code typedef void *(*H5MM_allocate_t)(size_t size, void *info) ; typedef void (*H5MM_free_t)(void *mem, void *free_info) ; \endcode The \em alloc_info and \em free_info parameters can be used to pass along any required information to the user's memory management routines. In summary, if the user has defined custom memory management routines, the name(s) of the routines are passed in the \em alloc and \em free parameters and the custom routines' parameters are passed in the \em alloc_info and \em free_info parameters. If the user wishes to use the system \em malloc and \em free functions, the \em alloc and/or \em free parameters are set to \em NULL and the \em alloc_info and \em free_info parameters are ignored.

Recovering memory from VL buffers read in

The complex memory buffers created for a VL datatype may be reclaimed with the #H5Dvlen_reclaim function call, as follows: \code herr_t H5Dvlen_reclaim(hid_t type_id, hid_t space_id, hid_t plist_id, void *buf); \endcode The \em type_id must be the datatype stored in the buffer, \em space_id describes the selection for the memory buffer to free the VL datatypes within, \em plist_id is the dataset transfer property list which was used for the I/O transfer to create the buffer, and \em buf is the pointer to the buffer to free the VL memory within. The VL structures (#hvl_t) in the user's buffer are modified to zero out the VL information after it has been freed. If nested VL datatypes were used to create the buffer, this routine frees them from the bottom up, releasing all the memory without creating memory leaks.
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics */