The Data Type Interface (H5T)

1. Introduction

The data type interface provides a mechanism to describe the storage format of individual data points of a data set and is hopefully designed in such a way as to allow new features to be easily added without disrupting applications that use the data type interface. A dataset (the H5D interface) is composed of a collection or raw data points of homogeneous type organized according to the data space (the H5S interface).

A data type is a collection of data type properties, all of which can be stored on disk, and which when taken as a whole, provide complete information for data conversion to or from that data type. The interface provides functions to set and query properties of a data type.

A data point is an instance of a data type, which is an instance of a type class. We have defined a set of type classes and properties which can be extended at a later time. The atomic type classes are those which describe types which cannot be decomposed at the data type interface level; all other classes are compound.

2. General Data Type Operations

The functions defined in this section operate on data types as a whole. New data types can be created from scratch or copied from existing data types. When a data type is no longer needed its resources should be released by calling H5Tclose().

Data types come in two flavors: named data types and transient data types. A named data type is stored in a file while the transient flavor is independent of any file. Named data types are always read-only, but transient types come in three varieties: modifiable, read-only, and immutable. The difference between read-only and immutable types is that immutable types cannot be closed except when the entire library is closed (the predefined types like H5T_NATIVE_INT are immutable transient types).

hid_t H5Tcreate (H5T_class_t class, size_t size)
Data types can be created by calling this function, where class is a data type class identifier. However, the only class currently allowed is H5T_COMPOUND to create a new empty compound data type where size is the total size in bytes of an instance of this data type. Other data types are created with H5Tcopy(). All functions that return data type identifiers return a negative value for failure.

hid_t H5Topen (hid_t location, const char *name)
A named data type can be opened by calling this function, which returns a handle to the data type. The handle should eventually be closed by calling H5Tclose() to release resources. The named data type returned by this function is read-only or a negative value is returned for failure. The location is either a file or group handle.

herr_t H5Tcommit (hid_t location, const char *name, hid_t type)
A transient data type (not immutable) can be committed to a file and turned into a named data type by calling this function. The location is either a file or group handle and when combined with name refers to a new named data type.

hbool_t H5Tcommitted (hid_t type)
A type can be queried to determine if it is a named type or a transient type. If this function returns a positive value then the type is named (that is, it has been committed perhaps by some other application). Datasets which return committed data types with H5Dget_type() are able to share the data type with other datasets in the same file.

hid_t H5Tcopy (hid_t type)
This function returns a modifiable transient data type which is a copy of type or a negative value for failure. If type is a dataset handle then the type returned is a modifiable transient copy of the data type of the specified dataset.

herr_t H5Tclose (hid_t type)
Releases resources associated with a data type. The data type identifier should not be subsequently used since the results would be unpredictable. It is illegal to close an immutable transient data type.

hbool_t H5Tequal (hid_t type1, hid_t type2)
Determines if two types are equal. If type1 and type2 are the same then this function returns TRUE, otherwise it returns FALSE (an error results in a negative return value).

herr_t H5Tlock (hid_t type)
A transient data type can be locked, making it immutable (read-only and not closable). The library does this to all predefined types to prevent the application from inadvertently modifying or deleting (closing) them, but the application is also allowed to do this for its own data types. Immutable data types are closed when the library closes (either by H5close() or by normal program termination).

3. Properties of Atomic Types

An atomic type is a type which cannot be decomposed into smaller units at the API level. All atomic types have a common set of properties which are augmented by properties specific to a particular type class. Some of these properties also apply to compound data types, but we discuss them only as they apply to atomic data types here. The properties and the functions that query and set their values are:

H5T_class_t H5Tget_class (hid_t type)
This property holds one of the class names: H5T_INTEGER, H5T_FLOAT, H5T_TIME, H5T_STRING, H5T_BITFIELD, or H5T_OPAQUE. This property is read-only and is set when the datatype is created or copied (see H5Tcreate(), H5Tcopy()). If this function fails it returns H5T_NO_CLASS which has a negative value (all other class constants are non-negative).

size_t H5Tget_size (hid_t type)
herr_t H5Tset_size (hid_t type, size_t size)
This property is total size of the datum in bytes, including padding which may appear on either side of the actual value. If this property is reset to a smaller value which would cause the significant part of the data to extend beyond the edge of the data type then the offset property is decremented a bit at a time. If the offset reaches zero and the significant part of the data still extends beyond the edge of the data type then the precision property is decremented a bit at a time. Decreasing the size of a data type may fail if the precesion must be decremented and the data type is of the H5T_OPAQUE class or the H5T_FLOAT bit fields would extend beyond the significant part of the type. Adjusting the size of an H5T_STRING automatically adjusts the precision as well. On error, H5Tget_size() returns zero which is never a valid size.

H5T_order_t H5Tget_order (hid_t type)
herr_t H5Tset_order (hid_t type, H5T_order_t order)
All atomic data types have a byte order which describes how the bytes of the data type are layed out in memory. If the lowest memory address contains the least significant byte of the datum then it is said to be little-endian or H5T_ORDER_LE. If the bytes are in the oposite order then they are said to be big-endian or H5T_ORDER_BE. Some data types have the same byte order on all machines and are H5T_ORDER_NONE (like character strings). If H5Tget_order() fails then it returns H5T_ORDER_ERROR which is a negative value (all successful return values are non-negative).

size_t H5Tget_precision (hid_t type)
herr_t H5Tset_precision (hid_t type, size_t precision)
Some data types occupy more bytes than what is needed to store the value. For instance, a short on a Cray is 32 significant bits in an eight-byte field. The precision property identifies the number of significant bits of a datatype and the offset property (defined below) identifies its location. The size property defined above represents the entire size (in bytes) of the data type. If the precision is decreased then padding bits are inserted on the MSB side of the significant bits (this will fail for H5T_FLOAT types if it results in the sign, mantissa, or exponent bit field extending beyond the edge of the significant bit field). On the other hand, if the precision is increased so that it "hangs over" the edge of the total size then the offset property is decremented a bit at a time. If the offset reaches zero and the significant bits still hang over the edge, then the total size is increased a byte at a time. The precision of an H5T_STRING is read-only and is always eight times the value returned by H5Tget_size(). H5Tget_precision() returns zero on failure since zero is never a valid precision.

size_t H5Tget_offset (hid_t type)
herr_t H5Tset_offset (hid_t type, size_t offset)
While the precision property defines the number of significant bits, the offset property defines the location of those bits within the entire datum. The bits of the entire data are numbered beginning at zero at the least significant bit of the least significant byte (the byte at the lowest memory address for a little-endian type or the byte at the highest address for a big-endian type). The offset property defines the bit location of the least signficant bit of a bit field whose length is precision. If the offset is increased so the significant bits "hang over" the edge of the datum, then the size property is automatically incremented. The offset is a read-only property of an H5T_STRING and is always zero. H5Tget_offset() returns zero on failure which is also a valid offset, but is guaranteed to succeed if a call to H5Tget_precision() succeeds with the same arguments.

herr_t H5Tget_pad (hid_t type, H5T_pad_t *lsb, H5T_pad_t *msb)
herr_t H5Tset_pad (hid_t type, H5T_pad_t lsb, H5T_pad_t msb)
The bits of a datum which are not significant as defined by the precision and offset properties are called padding. Padding falls into two categories: padding in the low-numbered bits is lsb padding and padding in the high-numbered bits is msb padding (bits are numbered according to the description for the offset property). Padding bits can always be set to zero (H5T_PAD_ZERO) or always set to one (H5T_PAD_ONE). The current pad types are returned through arguments of H5Tget_pad() either of which may be null pointers.

3.1. Properties of Integer Atomic Types

Integer atomic types (class=H5T_INTEGER) describe integer number formats. Such types include the following information which describes the type completely and allows conversion between various integer atomic types.

H5T_sign_t H5Tget_sign (hid_t type)
herr_t H5Tset_sign (hid_t type, H5T_sign_t sign)
Integer data can be signed two's complement (H5T_SGN_2) or unsigned (H5T_SGN_NONE). Whether data is signed or not becomes important when converting between two integer data types of differing sizes as it determines how values are truncated and sign extended.

3.2. Properties of Floating-point Atomic Types

The library supports floating-point atomic types (class=H5T_FLOAT) as long as the bits of the exponent are contiguous and stored as a biased positive number, the bits of the mantissa are contiguous and stored as a positive magnitude, and a sign bit exists which is set for negative values. Properties specific to floating-point types are:

herr_t H5Tget_fields (hid_t type, size_t *spos, size_t *epos, size_t *esize, size_t *mpos, size_t *msize)
herr_t H5Tset_fields (hid_t type, size_t spos, size_t epos, size_t esize, size_t mpos, size_t msize)
A floating-point datum has bit fields which are the exponent and mantissa as well as a mantissa sign bit. These properties define the location (bit position of least significant bit of the field) and size (in bits) of each field. The bit positions are numbered beginning at zero at the beginning of the significant part of the datum (see the descriptions of the precision and offset properties). The sign bit is always of length one and none of the fields are allowed to overlap. When expanding a floating-point type one should set the precision first; when decreasing the size one should set the field positions and sizes first.

size_t H5Tget_ebias (hid_t type)
herr_t H5Tset_ebias (hid_t type, size_t ebias)
The exponent is stored as a non-negative value which is ebias larger than the true exponent. H5Tget_ebias() returns zero on failure which is also a valid exponent bias, but the function is guaranteed to succeed if H5Tget_precision() succeeds when called with the same arguments.

H5T_norm_t H5Tget_norm (hid_t type)
herr_t H5Tset_norm (hid_t type, H5T_norm_t norm)
This property determines the normalization method of the mantissa.

H5T_pad_t H5Tget_inpad (hid_t type)
herr_t H5Tset_inpad (hid_t type, H5T_pad_t inpad)
If any internal bits (that is, bits between the sign bit, the mantissa field, and the exponent field but within the precision field) are unused, then they will be filled according to the value of this property. The inpad argument can be H5T_PAD_ZERO if the internal padding should always be set to zero, or H5T_PAD_ONE if it should always be set to one. H5Tget_inpad() returns H5T_PAD_ERROR on failure which is a negative value (successful return is always non-negative).

3.3. Properties of Date and Time Atomic Types

Dates and times (class=H5T_TIME) are stored as character strings in one of the ISO-8601 formats like "1997-12-05 16:25:30"; as character strings using the Unix asctime(3) format like "Thu Dec 05 16:25:30 1997"; as an integer value by juxtaposition of the year, month, and day-of-month, hour, minute and second in decimal like 19971205162530; as an integer value in Unix time(2) format; or other variations.

I'm deferring definition until later since they're probably not as important as the other data types.

3.4. Properties of Character String Atomic Types

Fixed-length character string types are used to store textual information. The offset property of a string is always zero and the precision property is eight times as large as the value returned by H5Tget_size() (since precision is measured in bits while size is measured in bytes). Both properties are read-only.

H5T_cset_t H5Tget_cset (hid_t type)
herr_t H5Tset_cset (hid_t type, H5T_cset_t cset)
HDF5 is able to distinguish between character sets of different nationalities and to convert between them to the extent possible. The only character set currently supported is H5T_CSET_ASCII.

H5T_str_t H5Tget_strpad (hid_t type)
herr_t H5Tset_strpad (hid_t type, H5T_str_t strpad)
The method used to store character strings differs with the programming language: C usually null terminates strings while Fortran left-justifies and space-pads strings. This property defines the storage mechanism and can be

H5T_STR_NULLTERM
A C-style string which is guaranteed to be null terminated. When converting from a longer string the value will be truncated and then a null character appended.

H5T_STR_NULLPAD
A C-style string which is padded with null characters but not necessarily null terminated. Conversion from a long string to a shorter H5T_STR_NULLPAD string will truncate but not null terminate. Conversion from a short value to a longer value will append null characters as with H5T_STR_NULLTERM.

H5T_STR_SPACEPAD
A Fortran-style string which is padded with space characters. This is the same as H5T_STR_NULLPAD except the padding character is a space instead of a null.

H5Tget_strpad() returns H5T_STR_ERROR on failure, a negative value (all successful return values are non-negative).

3.5. Properties of Bit Field Atomic Types

Converting a bit field (class=H5T_BITFIELD) from one type to another simply copies the significant bits. If the destination is smaller than the source then bits are truncated. Otherwise new bits are filled according to the msb padding type.

3.6. Properties of Opaque Atomic Types

Opaque atomic types (class=H5T_OPAQUE) act like bit fields except conversions which change the precision are not allowed. However, padding can be added or removed from either end and the bytes can be reordered. Opaque types can be used to create novel data types not directly supported by the library, but the application is responsible for data conversion of these types.

4. Properties of Compound Types

A compound data type is similar to a struct in C or a common block in Fortran: it is a collection of one or more atomic types or small arrays of such types. Each member of a compound type has a name which is unique within that type, and a byte offset that determines the first byte (smallest byte address) of that member in a compound datum. A compound data type has the following properties:

H5T_class_t H5Tget_class (hid_t type)
All compound data types belong to the type class H5T_COMPOUND. This property is read-only and is defined when a data type is created or copied (see H5Tcreate() or H5Tcopy()).

size_t H5Tget_size (hid_t type)
Compound data types have a total size in bytes which is returned by this function. All members of a compound data type must exist within this size. A value of zero is returned for failure; all successful return values are positive.

int H5Tget_nmembers (hid_t type)
A compound data type consists of zero or more members (defined in any order) with unique names and which occupy non-overlapping regions within the datum. In the functions that follow, individual members are referenced by an index number between zero and N-1, inclusive, where N is the value returned by this function. H5Tget_nmembers() returns -1 on failure.

char *H5Tget_member_name (hid_t type, int membno)
Each member has a name which is unique among its siblings in a compound data type. This function returns a pointer to a null-terminated copy of the name allocated with malloc() or the null pointer on failure. The caller is responsible for freeing the memory returned by this function.

size_t H5Tget_member_offset (hid_t type, int membno)
The byte offset of member number membno with respect to the beginning of the containing compound datum is returned by this function. A zero is returned on failure which is also a valid offset, but this function is guaranteed to succeed if a call to H5Tget_member_dims() succeeds when called with the same type and membno arguments.

int H5Tget_member_dims (hid_t type, int membno, int dims[4], int perm[4])
Each member can be a small array of up to four dimensions, making it convenient to describe things like transposition matrices. The dimensionality of the member is returned (or negative for failure) and the size in each dimension is returned through the dims argument. The perm argument describes how the array's elements are mapped to the linear address space of memory with respect to some reference order (the reference order is specified in natural language documentation which describes the compound data type). The application which "invented" the type will often use the identity permutation and other applications will use a permutation that causes the elements to be rearranged to the desired order. Only the first few elements of dims and perm are initialized according to the dimensionality of the member. Scalar members have dimensionality zero. The only permutations supported at this time are the identity permutation and the transpose permutation (in the 4d case, {0,1,2,3} and {3,2,1,0}).

hid_t H5Tget_member_type (hid_t type, int membno)
Each member has its own data type, a copy of which is returned by this function. The returned data type identifier should be released by eventually calling H5Tclose() on that type.

Properties of members of a compound data type are defined when the member is added to the compound type (see H5Tinsert()) and cannot be subsequently modified. This makes it imposible to define recursive data structures.

5. Predefined Atomic Data Types

The library predefines a modest number of data types having names like H5T_arch_base where arch is an architecture name and base is a programming type name. New types can be derived from the predifined types by copying the predefined type (see H5Tcopy()) and then modifying the result.

Architecture Name Description
IEEE This architecture defines standard floating point types in various byte orders.
STD This is an architecture that contains semi-standard datatypes like signed two's complement integers, unsigned integers, and bitfields in various byte orders.
UNIX Types which are specific to Unix operating systems are defined in this architecture. The only type currently defined is the Unix date and time types (time_t).
C
FORTRAN
Types which are specific to the C or Fortran programming languages are defined in these architectures. For instance, H5T_C_STRING defines a base string type with null termination which can be used to derive string types of other lengths.
NATIVE This architecture contains C-like data types for the machine on which the library was compiled. The types were actually defined by running the H5detect program when the library was compiled. In order to be portable, applications should almost always use this architecture to describe things in memory.
CRAY Cray architectures. These are word-addressable, big-endian systems with non-IEEE floating point.
INTEL All Intel and compatible CPU's including 80286, 80386, 80486, Pentium, Pentium-Pro, and Pentium-II. These are little-endian systems with IEEE floating-point.
MIPS All MIPS CPU's commonly used in SGI systems. These are big-endian systems with IEEE floating-point.
ALPHA All DEC Alpha CPU's, little-endian systems with IEEE floating-point.

The base name of most types consists of a letter, a precision in bits, and an indication of the byte order. The letters are:

B Bitfield
D Date and time
F Floating point
I Signed integer
R References
S Character string
U Unsigned integer

The byte order is a two-letter sequence:

BE Big endian
LE Little endian
VX Vax order



Example


Description
H5T_IEEE_F64LE Eight-byte, little-endian, IEEE floating-point
H5T_IEEE_F32BE Four-byte, big-endian, IEEE floating point
H5T_STD_I32LE Four-byte, little-endian, signed two's complement integer
H5T_STD_U16BE Two-byte, big-endian, unsigned integer
H5T_UNIX_D32LE Four-byte, little-endian, time_t
H5T_C_S1 One-byte, null-terminated string of eight-bit characters
H5T_INTEL_B64 Eight-byte bit field on an Intel CPU
H5T_CRAY_F64 Eight-byte Cray floating point
H5T_STD_ROBJ Reference to an entire object in a file

The NATIVE architecture has base names which don't follow the same rules as the others. Instead, native type names are similar to the C type names. Here are some examples:



Example


Corresponding C Type
H5T_NATIVE_CHAR signed char
H5T_NATIVE_UCHAR unsigned char
H5T_NATIVE_SHORT short
H5T_NATIVE_USHORT unsigned short
H5T_NATIVE_INT int
H5T_NATIVE_UINT unsigned
H5T_NATIVE_LONG long
H5T_NATIVE_ULONG unsigned long
H5T_NATIVE_LLONG long long
H5T_NATIVE_ULLONG unsigned long long
H5T_NATIVE_FLOAT float
H5T_NATIVE_DOUBLE double
H5T_NATIVE_LDOUBLE long double
H5T_NATIVE_HSIZE hsize_t
H5T_NATIVE_HSSIZE hssize_t
H5T_NATIVE_HERR herr_t
H5T_NATIVE_HBOOL hbool_t

Example: A 128-bit integer

To create a 128-bit, little-endian signed integer type one could use the following (increasing the precision of a type automatically increases the total size):

hid_t new_type = H5Tcopy (H5T_NATIVE_INT);
H5Tset_precision (new_type, 128);
H5Tset_order (new_type, H5T_ORDER_LE);
	      

Example: An 80-character string

To create an 80-byte null terminated string type one might do this (the offset of a character string is always zero and the precision is adjusted automatically to match the size):

hid_t str80 = H5Tcopy (H5T_C_S1);
H5Tset_size (str80, 80);
	      

6. Defining Compound Data Types

Unlike atomic data types which are derived from other atomic data types, compound data types are created from scratch. First, one creates an empty compound data type and specifies it's total size. Then members are added to the compound data type in any order.

Usually a C struct will be defined to hold a data point in memory, and the offsets of the members in memory will be the offsets of the struct members from the beginning of an instance of the struct.

HOFFSET(s,m)
This macro computes the offset of member m within a struct s.
offsetof(s,m)
This macro defined in stddef.h does exactly the same thing as the HOFFSET() macro.

Each member must have a descriptive name which is the key used to uniquely identify the member within the compound data type. A member name in an HDF5 data type does not necessarily have to be the same as the name of the member in the C struct, although this is often the case. Nor does one need to define all members of the C struct in the HDF5 compound data type (or vice versa).

Example: A simple struct

An HDF5 data type is created to describe complex numbers whose type is defined by the complex_t struct.

typedef struct {
   double re;   /*real part*/
   double im;   /*imaginary part*/
} complex_t;

hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);
H5Tinsert (complex_id, "real", HOFFSET(complex_t,re),
           H5T_NATIVE_DOUBLE);
H5Tinsert (complex_id, "imaginary", HOFFSET(complex_t,im),
           H5T_NATIVE_DOUBLE);
	      

Member alignment is handled by the HOFFSET macro. However, data stored on disk does not require alignment, so unaligned versions of compound data structures can be created to improve space efficiency on disk. These unaligned compound data types can be created by computing offsets by hand to eliminate inter-member padding, or the members can be packed by calling H5Tpack() (which modifies a data type directly, so it is usually preceded by a call to H5Tcopy()):

Example: A packed struct

This example shows how to create a disk version of a compound data type in order to store data on disk in as compact a form as possible. Packed compound data types should generally not be used to describe memory as they may violate alignment constraints for the architecture being used. Note also that using a packed data type for disk storage may involve a higher data conversion cost.

hid_t complex_disk_id = H5Tcopy (complex_id);
H5Tpack (complex_disk_id);
	      

Example: A flattened struct

Compound data types that have a compound data type member can be handled two ways. This example shows that the compound data type can be flattened, resulting in a compound type with only atomic members.

typedef struct {
   complex_t x;
   complex_t y;
} surf_t;

hid_t surf_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);
H5Tinsert (surf_id, "x-re", HOFFSET(surf_t,x.re),
           H5T_NATIVE_DOUBLE);
H5Tinsert (surf_id, "x-im", HOFFSET(surf_t,x.im),
           H5T_NATIVE_DOUBLE);
H5Tinsert (surf_id, "y-re", HOFFSET(surf_t,y.re),
           H5T_NATIVE_DOUBLE);
H5Tinsert (surf_id, "y-im", HOFFSET(surf_t,y.im),
           H5T_NATIVE_DOUBLE);
	      

Example: A nested struct

However, when the complex_t is used often it becomes inconvenient to list its members over and over again. So the alternative approach to flattening is to define a compound data type and then use it as the type of the compound members, as is done here (the typedefs are defined in the previous examples).

hid_t complex_id, surf_id; /*hdf5 data types*/

complex_id = H5Tcreate (H5T_COMPOUND, sizeof c);
H5Tinsert (complex_id, "re", HOFFSET(complex_t,re),
           H5T_NATIVE_DOUBLE);
H5Tinsert (complex_id, "im", HOFFSET(complex_t,im),
           H5T_NATIVE_DOUBLE);

surf_id = H5Tcreate (H5T_COMPOUND, sizeof s);
H5Tinsert (surf_id, "x", HOFFSET(surf_t,x), complex_id);
H5Tinsert (surf_id, "y", HOFFSET(surf_t,y), complex_id);
	      

7. Sharing Data Types among Datasets

If a file has lots of datasets which have a common data type then the file could be made smaller by having all the datasets share a single data type. Instead of storing a copy of the data type in each dataset object header, a single data type is stored and the object headers point to it. The space savings is probably only significant for datasets with a compound data type since the simple data types can be described with just a few bytes anyway.

To create a bunch of datasets that share a single data type just create the datasets with a committed (named) data type.

Example: Shared Types

To create two datasets that share a common data type one just commits the data type, giving it a name, and then uses that data type to create the datasets.

hid_t t1 = ...some transient type...;
H5Tcommit (file, "shared_type", t1);
hid_t dset1 = H5Dcreate (file, "dset1", t1, space, H5P_DEFAULT);
hid_t dset2 = H5Dcreate (file, "dset2", t1, space, H5P_DEFAULT);
	      

And to create two additional datasets later which share the same type as the first two datasets:

hid_t dset1 = H5Dopen (file, "dset1");
hid_t t2 = H5Dget_type (dset1);
hid_t dset3 = H5Dcreate (file, "dset3", t2, space, H5P_DEFAULT);
hid_t dset4 = H5Dcreate (file, "dset4", t2, space, H5P_DEFAULT);
	      

8. Data Conversion

The library is capable of converting data from one type to another and does so automatically when reading or writing the raw data of a dataset. The data type interface does not provide functions to the application for changing data types directly, but the user is allowed a certain amount of control over the conversion process.

In order to insure that data conversion exceeds disk I/O rates, common data conversion paths can be hand-tuned and optimized for performance. If a hand-tuned conversion function is not available, then the library falls back to a slower but more general conversion function. Although conversion paths include data space conversion, only data type conversions are described here. Most applications will not be concerned with data type conversions since the library will contain hand-tuned conversion functions for many common conversion paths. In fact, if an application does define a conversion function which would be of general interest, we request that the function be submitted to the HDF5 development team for inclusion in the library (there might be less overhead involved with calling an internal conversion functions than calling an application-defined conversion function).

Note: The HDF5 library contains a deliberately limited set of conversion routines. It can convert from one integer format to another, from one floating point format to another, and from one struct to another. It can also perform byte swapping when the source and destination types are otherwise the same. The library does not contain any functions for converting data between integer and floating point formats. It is anticipated that some users will find it necessary to develop float to integer or integer to float conversion functions at the application level; if they wish, users are invited to submit those functions to be considered for inclusion in future versions of the library.

A conversion path contains a source and destination data type and each path contains a hard conversion function and/or a soft conversion function. The only difference between hard and soft functions is the way in which the library chooses which function applies: A hard function applies to a specific conversion path while a soft function may apply to multiple paths. When both hard and soft functions apply to a conversion path, then the hard function is favored and when multiple soft functions apply, the one defined last is favored.

A data conversion function is of type H5T_conv_t which is defined as:

typedef herr_t (*H5T_conv_t)(hid_t src_type,
                             hid_t dest_type,
			     H5T_cdata_t *cdata,
			     size_t nelmts,
			     void *buffer,
                             void *background);
    

The conversion function is called with the source and destination data types (src_type and dst_type), path-constant data (cdata), the number of instances of the data type to convert (nelmts), a buffer which initially contains an array of data having the source type and on return will contain an array of data having the destination type (buffer), and a temporary or background buffer (background). Functions return a negative value on failure and some other value on success.

The command field of the cdata argument determines what happens within the conversion function. It's values can be:

H5T_CONV_INIT
This command is to hard conversion functions when they're registered or soft conversion functions when the library is determining if a conversion can be used for a particular path. The src_type and dst_type are the end-points of the path being queried and cdata is all zero. The library should examine the source and destination types and return zero if the conversion is possible and negative otherwise (hard conversions need not do this since they've presumably been registered only on paths they support). If the conversion is possible the library may allocate and initialize private data and assign the pointer to the priv field of cdata (or private data can be initialized later). It should also initialize the need_bkg field described below. The buf and background pointers will be null pointers.

H5T_CONV_CONV
This is the usually command which indicates that data points should be converted. The conversion function should initialize the priv field of cdata if it wasn't initialize during the H5T_CONV_INIT command and then convert nelmts instances of the src_type to the dst_type. The buffer serves as both input and output. The background buffer is supplied according to the value of the need_bkg field of cdata (the values are described below).

H5T_CONV_FREE
The conversion function is about to be removed from some path and the private data (the cdata->priv pointer) should be freed and set to null. All other pointer arguments are null and the nelmts argument is zero.

Others...
Other commands might be implemented later and conversion functions that don't support those commands should return a negative value.

Whether a background buffer is supplied to a conversion function, and whether the background buffer is initialized depends on the value of cdata->need_bkg which the conversion function should have initialized during the H5T_CONV_INIT command. It can have one of these values:

H5T_BKG_NONE
No background buffer will be supplied to the conversion function. This is the default.

H5T_BKG_TEMP
A background buffer will be supplied but it will not be initialized. This is useful for those functions requiring some extra buffer space as the buffer can probably be allocated more efficiently by the library (the application can supply the buffer as part of the dataset transfer property list).

H5T_BKG_YES
An initialized background buffer is passed to the conversion function. The buffer is initialized with the current values of the destination for the data which is passed in through the buffer argument. It can be used to "fill in between the cracks". For instance, if the destination type is a compound data type and we are initializing only part of the compound data type from the source type then the background buffer can be used to initialize the other part of the destination.

Other fields of cdata can be read or written by the conversion functions. Many of these contain performance-measuring fields which can be printed by the conversion function during the H5T_CONV_FREE command which is issued whenever the function is removed from a conversion path.

hbool_t recalc
This field is set by the library when any other data type conversion function is registered or unregistered. It allows conversion functions to cache pointers to other conversion functions and be notified when the cache should be recalculated.

unsigned long ncalls
This field contains the number of times the conversion function was called with the command H5T_CONV_CONV. It is updated automatically by the library.

unsigned long nelmts
This is the total number of data points converted by this function and is updated automatically by the library.

Once a conversion function is written it can be registered and unregistered with these functions:

herr_t H5Tregister_hard (const char *name, hid_t src_type, hid_t dest_type, H5T_conv_t func)
Once a conversion function is written, the library must be notified so it can be used. The function can be registered as a hard conversion for one or more conversion paths by calling H5Tregister_hard(), displacing any previous hard conversion for those paths. The name is used only for debugging but must be supplied.

herr_t H5Tregister_soft (const char *name, H5T_class_t src_class, H5T_class_t dest_class, H5T_conv_t func)
The function can be registered as a generic function which will be automatically added to any conversion path for which it returns an indication that it applies. The name is used only for debugging but must be supplied.

herr_t H5Tunregister (H5T_conv_t func)
A function can be removed from the set of known conversion functions by calling H5Tunregister(). The function is removed from all conversion paths.

Example: A conversion function

Here's an example application-level function that converts Cray unsigned short to any other 16-bit unsigned big-endian integer. A cray short is a big-endian value which has 32 bits of precision in the high-order bits of a 64-bit word.

 1 typedef struct {
 2     size_t dst_size;
 3     int direction;
 4 } cray_ushort2be_t;
 5 
 6 herr_t
 7 cray_ushort2be (hid_t src, hid_t dst,
 8                 H5T_cdata_t *cdata,
 9                 size_t nelmts, void *buf,
10                 const void *background)
11 {
12     unsigned char *src = (unsigned char *)buf;
13     unsigned char *dst = src;
14     cray_ushort2be_t *priv = NULL;
15 
16     switch (cdata->command) {
17     case H5T_CONV_INIT:
18         /*
19          * We are being queried to see if we handle this
20          * conversion.  We can handle conversion from
21          * Cray unsigned short to any other big-endian
22          * unsigned integer that doesn't have padding.
23          */
24         if (!H5Tequal (src, H5T_CRAY_USHORT) ||
25             H5T_ORDER_BE != H5Tget_order (dst) ||
26             H5T_SGN_NONE != H5Tget_signed (dst) ||
27             8*H5Tget_size (dst) != H5Tget_precision (dst)) {
28             return -1;
29         }
30 
31         /*
32          * Initialize private data.  If the destination size
33          * is larger than the source size, then we must
34          * process the elements from right to left.
35          */
36         cdata->priv = priv = malloc (sizeof(cray_ushort2be_t));
37         priv->dst_size = H5Tget_size (dst);
38         if (priv->dst_size>8) {
39             priv->direction = -1;
40         } else {
41             priv->direction = 1;
42         }
43         break;
44 
45     case H5T_CONV_FREE:
46         /*
47          * Free private data.
48          */
49         free (cdata->priv);
50         cdata->priv = NULL;
51         break;
52 
53     case H5T_CONV_CONV:
54         /*
55          * Convert each element, watch out for overlap src
56          * with dst on the left-most element of the buffer.
57          */
58         priv = (cray_ushort2be_t *)(cdata->priv);
59         if (priv->direction<0) {
60             src += (nelmts - 1) * 8;
61             dst += (nelmts - 1) * dst_size;
62         }
63         for (i=0; i<n; i++) {
64             if (src==dst && dst_size<4) {
65                 for (j=0; j<dst_size; j++) {
66                     dst[j] = src[j+4-dst_size];
67                 }
68             } else {
69                 for (j=0; j<4 && j<dst_size; j++) {
70                     dst[dst_size-(j+1)] = src[3-j];
71                 }
72                 for (j=4; j<dst_size; j++) {
73                     dst[dst_size-(j+1)] = 0;
74                 }
75             }
76             src += 8 * direction;
77             dst += dst_size * direction;
78         }
79         break;
80 
81     default:
82         /*
83          * Unknown command.
84          */
85         return -1;
86     }
87     return 0;
88 }
	      

The background argument is ignored since it's generally not applicable to atomic data types.

Example: Soft Registration

The convesion function described in the previous example applies to more than one conversion path. Instead of enumerating all possible paths, we register it as a soft function and allow it to decide which paths it can handle.

H5Tregister_soft ("cus2be", H5T_INTEGER, H5T_INTEGER, cray_ushort2be);
	      

This causes it to be consulted for any conversion from an integer type to another integer type. The first argument is just a short identifier which will be printed with the data type conversion statistics.

NOTE: The idea of a master soft list and being able to query conversion functions for their abilities tries to overcome problems we saw with AIO. Namely, that there was a dichotomy between generic conversions and specific conversions that made it very difficult to write a conversion function that operated on, say, integers of any size and order as long as they don't have zero padding. The AIO mechanism required such a function to be explicitly registered (like H5Tregister_hard()) for each an every possible conversion path whether that conversion path was actually used or not.


HDF Help Desk
Last modified: 27 October 1998