diff options
Diffstat (limited to 'src/H5module.h')
| -rw-r--r-- | src/H5module.h | 1404 |
1 files changed, 1403 insertions, 1 deletions
diff --git a/src/H5module.h b/src/H5module.h index 642683f..35de966 100644 --- a/src/H5module.h +++ b/src/H5module.h @@ -25,7 +25,1409 @@ #define H5_MY_PKG H5 #define H5_MY_PKG_ERR H5E_LIB -/**\defgroup H5 H5 +/** \page H5DM_UG The HDF5 Data Model and File Structure + * + * \section sec_data_model The HDF5 Data Model and File Structure + * \subsection subsec_data_model_intro Introduction + * The Hierarchical Data Format (HDF) implements a model for managing and storing data. The + * model includes an abstract data model and an abstract storage model (the data format), and + * libraries to implement the abstract model and to map the storage model to different storage + * mechanisms. The HDF5 library provides a programming interface to a concrete implementation + * of the abstract models. The library also implements a model of data transfer, an efficient + * movement of data from one stored representation to another stored representation. The figure + * below illustrates the relationships between the models and implementations. This chapter + * explains these models in detail. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig1.gif "HDF5 models and implementations" + * </td> + * </tr> + * </table> + * + * The <em>Abstract Data Model</em> is a conceptual model of data, data types, and data organization. The + * abstract data model is independent of storage medium or programming environment. The + * <em>Storage Model</em> is a standard representation for the objects of the abstract data model. The + * <a href="https://docs.hdfgroup.org/hdf5/develop/_s_p_e_c.html">HDF5 File Format Specification</a> + * defines the storage model. + * + * The <em>Programming Model</em> is a model of the computing environment and includes platforms from + * small single systems to large multiprocessors and clusters. The programming model manipulates + * (instantiates, populates, and retrieves) objects from the abstract data model. + * + * The <em>Library</em> is the concrete implementation of the programming model. The library exports the + * HDF5 APIs as its interface. In addition to implementing the objects of the abstract data model, + * the library manages data transfers from one stored form to another. Data transfer examples + * include reading from disk to memory and writing from memory to disk. + * + * <em>Stored Data</em> is the concrete implementation of the storage model. The <em>Storage Model</em> + * is mapped to several storage mechanisms including single disk files, multiple files (family of files), + * and memory representations. + * + * The HDF5 library is a C module that implements the programming model and abstract data + * model. The HDF5 library calls the operating system or other storage management software (for + * example, the MPI/IO Library) to store and retrieve persistent data. The HDF5 library may also + * link to other software such as filters for compression. The HDF5 library is linked to an + * application program which may be written in C, C++, Fortran, or Java. The application program + * implements problem specific algorithms and data structures and calls the HDF5 library to store + * and retrieve data. The figure below shows the dependencies of these modules. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig2.gif "The library, the application program, and other modules" + * </td> + * </tr> + * </table> + * + * It is important to realize that each of the software components manages data using models and + * data structures that are appropriate to the component. When data is passed between layers + * (during storage or retrieval), it is transformed from one representation to another. The figure + * below suggests some of the kinds of data structures used in the different layers. + * + * The <em>Application Program</em> uses data structures that represent the problem and algorithms + * including variables, tables, arrays, and meshes among other data structures. Depending on its + * design and function, an application may have quite a few different kinds of data structures and + * different numbers and sizes of objects. + * + * The <em>HDF5 Library</em> implements the objects of the HDF5 abstract data model. Some of these + * objects include groups, datasets, and attributes. The application program maps the application + * data structures to a hierarchy of HDF5 objects. Each application will create a mapping best + * suited to its purposes. + * + * The objects of the HDF5 abstract data model are mapped to the objects of the HDF5 storage + * model, and stored in a storage medium. The stored objects include header blocks, free lists, data + * blocks, B-trees, and other objects. Each group or dataset is stored as one or more header and data + * blocks. + * @see <a href="https://docs.hdfgroup.org/hdf5/develop/_s_p_e_c.html">HDF5 File Format Specification</a> + * for more information on how these objects are organized. The HDF5 library can also use other + * libraries and modules such as compression. + * + * <table> + * <caption>Data structures in different layers</caption> + * <tr> + * <td> + * \image html Dmodel_fig3_a.gif + * </td> + * <td> + * \image html Dmodel_fig2.gif + * </td> + * <td> + * \image html Dmodel_fig3_c.gif + * </td> + * </tr> + * </table> + * + * The important point to note is that there is not necessarily any simple correspondence between + * the objects of the application program, the abstract data model, and those of the Format + * Specification. The organization of the data of application program, and how it is mapped to the + * HDF5 abstract data model is up to the application developer. The application program only + * needs to deal with the library and the abstract data model. Most applications need not consider + * any details of the + * <a href="https://docs.hdfgroup.org/hdf5/develop/_s_p_e_c.html">HDF5 File Format Specification</a> + * or the details of how objects of abstract data model are translated to and from storage. + * + * \subsection subsec_data_model_abstract The Abstract Data Model + * The abstract data model (ADM) defines concepts for defining and describing complex data + * stored in files. The ADM is a very general model which is designed to conceptually cover many + * specific models. Many different kinds of data can be mapped to objects of the ADM, and + * therefore stored and retrieved using HDF5. The ADM is not, however, a model of any particular + * problem or application domain. Users need to map their data to the concepts of the ADM. + * + * The key concepts include: + * <ul><li>@ref subsubsec_data_model_abstract_file - a contiguous string of bytes in a computer + * store (memory, disk, etc.), and the bytes represent zero or more objects of the model</li> + * <li>@ref subsubsec_data_model_abstract_group - a collection of objects (including groups)</li> + * <li>@ref subsubsec_data_model_abstract_dataset - a multidimensional array of data elements with + * attributes and other metadata</li> + * <li>@ref subsubsec_data_model_abstract_space - a description of the dimensions of a multidimensional + * array</li> + * <li>@ref subsubsec_data_model_abstract_type - a description of a specific class of data element + * including its storage layout as a pattern of bits</li> + * <li>@ref subsubsec_data_model_abstract_attr - a named data value associated with a group, + * dataset, or named datatype</li> + * <li>@ref subsubsec_data_model_abstract_plist - a collection of parameters (some permanent and + * some transient) controlling options in the library</li> + * <li>@ref subsubsec_data_model_abstract_link - the way objects are connected</li></ul> + * + * These key concepts are described in more detail below. + * + * \subsubsection subsubsec_data_model_abstract_file File + * Abstractly, an HDF5 file is a container for an organized collection of objects. The objects are + * groups, datasets, and other objects as defined below. The objects are organized as a rooted, + * directed graph. Every HDF5 file has at least one object, the root group. See the figure below. All + * objects are members of the root group or descendants of the root group. + * + * <table> + * <caption>The HDF5 file</caption> + * <tr> + * <td> + * \image html Dmodel_fig4_b.gif + * </td> + * </tr> + * <tr> + * <td> + * \image html Dmodel_fig4_a.gif + * </td> + * </tr> + * </table> + * + * HDF5 objects have a unique identity within a single HDF5 file and can be accessed only by their + * names within the hierarchy of the file. HDF5 objects in different files do not necessarily have + * unique identities, and it is not possible to access a permanent HDF5 object except through a file. + * For more information, see \ref subsec_data_model_structure. + * + * When the file is created, the file creation properties specify settings for the file. The file creation + * properties include version information and parameters of global data structures. When the file is + * opened, the file access properties specify settings for the current access to the file. File access + * properties include parameters for storage drivers and parameters for caching and garbage + * collection. The file creation properties are set permanently for the life of the file, and the file + * access properties can be changed by closing and reopening the file. + * + * An HDF5 file can be “mounted” as part of another HDF5 file. This is analogous to Unix file + * system mounts. The root of the mounted file is attached to a group in the mounting file, and all + * the contents can be accessed as if the mounted file were part of the mounting file. + * + * @see @ref sec_file. + * + * \subsubsection subsubsec_data_model_abstract_group Group + * An HDF5 group is analogous to a file system directory. Abstractly, a group contains zero or + * more objects, and every object must be a member of at least one group. The root group is a + * special case; it may not be a member of any group. + * + * Group membership is actually implemented via link objects. See the figure below. A link object + * is owned by a group and points to a named object. Each link has a name, and each link points to + * exactly one object. Each named object has at least one and possibly many links to it. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig5.gif "Group membership via link objects" + * </td> + * </tr> + * </table> + * + * There are three classes of named objects: group, dataset, and committed (named) datatype. See + * the figure below. Each of these objects is the member of at least one group, and this means there + * is at least one link to it. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig6.gif "Classes of named objects" + * </td> + * </tr> + * </table> + * + * @see @ref sec_group. + * + * \subsubsection subsubsec_data_model_abstract_dataset Dataset + * An HDF5 dataset is a multidimensional (rectangular) array of data elements. See the figure + * below. The shape of the array (number of dimensions, size of each dimension) is described by + * the dataspace object (described in the next section below). + * + * A data element is a single unit of data which may be a number, a character, an array of numbers + * or characters, or a record of heterogeneous data elements. A data element is a set of bits. The + * layout of the bits is described by the datatype (see below). + * + * The dataspace and datatype are set when the dataset is created, and they cannot be changed for + * the life of the dataset. The dataset creation properties are set when the dataset is created. The + * dataset creation properties include the fill value and storage properties such as chunking and + * compression. These properties cannot be changed after the dataset is created. + * + * The dataset object manages the storage and access to the data. While the data is conceptually a + * contiguous rectangular array, it is physically stored and transferred in different ways depending + * on the storage properties and the storage mechanism used. The actual storage may be a set of + * compressed chunks, and the access may be through different storage mechanisms and caches. + * The dataset maps between the conceptual array of elements and the actual stored data. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig7_b.gif "The dataset" + * </td> + * </tr> + * </table> + * + * @see @ref sec_dataset. + * + * \subsubsection subsubsec_data_model_abstract_space Dataspace + * The HDF5 dataspace describes the layout of the elements of a multidimensional array. + * Conceptually, the array is a hyper-rectangle with one to 32 dimensions. HDF5 dataspaces can be + * extendable. Therefore, each dimension has a current size and a maximum size, and the maximum + * may be unlimited. The dataspace describes this hyper-rectangle: it is a list of dimensions with + * the current and maximum (or unlimited) sizes. See the figure below. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig8.gif "The dataspace" + * </td> + * </tr> + * </table> + * + * Dataspace objects are also used to describe hyperslab selections from a dataset. Any subset of the + * elements of a dataset can be selected for read or write by specifying a set of hyperslabs. A + * non-rectangular region can be selected by the union of several (rectangular) dataspaces. + * + * @see @ref sec_dataspace. + * + * \subsubsection subsubsec_data_model_abstract_type Datatype + * The HDF5 datatype object describes the layout of a single data element. A data element is a + * single element of the array; it may be a single number, a character, an array of numbers or + * carriers, or other data. The datatype object describes the storage layout of this data. + * + * Data types are categorized into 11 classes of datatype. Each class is interpreted according to a set + * of rules and has a specific set of properties to describe its storage. For instance, floating point + * numbers have exponent position and sizes which are interpreted according to appropriate + * standards for number representation. Thus, the datatype class tells what the element means, and + * the datatype describes how it is stored. + * + * The figure below shows the classification of datatypes. Atomic datatypes are indivisible. Each + * may be a single object such as a number or a string. Composite datatypes are composed of + * multiple elements of atomic datatypes. In addition to the standard types, users can define + * additional datatypes such as a 24-bit integer or a 16-bit float. + * A dataset or attribute has a single datatype object associated with it. See Figure 7 above. The + * datatype object may be used in the definition of several objects, but by default, a copy of the + * datatype object will be private to the dataset. + * + * Optionally, a datatype object can be stored in the HDF5 file. The datatype is linked into a group, + * and therefore given a name. A committed datatype (formerly called a named datatype) can be + * opened and used in any way that a datatype object can be used. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig9.gif "Datatype classifications" + * </td> + * </tr> + * </table> + * + * @see @ref sec_datatype. + * + * \subsubsection subsubsec_data_model_abstract_attr Attribute + * Any HDF5 named data object (group, dataset, or named datatype) may have zero or more user + * defined attributes. Attributes are used to document the object. The attributes of an object are + * stored with the object. + * + * An HDF5 attribute has a name and data. The data portion is similar in structure to a dataset: a + * dataspace defines the layout of an array of data elements, and a datatype defines the storage + * layout and interpretation of the elements See the figure below. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig10.gif "Attribute data elements" + * </td> + * </tr> + * </table> + * + * In fact, an attribute is very similar to a dataset with the following limitations: + * <ul><li>An attribute can only be accessed via the object</li> + * <li>Attribute names are significant only within the object</li> + * <li>An attribute should be a small object</li> + * <li>The data of an attribute must be read or written in a single access (partial reading or + * writing is not allowed)</li> + * <li>Attributes do not have attributes</li></ul> + * + * Note that the value of an attribute can be an object reference. A shared attribute or an attribute + * that is a large array can be implemented as a reference to a dataset. + * + * The name, dataspace, and datatype of an attribute are specified when it is created and cannot be + * changed over the life of the attribute. An attribute can be opened by name, by index, or by + * iterating through all the attributes of the object. + * + * @see @ref sec_attribute. + * + * \subsubsection subsubsec_data_model_abstract_plist Property List + * HDF5 has a generic property list object. Each list is a collection of name-value pairs. Each class + * of property list has a specific set of properties. Each property has an implicit name, a datatype, + * and a value. See the figure below. A property list object is created and used in ways similar to + * the other objects of the HDF5 library. + * + * Property Lists are attached to the object in the library, and they can be used by any part of the + * library. Some properties are permanent (for example, the chunking strategy for a dataset), others + * are transient (for example, buffer sizes for data transfer). A common use of a Property List is to + * pass parameters from the calling program to a VFL driver or a module of the pipeline. + * + * Property lists are conceptually similar to attributes. Property lists are information relevant to the + * behavior of the library while attributes are relevant to the user’s data and application. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig11_b.gif "The property list" + * </td> + * </tr> + * </table> + * + * Property lists are used to control optional behavior for file creation, file access, dataset creation, + * dataset transfer (read, write), and file mounting. Some property list classes are shown in the table + * below. Details of the different property lists are explained in the relevant sections of this + * document. + * + * <table> + * <caption>Property list classes and their usage</caption> + * <tr> + * <th>Property List Class</th> + * <th>Used</th> + * <th>Examples</th> + * </tr> + * <tr> + * <td>#H5P_FILE_CREATE</td> + * <td>Properties for file creation.</td> + * <td>Set size of user block.</td> + * </tr> + * <tr> + * <td>#H5P_FILE_ACCESS</td> + * <td>Properties for file access.</td> + * <td>Set parameters for VFL driver. An example is MPI I/O. </td> + * </tr> + * <tr> + * <td>#H5P_DATASET_CREATE</td> + * <td>Properties for dataset creation.</td> + * <td>Set chunking, compression, or fill value.</td> + * </tr> + * <tr> + * <td>#H5P_DATASET_XFER</td> + * <td>Properties for raw data transfer (read and write).</td> + * <td>Tune buffer sizes or memory management.</td> + * </tr> + * <tr> + * <td>#H5P_FILE_MOUNT</td> + * <td>Properties for file mounting.</td> + * <td></td> + * </tr> + * </table> + * + * @see @ref sec_plist. + * + * \subsubsection subsubsec_data_model_abstract_link Link + * This section is under construction. + * + * \subsection subsec_data_model_storage The HDF5 Storage Model + * \subsubsection subsubsec_data_model_storage_spec The Abstract Storage Model: the HDF5 Format Specification + * The <a href="https://docs.hdfgroup.org/hdf5/develop/_s_p_e_c.html">HDF5 File Format Specification</a> + * defines how HDF5 objects and data are mapped to a linear + * address space. The address space is assumed to be a contiguous array of bytes stored on some + * random access medium. The format defines the standard for how the objects of the abstract data + * model are mapped to linear addresses. The stored representation is self-describing in the sense + * that the format defines all the information necessary to read and reconstruct the original objects + * of the abstract data model. + * + * The HDF5 File Format Specification is organized in three parts: + * <ul><li>Level 0: File signature and super block</li> + * <li>Level 1: File infrastructure</li> + * <ul><li>Level 1A: B-link trees and B-tree nodes</li> + * <li>Level 1B: Group</li> + * <li>Level 1C: Group entry</li> + * <li>Level 1D: Local heaps</li> + * <li>Level 1E: Global heap</li> + * <li>Level 1F: Free-space index</li></ul> + * <li>Level 2: Data object</li> + * <ul><li>Level 2A: Data object headers</li> + * <li>Level 2B: Shared data object headers</li> + * <li>Level 2C: Data object data storage</li></ul></ul> + * + * The Level 0 specification defines the header block for the file. Header block elements include a + * signature, version information, key parameters of the file layout (such as which VFL file drivers + * are needed), and pointers to the rest of the file. Level 1 defines the data structures used + * throughout the file: the B-trees, heaps, and groups. Level 2 defines the data structure for storing + * the data objects and data. In all cases, the data structures are completely specified so that every + * bit in the file can be faithfully interpreted. + * + * It is important to realize that the structures defined in the HDF5 file format are not the same as + * the abstract data model: the object headers, heaps, and B-trees of the file specification are not + * represented in the abstract data model. The format defines a number of objects for managing the + * storage including header blocks, B-trees, and heaps. The HDF5 File Format Specification defines + * how the abstract objects (for example, groups and datasets) are represented as headers, B-tree + * blocks, and other elements. + * + * The HDF5 library implements operations to write HDF5 objects to the linear format and to read + * from the linear format to create HDF5 objects. It is important to realize that a single HDF5 + * abstract object is usually stored as several objects. A dataset, for example, might be stored in a + * header and in one or more data blocks, and these objects might not be contiguous on the hard + * disk. + * + * \subsubsection subsubsec_data_model_storage_imple Concrete Storage Model + * The HDF5 file format defines an abstract linear address space. This can be implemented in + * different storage media such as a single file or multiple files on disk or in memory. The HDF5 + * Library defines an open interface called the Virtual File Layer (VFL). The VFL allows different + * concrete storage models to be selected. + * + * The VFL defines an abstract model, an API for random access storage, and an API to plug in + * alternative VFL driver modules. The model defines the operations that the VFL driver must and + * may support, and the plug-in API enables the HDF5 library to recognize the driver and pass it + * control and data. + * + * A number of VFL drivers have been defined in the HDF5 library. Some work with a single file, + * and some work with multiple files split in various ways. Some work in serial computing + * environments, and some work in parallel computing environments. Most work with disk copies + * of HDF5 files, but one works with a memory copy. These drivers are listed in the + * \ref table_file_drivers "Supported file drivers" table. + * + * @see @ref subsec_file_alternate_drivers. + * + * Each driver isolates the details of reading and writing storage so that the rest of the HDF5 library + * and user program can be almost the same for different storage methods. The exception to this + * rule is that some VFL drivers need information from the calling application. This information is + * passed using property lists. For example, the Parallel driver requires certain control information + * that must be provided by the application. + * + * \subsection subsec_data_model_structure The Structure of an HDF5 File + * \subsubsection subsubsec_data_model_structure_file Overall File Structure + * An HDF5 file is organized as a rooted, directed graph. Named data objects are the nodes of the + * graph, and links are the directed arcs. Each arc of the graph has a name, and the root group has + * the name “/”. Objects are created and then inserted into the graph with the link operation which + * creates a named link from a group to the object. For example, the figure below illustrates the + * structure of an HDF5 file when one dataset is created. An object can be the target of more than + * one link. The names on the links must be unique within each group, but there may be many links + * with the same name in different groups. Link names are unambiguous: some ancestor will have a + * different name, or they are the same object. The graph is navigated with path names similar to + * Unix file systems. An object can be opened with a full path starting at the root group or with a + * relative path and a starting node (group). Note that all paths are relative to a single HDF5 file. In + * this sense, an HDF5 file is analogous to a single Unix file system. + * + * <table> + * <caption>An HDF5 file with one dataset</caption> + * <tr> + * <td> + * \image html Dmodel_fig12_a.gif + * </td> + * <td> + * \image html Dmodel_fig12_b.gif + * </td> + * </tr> + * </table> + * + * Note: In the figure above are two figures. The top figure represents a newly created file with one + * group, /. In the bottom figure, a dataset called /dset1 has been created. + * + * It is important to note that, just like the Unix file system, HDF5 objects do not have names. The + * names are associated with paths. An object has a unique (within the file) object identifier, but a + * single object may have many names because there may be many paths to the same object. An + * object can be renamed (moved to another group) by adding and deleting links. In this case, the + * object itself never moves. For that matter, membership in a group has no implication for the + * physical location of the stored object. + * + * Deleting a link to an object does not necessarily delete the object. The object remains available + * as long as there is at least one link to it. After all the links to an object are deleted, it can no + * longer be opened although the storage may or may not be reclaimed. + * + * It is important to realize that the linking mechanism can be used to construct very complex + * graphs of objects. For example, it is possible for an object to be shared between several groups + * and even to have more than one name in the same group. It is also possible for a group to be a + * member of itself or to be in a “cycle” in the graph. An example of a cycle is where a child is the + * parent of one of its own ancestors. + * + * \subsubsection subsubsec_data_model_structure_path HDF5 Path Names and Navigation + * The structure of the file constitutes the name space for the objects in the file. A path name is a + * string of components separated by ‘/’. Each component is the name of a link or the special + * character “.” for the current group. Link names (components) can be any string of ASCII + * characters not containing ‘/’ (except the string “.” which is reserved). However, users are advised + * to avoid the use of punctuation and non-printing characters because they may create problems for + * other software. The figure below gives a BNF grammar for HDF5 path names. + * + * <em>A BNF grammar for path names</em> + * \code + * PathName ::= AbsolutePathName | RelativePathName + * Separator ::= "/" ["/"]* + * AbsolutePathName ::= Separator [ RelativePathName ] + * RelativePathName ::= Component [ Separator RelativePathName ]* + * Component ::= "." | Name + * Name ::= Character+ - {"."} + * Character ::= {c: c in {{ legal ASCII characters } - {'/'}} + * \endcode + * + * An object can always be addressed by a full or absolute path which would start at the root group. + * As already noted, a given object can have more than one full path name. An object can also be + * addressed by a relative path which would start at a group and include the path to the object. + * + * The structure of an HDF5 file is “self-describing.” This means that it is possible to navigate the + * file to discover all the objects in the file. Basically, the structure is traversed as a graph starting at + * one node and recursively visiting the nodes of the graph. + * + * \subsubsection subsubsec_data_model_structure_example Examples of HDF5 File Structures + * The figures below show some possible HDF5 file structures with groups and datasets. The first + * figure shows the structure of a file with three groups. The second shows a dataset created in + * “/group1”. The third figure shows the structure after a dataset called dset2 has been added to the + * root group. The fourth figure shows the structure after another group and dataset have been + * added. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig14_a.gif "An HDF5 file structure with groups" + * </td> + * </tr> + * </table> + * + * Note: The figure above shows three groups; /group1 and /group2 are members of the root group. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig14_b.gif "An HDF5 file structure with groups and a dataset" + * </td> + * </tr> + * </table> + * + * Note: The figure above shows that a dataset has been created in /group1: /group1/dset1. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig14_c.gif " An HDF5 file structure with groups and datasets" + * </td> + * </tr> + * </table> + * + * Note: In the figure above, another dataset has been added as a member of the root group: /dset2. + * + * <table> + * <tr> + * <td> + * \image html Dmodel_fig14_c.gif " Another HDF5 file structure with groups and datasets" + * </td> + * </tr> + * </table> + * + * Note: In the figure above, another group and dataset have been added reusing object names: + * <em>/group2/group2/dset2</em>. + * <ol><li>HDF5 requires random access to the linear address space. For this reason it is not + * well suited for some data media such as streams.</li> + * <li>It could be said that HDF5 extends the organizing concepts of a file system to the internal + * structure of a single file.</li> + * <li>As of HDF5-1.4, the storage used for an object is reclaimed, even if all links are + * deleted.</li></ol> + * + * Next Chapter \ref sec_program + * + */ + +/** \page H5_UG The HDF5 Library and Programming Model + * + * \section sec_program The HDF5 Library and Programming Model + * \subsection subsec_program_intro Introduction + * The HDF5 library implements the HDF5 abstract data model and storage model. These models + * were described in the preceding chapter. + * + * Two major objectives of the HDF5 products are to provide tools that can be used on as many + * computational platforms as possible (portability), and to provide a reasonably object-oriented + * data model and programming interface. + * + * To be as portable as possible, the HDF5 library is implemented in portable C. C is not an + * object-oriented language, but the library uses several mechanisms and conventions to implement an + * object model. + * + * One mechanism the HDF5 library uses is to implement the objects as data structures. To refer to + * an object, the HDF5 library implements its own pointers. These pointers are called identifiers. + * An identifier is then used to invoke operations on a specific instance of an object. For example, + * when a group is opened, the API returns a group identifier. This identifier is a reference to that + * specific group and will be used to invoke future operations on that group. The identifier is valid + * only within the context it is created and remains valid until it is closed or the file is closed. This + * mechanism is essentially the same as the mechanism that C++ or other object-oriented languages + * use to refer to objects except that the syntax is C. + * + * Similarly, object-oriented languages collect all the methods for an object in a single name space. + * An example is the methods of a C++ class. The C language does not have any such mechanism, + * but the HDF5 library simulates this through its API naming convention. API function names + * begin with a common prefix that is related to the class of objects that the function operates on. + * The table below lists the HDF5 objects and the standard prefixes used by the corresponding + * HDF5 APIs. For example, functions that operate on datatype objects all have names beginning + * with H5T. + * + * <table> + * <caption>Access flags and modes</caption> + * <tr> + * <th>Prefix</th> + * <th>Operates on</th> + * </tr> + * <tr> + * <td>@ref H5A</td> + * <td>Attributes</td> + * </tr> + * <tr> + * <td>@ref H5D</td> + * <td>Datasets</td> + * </tr> + * <tr> + * <td>@ref H5E</td> + * <td>Error reports</td> + * </tr> + * <tr> + * <td>@ref H5F</td> + * <td>Files</td> + * </tr> + * <tr> + * <td>@ref H5G</td> + * <td>Groups</td> + * </tr> + * <tr> + * <td>@ref H5I</td> + * <td>Identifiers</td> + * </tr> + * <tr> + * <td>@ref H5L</td> + * <td>Links</td> + * </tr> + * <tr> + * <td>@ref H5O</td> + * <td>Objects</td> + * </tr> + * <tr> + * <td>@ref H5P</td> + * <td>Property lists</td> + * </tr> + * <tr> + * <td>@ref H5R</td> + * <td>References</td> + * </tr> + * <tr> + * <td>@ref H5S</td> + * <td>Dataspaces</td> + * </tr> + * <tr> + * <td>@ref H5T</td> + * <td>Datatypes</td> + * </tr> + * <tr> + * <td>@ref H5Z</td> + * <td>Filters</td> + * </tr> + * </table> + * + * \subsection subsec_program_model The HDF5 Programming Model + * In this section we introduce the HDF5 programming model by means of a series of short code + * samples. These samples illustrate a broad selection of common HDF5 tasks. More details are + * provided in the following chapters and in the HDF5 Reference Manual. + * + * \subsubsection subsubsec_program_model_create Creating an HDF5 File + * Before an HDF5 file can be used or referred to in any manner, it must be explicitly created or + * opened. When the need for access to a file ends, the file must be closed. The example below + * provides a C code fragment illustrating these steps. In this example, the values for the file + * creation property list and the file access property list are set to the defaults #H5P_DEFAULT. + * + * <em>Creating and closing an HDF5 file</em> + * \code + * hid_t file; // declare file identifier + * + * // Create a new file using H5F_ACC_TRUNC to truncate and overwrite + * // any file of the same name, default file creation properties, and + * // default file access properties. Then close the file. + * file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); + * status = H5Fclose(file); + * \endcode + * + * Note: If there is a possibility that a file of the declared name already exists and you wish to open + * a new file regardless of that possibility, the flag #H5F_ACC_TRUNC will cause the operation to + * overwrite the previous file. If the operation should fail in such a circumstance, use the flag + * #H5F_ACC_EXCL instead. + * + * \subsubsection subsubsec_program_model_dset Creating and Initializing a Dataset + * The essential objects within a dataset are datatype and dataspace. These are independent objects + * and are created separately from any dataset to which they may be attached. Hence, creating a + * dataset requires, at a minimum, the following steps: + * <ol><li>Create and initialize a dataspace for the dataset</li> + * <li>Define a datatype for the dataset</li> + * <li>Create and initialize the dataset</li></ol> + * + * The code in the example below illustrates the execution of these steps. + * + * <em>Create a dataset</em> + * \code + * hid_t dataset, datatype, dataspace; // declare identifiers + * + * // Create a dataspace: Describe the size of the array and + * // create the dataspace for a fixed-size dataset. + * dimsf[0] = NX; + * dimsf[1] = NY; + * dataspace = H5Screate_simple(RANK, dimsf, NULL); + * + * // Define a datatype for the data in the dataset. + * // We will store little endian integers. + * datatype = H5Tcopy(H5T_NATIVE_INT); + * status = H5Tset_order(datatype, H5T_ORDER_LE); + * + * // Create a new dataset within the file using the defined + * // dataspace and datatype and default dataset creation + * // properties. + * // NOTE: H5T_NATIVE_INT can be used as the datatype if + * // conversion to little endian is not needed. + * dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + * \endcode + * + * \subsubsection subsubsec_program_model_close Closing an Object + * An application should close an object such as a datatype, dataspace, or dataset once the object is + * no longer needed. Since each is an independent object, each must be released (or closed) + * separately. This action is frequently referred to as releasing the object’s identifier. The code in + * the example below closes the datatype, dataspace, and dataset that were created in the preceding + * section. + * + * <em>Close an object</em> + * \code + * H5Tclose(datatype); + * H5Dclose(dataset); + * H5Sclose(dataspace); + * \endcode + * + * There is a long list of HDF5 library items that return a unique identifier when the item is created + * or opened. Each time that one of these items is opened, a unique identifier is returned. Closing a + * file does not mean that the groups, datasets, or other open items are also closed. Each opened + * item must be closed separately. + * + * For more information, + * @see <a href="http://www.hdfgroup.org/HDF5/doc/Advanced/UsingIdentifiers/index.html">Using Identifiers</a> + * in the HDF5 Application Developer’s Guide under General Topics in HDF5. + * + * <h4>How Closing a File Effects Other Open Structural Elements</h4> + * Every structural element in an HDF5 file can be opened, and these elements can be opened more + * than once. Elements range in size from the entire file down to attributes. When an element is + * opened, the HDF5 library returns a unique identifier to the application. Every element that is + * opened must be closed. If an element was opened more than once, each identifier that was + * returned to the application must be closed. For example, if a dataset was opened twice, both + * dataset identifiers must be released (closed) before the dataset can be considered closed. Suppose + * an application has opened a file, a group in the file, and two datasets in the group. In order for + * the file to be totally closed, the file, group, and datasets must each be closed. Closing the file + * before the group or the datasets will not affect the state of the group or datasets: the group and + * datasets will still be open. + * + * There are several exceptions to the above general rule. One is when the #H5close function is used. + * #H5close causes a general shutdown of the library: all data is written to disk, all identifiers are + * closed, and all memory used by the library is cleaned up. Another exception occurs on parallel + * processing systems. Suppose on a parallel system an application has opened a file, a group in the + * file, and two datasets in the group. If the application uses the #H5Fclose function to close the file, + * the call will fail with an error. The open group and datasets must be closed before the file can be + * closed. A third exception is when the file access property list includes the property + * #H5F_CLOSE_STRONG. This property closes any open elements when the file is closed with + * #H5Fclose. For more information, see the #H5Pset_fclose_degree function in the HDF5 Reference + * Manual. + * + * \subsubsection subsubsec_program_model_data Writing or Reading a Dataset to or from a File + * Having created the dataset, the actual data can be written with a call to #H5Dwrite. See the + * example below. + * + * <em>Writing a dataset</em> + * \code + * // Write the data to the dataset using default transfer properties. + * status = H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, data); + * \endcode + * + * Note that the third and fourth #H5Dwrite parameters in the above example describe the + * dataspaces in memory and in the file, respectively. For now, these are both set to + * #H5S_ALL which indicates that the entire dataset is to be written. The selection of partial datasets + * and the use of differing dataspaces in memory and in storage will be discussed later in this + * chapter and in more detail elsewhere in this guide. + * + * Reading the dataset from storage is similar to writing the dataset to storage. To read an entire + * dataset, substitute #H5Dread for #H5Dwrite in the above example. + * + * \subsubsection subsubsec_program_model_partial Reading and Writing a Portion of a Dataset + * The previous section described writing or reading an entire dataset. HDF5 also supports access to + * portions of a dataset. These parts of datasets are known as selections. + * + * The simplest type of selection is a simple hyperslab. This is an n-dimensional rectangular sub-set + * of a dataset where n is equal to the dataset’s rank. Other available selections include a more + * complex hyperslab with user-defined stride and block size, a list of independent points, or the + * union of any of these. + * + * The figure below shows several sample selections. + * + * <table> + * <caption align=top>Dataset selections</caption> + * <tr> + * <td> + * \image html Pmodel_fig5_a.gif + * </td> + * </tr> + * <tr> + * <td> + * \image html Pmodel_fig5_b.gif + * </td> + * </tr> + * <tr> + * <td> + * \image html Pmodel_fig5_c.gif + * </td> + * </tr> + * <tr> + * <td> + * \image html Pmodel_fig5_d.gif<br /> + * \image html Pmodel_fig5_e.gif + * </td> + * </tr> + * </table> + * + * Note: In the figure above, selections can take the form of a simple hyperslab, a hyperslab with + * user-defined stride and block, a selection of points, or a union of any of these forms. + * + * Selections and hyperslabs are portions of a dataset. As described above, a simple hyperslab is a + * rectangular array of data elements with the same rank as the dataset’s dataspace. Thus, a simple + * hyperslab is a logically contiguous collection of points within the dataset. + * + * The more general case of a hyperslab can also be a regular pattern of points or blocks within the + * dataspace. Four parameters are required to describe a general hyperslab: the starting coordinates, + * the block size, the stride or space between blocks, and the number of blocks. These parameters + * are each expressed as a one-dimensional array with length equal to the rank of the dataspace and + * are described in the table below. + * + * <table> + * <caption></caption> + * <tr> + * <th>Parameter</th> + * <th>Definition</th> + * </tr> + * <tr> + * <td>start</td> + * <td>The coordinates of the starting location of the hyperslab in the dataset’s dataspace.</td> + * </tr> + * <tr> + * <td>block</td> + * <td>The size of each block to be selected from the dataspace. If the block parameter + * is set to NULL, the block size defaults to a single element in each dimension, as + * if the block array was set to all 1s (all ones). This will result in the selection of a + * uniformly spaced set of count points starting at start and on the interval defined + * by stride.</td> + * </tr> + * <tr> + * <td>stride</td> + * <td>The number of elements separating the starting point of each element or block to + * be selected. If the stride parameter is set to NULL, the stride size defaults to 1 + * (one) in each dimension and no elements are skipped.</td> + * </tr> + * <tr> + * <td>count</td> + * <td>The number of elements or blocks to select along each dimension.</td> + * </tr> + * </table> + * + * <h4>Reading Data into a Differently Shaped Memory Block</h4> + * For maximum flexibility in user applications, a selection in storage can be mapped into a + * differently-shaped selection in memory. All that is required is that the two selections contain the + * same number of data elements. In this example, we will first define the selection to be read from + * the dataset in storage, and then we will define the selection as it will appear in application + * memory. + * + * Suppose we want to read a 3 x 4 hyperslab from a two-dimensional dataset in a file beginning at + * the dataset element <1,2>. The first task is to create the dataspace that describes the overall rank + * and dimensions of the dataset in the file and to specify the position and size of the in-file + * hyperslab that we are extracting from that dataset. See the code below. + * + * <em>Define the selection to be read from storage</em> + * \code + * // Define dataset dataspace in file. + * dataspace = H5Dget_space(dataset); // dataspace identifier + * rank = H5Sget_simple_extent_ndims(dataspace); + * + * status_n = H5Sget_simple_extent_dims(dataspace, dims_out, NULL); + * + * // Define hyperslab in the dataset. + * offset[0] = 1; + * offset[1] = 2; + * count[0] = 3; + * count[1] = 4; + * status = H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, offset, NULL, count, NULL); + * \endcode + * + * The next task is to define a dataspace in memory. Suppose that we have in memory a + * three-dimensional 7 x 7 x 3 array into which we wish to read the two-dimensional 3 x 4 hyperslab + * described above and that we want the memory selection to begin at the element <3,0,0> and + * reside in the plane of the first two dimensions of the array. Since the in-memory dataspace is + * three-dimensional, we have to describe the in-memory selection as three-dimensional. Since we + * are keeping the selection in the plane of the first two dimensions of the in-memory dataset, the + * in-memory selection will be a 3 x 4 x 1 array defined as <3,4,1>. + * + * Notice that we must describe two things: the dimensions of the in-memory array, and the size + * and position of the hyperslab that we wish to read in. The code below illustrates how this would + * be done. + * + * <em>Define the memory dataspace and selection</em> + * \code + * // Define memory dataspace. + * dimsm[0] = 7; + * dimsm[1] = 7; + * dimsm[2] = 3; + * memspace = H5Screate_simple(RANK_OUT,dimsm,NULL); + * + * // Define memory hyperslab. + * offset_out[0] = 3; + * offset_out[1] = 0; + * offset_out[2] = 0; + * count_out[0] = 3; + * count_out[1] = 4; + * count_out[2] = 1; + * status = H5Sselect_hyperslab(memspace, H5S_SELECT_SET, offset_out, NULL, count_out, NULL); + * \endcode + * + * The hyperslab defined in the code above has the following parameters: start=(3,0,0), + * count=(3,4,1), stride and block size are NULL. + * + * <h4>Writing Data into a Differently Shaped Disk Storage Block</h4> + * Now let’s consider the opposite process of writing a selection from memory to a selection in a + * dataset in a file. Suppose that the source dataspace in memory is a 50-element, one-dimensional + * array called vector and that the source selection is a 48-element simple hyperslab that starts at the + * second element of vector. See the figure below. + * + * <table> + * <tr> + * <td> + * \image html Pmodel_fig2.gif "A one-dimensional array" + * </td> + * </tr> + * </table> + * + * Further suppose that we wish to write this data to the file as a series of 3 x 2-element blocks in a + * two-dimensional dataset, skipping one row and one column between blocks. Since the source + * selection contains 48 data elements and each block in the destination selection contains 6 data + * elements, we must define the destination selection with 8 blocks. We will write 2 blocks in the + * first dimension and 4 in the second. The code below shows how to achieve this objective. + * + * <em>The destination selection</em> + * \code + * // Select the hyperslab for the dataset in the file, using + * // 3 x 2 blocks, a (4,3) stride, a (2,4) count, and starting + * // at the position (0,1). + * start[0] = 0; start[1] = 1; + * stride[0] = 4; stride[1] = 3; + * count[0] = 2; count[1] = 4; + * block[0] = 3; block[1] = 2; + * ret = H5Sselect_hyperslab(fid, H5S_SELECT_SET, start, stride, count, block); + * + * // Create dataspace for the first dataset. + * mid1 = H5Screate_simple(MSPACE1_RANK, dim1, NULL); + * + * // Select hyperslab. + * // We will use 48 elements of the vector buffer starting at the + * // second element. Selected elements are 1 2 3 . . . 48 + * start[0] = 1; + * stride[0] = 1; + * count[0] = 48; + * block[0] = 1; + * ret = H5Sselect_hyperslab(mid1, H5S_SELECT_SET, start, stride, count, block); + * + * // Write selection from the vector buffer to the dataset in the file. + * ret = H5Dwrite(dataset, H5T_NATIVE_INT, mid1, fid, H5P_DEFAULT, vector); + * \endcode + * + * \subsubsection subsubsec_program_model_info Getting Information about a Dataset + * Although reading is analogous to writing, it is often first necessary to query a file to obtain + * information about the dataset to be read. For instance, we often need to determine the datatype + * associated with a dataset, or its dataspace (in other words, rank and dimensions). As illustrated in + * the code example below, there are several get routines for obtaining this information. + * + * <em>Routines to get dataset parameters</em> + * \code + * // Get datatype and dataspace identifiers, + * // then query datatype class, order, and size, and + * // then query dataspace rank and dimensions. + * datatype = H5Dget_type (dataset); // datatype identifier + * class = H5Tget_class (datatype); + * if (class == H5T_INTEGER) + * printf("Dataset has INTEGER type \n"); + * + * order = H5Tget_order (datatype); + * if (order == H5T_ORDER_LE) + * printf("Little endian order \n"); + * + * size = H5Tget_size (datatype); + * printf ("Size is %d \n", size); + * + * dataspace = H5Dget_space (dataset); // dataspace identifier + * + * // Find rank and retrieve current and maximum dimension sizes. + * rank = H5Sget_simple_extent_dims (dataspace, dims, max_dims); + * \endcode + * + * \subsubsection subsubsec_program_model_compound Creating and Defining Compound Datatypes + * A compound datatype is a collection of one or more data elements. Each element might be an + * atomic type, a small array, or another compound datatype. + * + * The provision for nested compound datatypes allows these structures to become quite complex. + * An HDF5 compound datatype has some similarities to a C struct or a Fortran common block. + * Though not originally designed with databases in mind, HDF5 compound datatypes are + * sometimes used in a way that is similar to a database record. Compound datatypes can become + * either a powerful tool or a complex and difficult-to-debug construct. Reasonable caution is + * advised. + * + * To create and use a compound datatype, you need to create a datatype with class compound + * (#H5T_COMPOUND) and specify the total size of the data element in bytes. A compound + * datatype consists of zero or more uniquely named members. Members can be defined in any + * order but must occupy non-overlapping regions within the datum. The table below lists the + * properties of compound datatype members. + * + * <table> + * <caption></caption> + * <tr> + * <th>Parameter</th> + * <th>Definition</th> + * </tr> + * <tr> + * <td>Index</td> + * <td>An index number between zero and N-1, where N is the number of + * members in the compound. The elements are indexed in the order of their + * location in the array of bytes.</td> + * </tr> + * <tr> + * <td>Name</td> + * <td>A string that must be unique within the members of the same datatype.</td> + * </tr> + * <tr> + * <td>Datatype</td> + * <td>An HDF5 datatype.</td> + * </tr> + * <tr> + * <td>Offset</td> + * <td>A fixed byte offset which defines the location of the first byte of that + * member in the compound datatype.</td> + * </tr> + * </table> + * + * Properties of the members of a compound datatype are defined when the member is added to the + * compound type. These properties cannot be modified later. + * + * <h4>Defining Compound Datatypes</h4> + * Compound datatypes must be built out of other datatypes. To do this, you first create an empty + * compound datatype and specify its total size. Members are then added to the compound datatype + * in any order. + * + * Each member must have a descriptive name. This is the key used to uniquely identify the + * member within the compound datatype. A member name in an HDF5 datatype does not + * necessarily have to be the same as the name of the corresponding member in the C struct in + * memory although this is often the case. You also do not need to define all the members of the C + * struct in the HDF5 compound datatype (or vice versa). + * + * Usually a C struct will be defined to hold a data point in memory, and the offsets of the members + * in memory will be the offsets of the struct members from the beginning of an instance of the + * struct. The library defines the macro that computes the offset of member m within a struct + * variable s: + * \code + * HOFFSET(s,m) + * \endcode + * + * The code below shows an example in which a compound datatype is created to describe complex + * numbers whose type is defined by the complex_t struct. + * + * <em>A compound datatype for complex numbers</em> + * \code + * Typedef struct { + * double re; //real part + * double im; //imaginary part + * } complex_t; + * + * complex_t tmp; //used only to compute offsets + * hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp); + * H5Tinsert (complex_id, "real", HOFFSET(tmp,re), H5T_NATIVE_DOUBLE); + * H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im), H5T_NATIVE_DOUBLE); + * \endcode + * + * \subsubsection subsubsec_program_model_extend Creating and Writing Extendable Datasets + * An extendable dataset is one whose dimensions can grow. One can define an HDF5 dataset to + * have certain initial dimensions with the capacity to later increase the size of any of the initial + * dimensions. For example, the figure below shows a 3 x 3 dataset (a) which is later extended to + * be a 10 x 3 dataset by adding 7 rows (b), and further extended to be a 10 x 5 dataset by adding + * two columns (c). + * + * <table> + * <tr> + * <td> + * \image html Pmodel_fig3.gif "Extending a dataset" + * </td> + * </tr> + * </table> + * + * HDF5 requires the use of chunking when defining extendable datasets. Chunking makes it + * possible to extend datasets efficiently without having to reorganize contiguous storage + * excessively. + * + * To summarize, an extendable dataset requires two conditions: + * <ol><li>Define the dataspace of the dataset as unlimited in all dimensions that might eventually be + * extended</li> + * <li>Enable chunking in the dataset creation properties</li></ol> + * + * For example, suppose we wish to create a dataset similar to the one shown in the figure above. + * We want to start with a 3 x 3 dataset, and then later we will extend it. To do this, go through the + * steps below. + * + * First, declare the dataspace to have unlimited dimensions. See the code shown below. Note the + * use of the predefined constant #H5S_UNLIMITED to specify that a dimension is unlimited. + * + * <em>Declaring a dataspace with unlimited dimensions</em> + * \code + * // dataset dimensions at creation time + * hsize_t dims[2] = {3, 3}; + * hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED}; + * + * // Create the data space with unlimited dimensions. + * dataspace = H5Screate_simple(RANK, dims, maxdims); + * \endcode + * + * Next, set the dataset creation property list to enable chunking. See the code below. + * + * <em>Enable chunking</em> + * \code + * hid_t cparms; + * hsize_t chunk_dims[2] ={2, 5}; + * + * // Modify dataset creation properties to enable chunking. + * cparms = H5Pcreate (H5P_DATASET_CREATE); + * status = H5Pset_chunk(cparms, RANK, chunk_dims); + * \endcode + * + * The next step is to create the dataset. See the code below. + * + * <em>Create a dataset</em> + * \code + * // Create a new dataset within the file using cparms creation properties. + * dataset = H5Dcreate(file, DATASETNAME, H5T_NATIVE_INT, dataspace, H5P_DEFAULT, cparms, H5P_DEFAULT); + * \endcode + * + * Finally, when the time comes to extend the size of the dataset, invoke #H5Dextend. Extending the + * dataset along the first dimension by seven rows leaves the dataset with new dimensions of + * <10,3>. See the code below. + * + * <em>Extend the dataset by seven rows</em> + * \code + * // Extend the dataset. Dataset becomes 10 x 3. + * dims[0] = dims[0] + 7; + * size[0] = dims[0]; + * size[1] = dims[1]; + * + * status = H5Dextend (dataset, size); + * \endcode + * + * \subsubsection subsubsec_program_model_group Creating and Working with Groups + * Groups provide a mechanism for organizing meaningful and extendable sets of datasets within + * an HDF5 file. The @ref H5G API provides several routines for working with groups. + * + * <h4>Creating a Group</h4> + * With no datatype, dataspace, or storage layout to define, creating a group is considerably simpler + * than creating a dataset. For example, the following code creates a group called Data in the root + * group of file. + * + * <em> Create a group</em> + * \code + * // Create a group in the file. + * grp = H5Gcreate(file, "/Data", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + * \endcode + * + * A group may be created within another group by providing the absolute name of the group to the + * #H5Gcreate function or by specifying its location. For example, to create the group Data_new in + * the group Data, you might use the sequence of calls shown below. + * + * <em>Create a group within a group</em> + * \code + * // Create group "Data_new" in the group "Data" by specifying + * // absolute name of the group. + * grp_new = H5Gcreate(file, "/Data/Data_new", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + * + * // or + * + * // Create group "Data_new" in the "Data" group. + * grp_new = H5Gcreate(grp, "Data_new", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + * \endcode + * + * This first parameter of #H5Gcreate is a location identifier. file in the first example specifies only + * the file. \em grp in the second example specifies a particular group in a particular file. Note that in + * this instance, the group identifier \em grp is used as the first parameter in the #H5Gcreate call so that + * the relative name of Data_new can be used. + * + * The third parameter of #H5Gcreate optionally specifies how much file space to reserve to store + * the names of objects that will be created in this group. If a non-positive value is supplied, the + * library provides a default size. + * + * Use #H5Gclose to close the group and release the group identifier. + * + * <h4>Creating a Dataset within a Group</h4> + * As with groups, a dataset can be created in a particular group by specifying either its absolute + * name in the file or its relative name with respect to that group. The next code excerpt uses the + * absolute name. + * + * <em>Create a dataset within a group using a relative name</em> + * \code + * // Create the dataset "Compressed_Data" in the group Data using + * // the absolute name. The dataset creation property list is + * // modified to use GZIP compression with the compression + * // effort set to 6. Note that compression can be used only when + * // the dataset is chunked. + * dims[0] = 1000; + * dims[1] = 20; + * cdims[0] = 20; + * cdims[1] = 20; + * dataspace = H5Screate_simple(RANK, dims, NULL); + * + * plist = H5Pcreate(H5P_DATASET_CREATE); + * H5Pset_chunk(plist, 2, cdims); + * H5Pset_deflate(plist, 6); + * + * dataset = H5Dcreate(file, "/Data/Compressed_Data", H5T_NATIVE_INT, dataspace, H5P_DEFAULT, + * plist, H5P_DEFAULT); + * \endcode + * + * Alternatively, you can first obtain an identifier for the group in which the dataset is to be + * created, and then create the dataset with a relative name. + * + * <em>Create a dataset within a group using a relative name</em> + * \code + * // Open the group. + * grp = H5Gopen(file, "Data", H5P_DEFAULT); + * + * // Create the dataset "Compressed_Data" in the "Data" group + * // by providing a group identifier and a relative dataset + * // name as parameters to the H5Dcreate function. + * dataset = H5Dcreate(grp, "Compressed_Data", H5T_NATIVE_INT, dataspace, H5P_DEFAULT, plist, H5P_DEFAULT); + * \endcode + * + * <h4>Accessing an Object in a Group</h4> + * Any object in a group can be accessed by its absolute or relative name. The first code snippet + * below illustrates the use of the absolute name to access the dataset <em>Compressed_Data</em> in the + * group <em>Data</em> created in the examples above. The second code snippet illustrates the use of the + * relative name. + * + * <em>Accessing a group using its relative name</em> + * \code + * // Open the dataset "Compressed_Data" in the "Data" group. + * dataset = H5Dopen(file, "/Data/Compressed_Data", H5P_DEFAULT); + * + * // Open the group "data" in the file. + * grp = H5Gopen(file, "Data", H5P_DEFAULT); + * + * // Access the "Compressed_Data" dataset in the group. + * dataset = H5Dopen(grp, "Compressed_Data", H5P_DEFAULT); + * \endcode + * + * \subsubsection subsubsec_program_model_attr Working with Attributes + * An attribute is a small dataset that is attached to a normal dataset or group. Attributes share many + * of the characteristics of datasets, so the programming model for working with attributes is + * similar in many ways to the model for working with datasets. The primary differences are that an + * attribute must be attached to a dataset or a group and sub-setting operations cannot be performed + * on attributes. + * + * To create an attribute belonging to a particular dataset or group, first create a dataspace for the + * attribute with the call to #H5Screate, and then create the attribute using #H5Acreate. For example, + * the code shown below creates an attribute called “Integer attribute” that is a member of a dataset + * whose identifier is dataset. The attribute identifier is attr2. #H5Awrite then sets the value of the + * attribute of that of the integer variable point. #H5Aclose then releases the attribute identifier. + * + * <em>Create an attribute</em> + * \code + * int point = 1; // Value of the scalar attribute + * + * // Create scalar attribute. + * aid2 = H5Screate(H5S_SCALAR); + * attr2 = H5Acreate(dataset, "Integer attribute", H5T_NATIVE_INT, aid2, H5P_DEFAULT, H5P_DEFAULT); + * + * // Write scalar attribute. + * ret = H5Awrite(attr2, H5T_NATIVE_INT, &point); + * + * // Close attribute dataspace. + * ret = H5Sclose(aid2); + * + * // Close attribute. + * ret = H5Aclose(attr2); + * \endcode + * + * <em>Read a known attribute</em> + * \code + * // Attach to the scalar attribute using attribute name, then + * // read and display its value. + * attr = H5Aopen_by_name(file_id, dataset_name, "Integer attribute", H5P_DEFAULT, H5P_DEFAULT); + * ret = H5Aread(attr, H5T_NATIVE_INT, &point_out); + * printf("The value of the attribute \"Integer attribute\" is %d \n", point_out); + * ret = H5Aclose(attr); + * \endcode + * + * To read a scalar attribute whose name and datatype are known, first open the attribute using + * #H5Aopen_by_name, and then use #H5Aread to get its value. For example, the code shown below + * reads a scalar attribute called “Integer attribute” whose datatype is a native integer and whose + * parent dataset has the identifier dataset. + * + * To read an attribute whose characteristics are not known, go through these steps. First, query the + * file to obtain information about the attribute such as its name, datatype, rank, and dimensions, + * and then read the attribute. The following code opens an attribute by its index value using + * #H5Aopen_by_idx, and then it reads in information about the datatype with #H5Aread. + * + * <em>Read an unknown attribute</em> + * \code + * // Attach to the string attribute using its index, then read and + * // display the value. + * attr = H5Aopen_by_idx(file_id, dataset_name, index_type, iter_order, 2, H5P_DEFAULT, H5P_DEFAULT); + * + * atype = H5Tcopy(H5T_C_S1); + * H5Tset_size(atype, 4); + * + * ret = H5Aread(attr, atype, string_out); + * printf("The value of the attribute with the index 2 is %s \n", string_out); + * \endcode + * + * In practice, if the characteristics of attributes are not known, the code involved in accessing and + * processing the attribute can be quite complex. For this reason, HDF5 includes a function called + * #H5Aiterate. This function applies a user-supplied function to each of a set of attributes. The + * user-supplied function can contain the code that interprets, accesses, and processes each attribute. + * + * \subsection subsec_program_transfer_pipeline The Data Transfer Pipeline + * The HDF5 library implements data transfers between different storage locations. At the lowest + * levels, the HDF5 Library reads and writes blocks of bytes to and from storage using calls to the + * virtual file layer (VFL) drivers. In addition to this, the HDF5 library manages caches of metadata + * and a data I/O pipeline. The data I/O pipeline applies compression to data blocks, transforms + * data elements, and implements selections. + * + * A substantial portion of the HDF5 library’s work is in transferring data from one environment or + * media to another. This most often involves a transfer between system memory and a storage + * medium. Data transfers are affected by compression, encryption, machine-dependent differences + * in numerical representation, and other features. So, the bit-by-bit arrangement of a given dataset + * is often substantially different in the two environments. + * + * Consider the representation on disk of a compressed and encrypted little-endian array as + * compared to the same array after it has been read from disk, decrypted, decompressed, and + * loaded into memory on a big-endian system. HDF5 performs all of the operations necessary to + * make that transition during the I/O process with many of the operations being handled by the + * VFL and the data transfer pipeline. + * + * The figure below provides a simplified view of a sample data transfer with four stages. Note that + * the modules are used only when needed. For example, if the data is not compressed, the + * compression stage is omitted. + * + * <table> + * <tr> + * <td> + * \image html Pmodel_fig6.gif "A data transfer from storage to memory" + * </td> + * </tr> + * </table> + * + * For a given I/O request, different combinations of actions may be performed by the pipeline. The + * library automatically sets up the pipeline and passes data through the processing steps. For + * example, for a read request (from disk to memory), the library must determine which logical + * blocks contain the requested data elements and fetch each block into the library’s cache. If the + * data needs to be decompressed, then the compression algorithm is applied to the block after it is + * read from disk. If the data is a selection, the selected elements are extracted from the data block + * after it is decompressed. If the data needs to be transformed (for example, byte swapped), then + * the data elements are transformed after decompression and selection. + * + * While an application must sometimes set up some elements of the pipeline, use of the pipeline is + * normally transparent to the user program. The library determines what must be done based on the + * metadata for the file, the object, and the specific request. An example of when an application + * might be required to set up some elements in the pipeline is if the application used a custom + * error-checking algorithm. + * + * In some cases, it is necessary to pass parameters to and from modules in the pipeline or among + * other parts of the library that are not directly called through the programming API. This is + * accomplished through the use of dataset transfer and data access property lists. + * + * The VFL provides an interface whereby user applications can add custom modules to the data + * transfer pipeline. For example, a custom compression algorithm can be used with the HDF5 + * Library by linking an appropriate module into the pipeline through the VFL. This requires + * creating an appropriate wrapper for the compression module and registering it with the library + * with #H5Zregister. The algorithm can then be applied to a dataset with an #H5Pset_filter call which + * will add the algorithm to the selected dataset’s transfer property list. + * + * Previous Chapter \ref sec_data_model - Next Chapter \ref sec_file + * + */ + +/** + * \defgroup H5 Library General (H5) * * Use the functions in this module to manage the life cycle of HDF5 library * instances. |