+ +\section sec_learn Learning HDF5 +There are several resources for learning about HDF5. The HDF Group provides an on-line HDF5 tutorial, +documentation, examples, and videos. There are also tutorials provided by other organizations that are very useful for learning about HDF5. + +\subsection subsec_learn_intro The HDF Group Resources +For a quick introduction to HDF5 see the following: +
| +@ref IntroHDF5 + | ++A very brief introduction to HDF5 and the HDF5 programming model and APIs + | +
| +@ref LearnHDFView + | ++A tutorial for learning how to use HDFView. NO programming involved! + | +
| +@ref LearnBasics + | ++Step by step instructions for learning HDF5 that include programming examples + | +
| +Using the High Level APIs + | ++\ref H5LT \ref H5IM \ref H5TB \ref H5PT \ref H5DS + | +
| +Introduction to Parallel HDF5 + | ++A brief introduction to Parallel HDF5. If you are new to HDF5 please see the @ref LearnBasics topic first. + | +
| +\ref ViewTools + | ++\li @ref LearnHDFView +\li @ref ViewToolsCommand +\li @ref ViewToolsJPSS + | +
| +HDF5-1.10 New Features + | ++\li Introduction to the Virtual Dataset - VDS +\li Introduction to Single-Writer/Multiple-Reader (SWMR) + | +
| +Example Programs + | ++\ref HDF5Examples + | +
| +Videos + | ++\li Introduction to HDF5 +\li Parallel HDF5 + | +
+Navigate back: \ref index "Main" + +*/ diff --git a/doxygen/dox/IntroHDF5.dox b/doxygen/dox/IntroHDF5.dox new file mode 100644 index 0000000..ec46217 --- /dev/null +++ b/doxygen/dox/IntroHDF5.dox @@ -0,0 +1,627 @@ +/** @page IntroHDF5 Introduction to HDF5 + +Navigate back: \ref index "Main" / \ref GettingStarted +
+ +\section sec_intro_desc HDF5 Description +HDF5 consists of a file format for storing HDF5 data, a data model for logically organizing and accessing +HDF5 data from an application, and the software (libraries, language interfaces, and tools) for working with this format. + +\subsection subsec_intro_desc_file File Format +HDF5 consists of a file format for storing HDF5 data, a data model for logically organizing and accessing HDF5 data from an application, +and the software (libraries, language interfaces, and tools) for working with this format. + +\subsection subsec_intro_desc_dm Data Model +The HDF5 Data Model, also known as the HDF5 Abstract (or Logical) Data Model consists of +the building blocks for data organization and specification in HDF5. + +An HDF5 file (an object in itself) can be thought of as a container (or group) that holds +a variety of heterogeneous data objects (or datasets). The datasets can be images, tables, +graphs, and even documents, such as PDF or Excel: + +
| +\image html fileobj.png + | +
| +\image html group.png + | +
| +\image html dataset.png + | +
| +\image html datatype.png + | +
-
+
-
+Pre-Defined Datatypes: These are datatypes that are created by HDF5. They are actually opened (and closed)
+by HDF5 and can have different values from one HDF5 session to the next. There are two types of pre-defined datatypes:
+
-
+
- +Standard datatypes are the same on all platforms and are what you see in an HDF5 file. Their names are of the form +H5T_ARCH_BASE where ARCH is an architecture name and BASE is a programming type name. For example, #H5T_IEEE_F32BE +indicates a standard Big Endian floating point type. + +
- +Native datatypes are used to simplify memory operations (reading, writing) and are NOT the same on different platforms. +For example, #H5T_NATIVE_INT indicates an int (C). + +
+ -
+Derived Datatypes: These are datatypes that are created or derived from the pre-defined datatypes.
+An example of a commonly used derived datatype is a string of more than one character. Compound datatypes
+are also derived types. A compound datatype can be used to create a simple table, and can also be nested,
+in which it includes one more other compound datatypes.
+
+
+This is an example of a dataset with a compound datatype. Each element in the dataset consists +of a 16-bit integer, a character, a 32-bit integer, and a 2x3x2 array of 32-bit floats (the datatype). +It is a 2-dimensional 5 x 3 array (the dataspace). The datatype should not be confused with the dataspace. + ++ ++\image html cmpnddtype.png + +
+
| +\image html dataspace1.png + | +
| +\image html dataspace.png + | +
| +\image html properties.png + | +
h5dump
+The h5dump utility displays the contents of an HDF5 file in Data Description Language (\ref DDLBNF110). +Below is an example of h5dump output for an HDF5 file that contains no objects: +\code +$ h5dump file.h5 + HDF5 "file.h5" { + GROUP "/" { + } + } +\endcode + +With large files and datasets the output from h5dump can be overwhelming. +There are options that can be used to examine specific parts of an HDF5 file. +Some useful h5dump options are included below: +\code + -H, --header Display header information only (no data) + -dh5cc, h5fc, h5c++
+The built HDF5 binaries include the h5cc, h5fc, h5c++ compile scripts for compiling applications. +When using these scripts there is no need to specify the HDF5 libraries and include files. +Compiler options can be passed to the scripts. + +HDFView
+The HDFView tool allows browsing of data in HDF (HDF4 and HDF5) files. + +\section sec_intro_pm Introduction to the HDF5 Programming Model and APIs +The HDF5 Application Programming Interface is extensive, but a few functions do most of the work. + +To introduce the programming model, examples in Python and C are included below. The Python examples +use the HDF5 Python APIs (h5py). See the Examples from "Learning the Basics" page for complete examples +that can be downloaded and run for C, FORTRAN, C++, Java and Python. + +The general paradigm for working with objects in HDF5 is to: +\li Open the object. +\li Access the object. +\li Close the object. + +The library imposes an order on the operations by argument dependencies. For example, a file must be +opened before a dataset because the dataset open call requires a file handle as an argument. Objects +can be closed in any order. However, once an object is closed it no longer can be accessed. + +Keep the following in mind when looking at the example programs included in this section: +-
+
-
+
-
+
- +C routines begin with the prefix “H5*” where * is a single letter indicating the object on which the +operation is to be performed. + +
- +FORTRAN routines are similar; they begin with “h5*” and end with “_f”. + +
- +Java routines are similar; the routine names begin with “H5*” and are prefixed with “H5.” as the class. Constants are +in the HDF5Constants class and are prefixed with "HDF5Constants.". The function arguments +are usually similar, @see @ref HDF5LIB + +
-
+
-
+File Interface:
- #H5Fopen (C)
- h5fopen_f (FORTRAN)
- H5.H5Fopen (Java)
+ -
+Dataset Interface:
- #H5Dopen (C)
- h5dopen_f (FORTRAN)
- H5.H5Dopen (Java)
+ -
+Dataspace interface:
- #H5Sclose (C)
- h5sclose_f (FORTRAN)
- H5.H5Sclose (Java)
+
+ -
+For portability, the HDF5 library has its own defined types. Some common types that you will see
+in the example code are:
+
-
+
- +#hid_t is used for object handles + +
- +hsize_t is used for dimensions + +
- +#herr_t is used for many return values + +
+ -
+Language specific files must be included in applications:
+
-
+
-
+Python: Add
"import h5py / import numpy"+
+ -
+C: Add
"#include hdf5.h"+
+ -
+FORTRAN: Add
"USE HDF5"and call h5open_f and h5close_f to initialize and close the HDF5 FORTRAN interface +
+ -
+Java: Add
"import hdf.hdf5lib.H5; + import hdf.hdf5lib.HDF5Constants;"+
+
+ -
+Python: Add
| +\image html crtf-pic.png + | +
| +Python +\code + import h5py + file = h5py.File (‘file.h5’, ‘w’) + file.close () +\endcode + | +
| +C +\code + #include “hdf5.h” + + int main() { + hid_t file_id; + herr_t status; + + file_id = H5Fcreate ("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); + status = H5Fclose (file_id); + } +\endcode + | +
| +\image html crtdset.png + | +
| +Python +\code + dataset = file.create_dataset("dset",(4, 6), h5py.h5t.STD_I32BE) +\endcode + | +
| +C +\code + // Create the dataspace for the dataset. + dims[0] = 4; + dims[1] = 6; + + dataspace_id = H5Screate_simple(2, dims, NULL); + + // Create the dataset. + dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + + // Close the dataset and dataspace + status = H5Dclose(dataset_id); + status = H5Sclose(dataspace_id); +\endcode + | +
| +Python +\code + data = np.zeros((4,6)) + for i in range(4): + for j in range(6): + data[i][j]= i*6+j+1 + + dataset[...] = data <-- Write data to dataset + data_read = dataset[...] <-- Read data from dataset +\endcode + | +
| +C +\code + status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); + status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); +\endcode + | +
| +\image html crtgrp.png + | +
| +Python +\code + import h5py + file = h5py.File('dset.h5', 'r+') + group = file.create_group ('MyGroup') + file.close() +\endcode + | +
| +C +\code + group_id = H5Gcreate (file_id, "MyGroup", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + status = H5Gclose (group_id); +\endcode + | +
| +\image html crtatt.png + | +
| +Python +\code + dataset.attrs["Units"] = “Meters per second” <-- Create string + attr_data = np.zeros((2,)) + attr_data[0] = 100 + attr_data[1] = 200 + dataset.attrs.create("Speed", attr_data, (2,), “i”) <-- Create Integer +\endcode + | +
| +C +\code + hid_t attribute_id, dataspace_id; // identifiers + hsize_t dims; + int attr_data[2]; + herr_t status; + + ... + + // Initialize the attribute data. + attr_data[0] = 100; + attr_data[1] = 200; + + // Create the data space for the attribute. + dims = 2; + dataspace_id = H5Screate_simple(1, &dims, NULL); + + // Create a dataset attribute. + attribute_id = H5Acreate2 (dataset_id, "Units", H5T_STD_I32BE, + dataspace_id, H5P_DEFAULT, H5P_DEFAULT); + + // Write the attribute data. + status = H5Awrite(attribute_id, H5T_NATIVE_INT, attr_data); + + // Close the attribute. + status = H5Aclose(attribute_id); + + // Close the dataspace. + status = H5Sclose(dataspace_id); +\endcode + | +
+Navigate back: \ref index "Main" / \ref GettingStarted + + +@page HDF5Examples HDF5 Examples +Example programs of how to use HDF5 are provided below. +For HDF-EOS specific examples, see the examples +of how to access and visualize NASA HDF-EOS files using IDL, MATLAB, and NCL on the +HDF-EOS Tools and Information Center page. + +\section secHDF5Examples Examples +\li \ref LBExamples +\li Examples by API +\li Examples in the Source Code +\li Other Examples + +\section secHDF5ExamplesCompile How To Compile +For information on compiling in C, C++ and Fortran, see: \ref LBCompiling + +\section secHDF5ExamplesOther Other Examples +IDL, MATLAB, and NCL Examples for HDF-EOS +Examples of how to access and visualize NASA HDF-EOS files using IDL, MATLAB, and NCL. + +Miscellaneous Examples +These (very old) examples resulted from working with users, and are not fully tested. Most of them are in C, with a few in Fortran and Java. + +Using Special Values +These examples show how to create special values in an HDF5 application. + +*/ diff --git a/doxygen/dox/LearnBasics.dox b/doxygen/dox/LearnBasics.dox new file mode 100644 index 0000000..298672d --- /dev/null +++ b/doxygen/dox/LearnBasics.dox @@ -0,0 +1,183 @@ +/** @page LearnBasics Learning the Basics + +Navigate back: \ref index "Main" / \ref GettingStarted +
+ +\section secIntro Introduction +The following topics cover the basic features in HDF5. The topics build on each other and are +intended to be completed in order. Some sections use files created in earlier sections. The +examples used can also be found on the \ref LBExamples +page and in the HDF5 source code (C, C++, Fortran). + +\section Topics Topics +\li @subpage LBFileOrg +\li @subpage LBAPI +\li @subpage LBProg +\li @subpage LBFileCreate +\li @subpage LBDsetCreate +\li @subpage LBDsetRW +\li @subpage LBAttrCreate +\li @subpage LBGrpCreate +\li @subpage LBGrpCreateNames +\li @subpage LBGrpDset +\li @subpage LBDsetSubRW +\li @subpage LBDatatypes +\li @subpage LBPropsList +\li @subpage LBDsetLayout +\li @subpage LBExtDset +\li @subpage LBComDset +\li @subpage LBContents +\li @subpage LBQuiz +\li @subpage LBQuizAnswers +\li @subpage LBCompiling +\li @subpage LBTraining + +
+Navigate back: \ref index "Main" / \ref GettingStarted + + +@page LBExamples Examples from Learning the Basics + +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBExamples +These examples are used in the \ref LearnBasics topic. See \ref LBCompiling for details on compiling them. +PLEASE NOTE that the example programs are listed in the order they are expected to be run. Some example +programs use files created in earlier examples. + +\section secLBExamplesSrc HDF5 Source Code Examples +These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) binaries. +
| Feature + | +Examples + | +Comments + | +
|---|---|---|
| Create a file + | +C Fortran C++ Java Python + | ++ | +
| Create a dataset + | +C Fortran C++ Java Python + | ++ | +
| Read and write to a dataset + | +C Fortran C++ Java Python + | ++ | +
| Create an attribute + | +C Fortran C++ Java Python + | ++ | +
| Create a group + | +C Fortran C++ Java Python + | ++ | +
| Create groups in a file using absolute and relative paths + | +C Fortran C++ Java Python + | ++ | +
| Create datasets in a group + | +C Fortran C++ Java Python + | ++ | +
| Create a file and dataset and select/read a subset from the dataset + | +C Fortran C++ Java Python + | +Also see examples to Write by row (and column) below. + | +
| Create an extendible (unlimited dimension) dataset + | +C Fortran C++ Java Python + | +Also see examples to Extend by row (and column) below + | +
| Create a chunked and compressed dataset + | +C Fortran C++ Java Python + | ++ | +
| Feature + | +Examples + | +
|---|---|
| Write by row + | +C Fortran + | +
| Write by column + | +C Fortran + | +
| Extend by row + | +C Fortran + | +
| Extend by column + | +C Fortran + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnBasics1.dox b/doxygen/dox/LearnBasics1.dox new file mode 100644 index 0000000..a9b6d0e --- /dev/null +++ b/doxygen/dox/LearnBasics1.dox @@ -0,0 +1,1023 @@ +/** @page LBFileOrg HDF5 File Organization + +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBFileOrg HDF5 file +An HDF5 file is a container for storing a variety of scientific data and is composed of two primary types of objects: groups and datasets. + +\li HDF5 group: a grouping structure containing zero or more HDF5 objects, together with supporting metadata +\li HDF5 dataset: a multidimensional array of data elements, together with supporting metadata + +Any HDF5 group or dataset may have an associated attribute list. An HDF5 attribute is a user-defined HDF5 structure +that provides extra information about an HDF5 object. + +Working with groups and datasets is similar in many ways to working with directories and files in UNIX. As with UNIX +directories and files, an HDF5 object in an HDF5 file is often referred to by its full path name (also called an absolute path name). + +\li
/ signifies the root group.
+
+\li /foo signifies a member of the root group called foo.
+
+\li /foo/zoo signifies a member of the group foo, which in turn is a member of the root group.
+
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
+
+@page LBAPI The HDF5 API
+
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
++ +\section secLBAPI HDF5 C API +The HDF5 library provides several interfaces, or APIs. These APIs provide routines for creating, +accessing, and manipulating HDF5 files and objects. + +The library itself is implemented in C. To facilitate the work of FORTRAN 90, C++ and Java programmers, +HDF5 function wrappers have been developed in each of these languages. This tutorial discusses the use +of the C functions and the FORTRAN wrappers. + +All C routines in the HDF5 library begin with a prefix of the form H5*, where * is one or two uppercase +letters indicating the type of object on which the function operates. +The FORTRAN wrappers come in the form of subroutines that begin with h5 and end with _f. +Java routine names begin with “H5*” and are prefixed with “H5.” as the class. Constants are +in the HDF5Constants class and are prefixed with "HDF5Constants.". +The APIs are listed below: +
| API + | +DESCRIPTION + | +
|---|---|
| H5 + | +Library Functions: general-purpose H5 functions + | +
| H5A + | +Annotation Interface: attribute access and manipulation routines + | +
| H5D + | +Dataset Interface: dataset access and manipulation routines + | +
| H5E + | +Error Interface: error handling routines + | +
| H5F + | +File Interface: file access routines + | +
| H5G + | +Group Interface: group creation and operation routines + | +
| H5I + | +Identifier Interface: identifier routines + | +
| H5L + | +Link Interface: link routines + | +
| H5O + | +Object Interface: object routines + | +
| H5P + | +Property List Interface: object property list manipulation routines + | +
| H5R + | +Reference Interface: reference routines + | +
| H5S + | +Dataspace Interface: dataspace definition and access routines + | +
| H5T + | +Datatype Interface: datatype creation and manipulation routines + | +
| H5Z + | +Compression Interface: compression routine(s) + | +
+ +Keep the following in mind when looking at the example programs included in this tutorial: + +\section LBProgAPI APIs vary with different languages +\li C routines begin with the prefix “H5*” where * is a single letter indicating the object on which the operation is to be performed: +
| File Interface: | +#H5Fopen | +
| Dataset Interface: | +#H5Dopen | +
| File Interface: | +h5fopen_f | +
| Dataset Interface: | +h5dopen_f | +
| File Interface: | +H5.H5Fopen | +
| Dataset Interface: | +H5.H5Dopen | +
-
+
-
+Python: Add
"import h5py / import numpy"+
+ -
+C: Add
"#include hdf5.h"+
+ -
+FORTRAN: Add
"USE HDF5"and call h5open_f and h5close_f to initialize and close the HDF5 FORTRAN interface +
+ -
+Java: Add
"import hdf.hdf5lib.H5; + import hdf.hdf5lib.HDF5Constants;"+
+
+ +An HDF5 file is a binary file containing scientific data and supporting metadata. +\section secLBFileCreate HDF5 File Access +To create an HDF5 file, an application must specify not only a file name, but a file access mode, +a file creation property list, and a file access property list. These terms are described below: +
-
+
- File access mode:
+When creating a file, the file access mode specifies the action to take if the file already exists: +-
+
- #H5F_ACC_TRUNC specifies that if the file already exists, the current contents will be deleted so +that the application can rewrite the file with new data. + +
- #H5F_ACC_EXCL specifies that the open will fail if the file already exists. If the file does not +already exist, the file access parameter is ignored. + +
+Note that there are two different access modes for opening existing files: +-
+
- #H5F_ACC_RDONLY specifies that the application has read access but will not be allowed to write any data. + +
- #H5F_ACC_RDWR specifies that the application has read and write access. + +
+ - File creation property list:
The file creation property list is used to +control the file metadata. File metadata contains information about the size of the user-block*, +the size of various file data structures used by the HDF5 library, etc. In this tutorial, the +default file creation property list, #H5P_DEFAULT, is used.
+ *The user-block is a fixed-length block of data located at the beginning of the file which is +ignored by the HDF5 library. The user-block may be used to store any data or information found +to be useful to applications. +
+ - File access property list:
The file access property list is used to +control different methods of performing I/O on files. It also can be used to control how a file +is closed (whether or not to delay the actual file close until all objects in a file are closed). +The default file access property list, #H5P_DEFAULT, is used in this tutorial. +
+
-
+
- Specify the file creation and access property lists, if necessary. +
- Create the file. +
- Close the file, and if necessary, close the property lists. +
hdf5.h contains definitions and declarations and must be included
+in any program that uses the HDF5 library.
++In FORTRAN: The module
HDF5 contains definitions and declarations and must be used in any
+program that uses the HDF5 library. Also note that #H5open MUST be called at the beginning of an HDF5 Fortran
+application (prior to any HDF5 calls) to initialize the library and variables. The #H5close call MUST be at
+the end of the HDF5 Fortran application.
+\li #H5Fcreate creates an HDF5 file and returns the file identifier.+For Fortran, the file creation property list and file access property list are optional. They can be omitted if the +default values are to be used.
+The root group is automatically created when a file is created. Every file has a root group and the path name of +the root group is always
/.
+\li #H5Fclose terminates access to an HDF5 file.+When an HDF5 file is no longer accessed by a program, #H5Fclose must be called to release the resources used by the file. +This call is mandatory.
+Note that if #H5Fclose is called for a file, but one or more objects within the file remain open, those objects will +remain accessible until they are individually closed. This can cause access problems for other users, if objects were +inadvertently left open. A File Access property controls how the file is closed. + +\subsection subsecLBFileExampleCont File Contents +The HDF Group has developed tools for examining the contents of HDF5 files. The tool used throughout the HDF5 tutorial +is the HDF5 dumper,
h5dump, which displays the file contents in human-readable form. The output of h5dump is an ASCII
+display formatted according to the HDF5 DDL grammar. This grammar is defined, using Backus-Naur Form, in the
+\ref DDLBNF110.
+
+To view the HDF5 file contents, simply type:
+\code
+h5dump | +\image html imgLBFile.gif + | +
file.h5, as generated by h5dump. The HDF5 file called file.h5
+contains a group called /, or the root group. (The file called filef.h5, created by the FORTRAN version of the example,
+has the same output except that the filename shown is filef.h5.)
+\code
+HDF5 "file.h5" {
+ GROUP "/" {
+ }
+ }
+\endcode
+
+\subsection subsecLBFileExampleDDL File Definition in DDL
+The simplified DDL file definition for creating an HDF5 file. For simplicity, a simplified DDL is used in this tutorial. A
+complete and more rigorous DDL can be found in the \ref DDLBNF110.
+
+The following symbol definitions are used in the DDL:
+\code
+ ::= defined as
+ + +A dataset is a multidimensional array of data elements, together with supporting metadata. To create +a dataset, the application program must specify the location at which to create the dataset, the +dataset name, the datatype and dataspace of the data array, and the property lists. + +\section secLBDsetCreateDtype Datatypes +A datatype is a collection of properties, all of which can be stored on disk, and which, when taken as +a whole, provide complete information for data conversion to or from that datatype. + +There are two categories of datatypes in HDF5: +
-
+
- Pre-defined: These datatypes are opened and closed by HDF5.
+Pre-defined datatypes can be atomic or composite: +- Atomic datatypes cannot be decomposed into smaller datatype units at the API level. For example: integer, float, reference, string. +
- Composite datatypes are aggregations of one or more datatypes. For example: array, variable length, enumeration, compound.
+ - Derived: These datatypes are created or derived from the pre-defined types.
+A simple example of creating a derived datatype is using the string datatype, H5T_C_S1, to create strings of more than one character:
+\code + hid_t strtype; // Datatype ID + herr_t status; + + strtype = H5Tcopy (H5T_C_S1); + status = H5Tset_size (strtype, 5); // create string of length 5 +\endcode +
+
| Datatype | +Description | +
|---|---|
| H5T_STD_I32LE | +Four-byte, little-endian, signed, two's complement integer | +
| H5T_STD_U16BE | +Two-byte, big-endian, unsigned integer | +
| H5T_IEEE_F32BE | +Four-byte, big-endian, IEEE floating point | +
| H5T_IEEE_F64LE | +Eight-byte, little-endian, IEEE floating point | +
| H5T_C_S1 | +One-byte, null-terminated string of eight-bit characters | +
| Native Datatype | +Corresponding C or FORTRAN Type | +
|---|---|
| C | +|
| H5T_NATIVE_INT | +int | +
| H5T_NATIVE_FLOAT | +float | +
| H5T_NATIVE_CHAR | +char | +
| H5T_NATIVE_DOUBLE | +double | +
| H5T_NATIVE_LDOUBLE | +long double | +
| Fortran | +|
| H5T_NATIVE_INTEGER | +integer | +
| H5T_NATIVE_REAL | +real | +
| H5T_NATIVE_DOUBLE | +double precision | +
| H5T_NATIVE_CHARACTER | +character | +
+When creating a dataset, HDF5 allows the user to specify how raw data is organized and/or compressed on +disk. This information is stored in a dataset creation property list and passed to the dataset interface. +The raw data on disk can be stored contiguously (in the same linear way that it is organized in memory), +partitioned into chunks, stored externally, etc. In this tutorial, we use the default dataset creation +property list (contiguous storage layout and no compression). For more information about dataset creation +property lists, see \ref sec_dataset in the \ref UG. +\li Link Creation Property List
+The link creation property list governs creation of the link(s) by which a new dataset is accessed and the +creation of any intermediate groups that may be missing. +\li Dataset Access Property List
+Dataset access property lists are properties that can be specified when accessing a dataset. + +\section secLBDsetCreateSteps Steps to Create a Dataset +To create an empty dataset (no data written) the following steps need to be taken: +
-
+
- Obtain the location identifier where the dataset is to be created. +
- Define or specify the dataset characteristics:
+
-
+
- Define a datatype or specify a pre-defined datatype. +
- Define a dataspace. +
- Specify the property list(s) or use the default. +
+ - Create the dataset. +
- Close the datatype, the dataspace, and the property list(s) if necessary. +
- Close the dataset. +
dset.h5
+in the C version (dsetf.h5 in Fortran), defines the dataset dataspace, creates a
+dataset which is a 4x6 integer array, and then closes the dataspace, the dataset, and the file.
+
+For details on compiling an HDF5 application: [ \ref LBCompiling ]
+
+\subsection subsecLBDsetCreateProgRem Remarks
+#H5Screate_simple creates a new simple dataspace and returns a dataspace identifier.
+#H5Sclose releases and terminates access to a dataspace.
+
+C
+\code
+ dataspace_id = H5Screate_simple (rank, dims, maxdims);
+ status = H5Sclose (dataspace_id );
+\endcode
+
+FORTRAN
+\code
+ CALL h5screate_simple_f (rank, dims, dataspace_id, hdferr, maxdims=max_dims)
+ or
+ CALL h5screate_simple_f (rank, dims, dataspace_id, hdferr)
+
+ CALL h5sclose_f (dataspace_id, hdferr)
+\endcode
+
+#H5Dcreate creates an empty dataset at the specified location and returns a dataset identifier.
+#H5Dclose closes the dataset and releases the resource used by the dataset. This call is mandatory.
+
+C
+\code
+ dataset_id = H5Dcreate(file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
+ status = H5Dclose (dataset_id);
+\endcode
+
+FORTRAN
+\code
+ CALL h5dcreate_f (loc_id, name, type_id, dataspace_id, dset_id, hdferr)
+ CALL h5dclose_f (dset_id, hdferr)
+\endcode
+
+Note that if using the pre-defined datatypes in FORTRAN, then a call must be made to initialize and terminate access to the pre-defined datatypes:
+\code
+ CALL h5open_f (hdferr)
+ CALL h5close_f (hdferr)
+\endcode
+
+H5open must be called before any HDF5 library subroutine calls are made;
+H5close must be called after the final HDF5 library subroutine call.
+
+See the programming example for an illustration of the use of these calls.
+
+\subsection subsecLBDsetCreateContent File Contents
+The contents of the file dset.h5 (dsetf.h5 for FORTRAN) are shown below:
+| +\image html imgLBDsetCreate.gif + | +
| dset.h5 in DDL | +dsetf.h5 in DDL | +
|---|---|
| +\code +HDF5 "dset.h5" { +GROUP "/" { + DATASET "dset" { + DATATYPE { H5T_STD_I32BE } + DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) } + DATA { + 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0 + } + } +} +} +\endcode + | ++\code +HDF5 "dsetf.h5" { +GROUP "/" { + DATASET "dset" { + DATATYPE { H5T_STD_I32BE } + DATASPACE { SIMPLE ( 6, 4 ) / ( 6, 4 ) } + DATA { + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0 + } + } +} +} +\endcode + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDsetRW Reading From and Writing To a Dataset +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDsetRW Dataset I/O Operation +During a dataset I/O operation, the library transfers raw data between memory and the file. The data in memory +can have a datatype different from that of the file and can also be of a different size (i.e., the data in +memory is a subset of the dataset elements, or vice versa). Therefore, to perform read or write operations, +the application program must specify: +\li The dataset +\li The dataset's datatype in memory +\li The dataset's dataspace in memory +\li The dataset's dataspace in the file +\li The dataset transfer property list
+
-
+
- (The dataset transfer property list controls various aspects of the I/O operations, such as the number +of processes participating in a collective I/O request or hints to the library to control caching of raw +data. In this tutorial, we use the default dataset transfer property list.) +
-
+
- Obtain the dataset identifier. +
- Specify the memory datatype. +
- Specify the memory dataspace. +
- Specify the file dataspace. +
- Specify the transfer properties. +
- Perform the desired operation on the dataset. +
- Close the dataset. +
- Close the dataspace, datatype, and property list if necessary. +
/dset, writes the dataset to the file, then reads
+the dataset back. It then closes the dataset and file.
+
+Note that #H5S_ALL is passed in for both the memory and file dataspace parameters in the read and write calls.
+This indicates that the entire dataspace of the dataset will be read or written to. #H5S_ALL by itself does not
+necessarily have this meaning. See the \ref RM entry for #H5Dread or #H5Dwrite for more information on using #H5S_ALL.
+
+For details on compiling an HDF5 application:
+[ \ref LBCompiling ]
+
+\subsection secLBDsetRWExRem Remarks
+#H5Fopen opens an existing file and returns a file identifier.
+
+#H5Dopen opens an existing dataset with the specified name and location.
+
+#H5Dwrite writes raw data from an application buffer to the specified dataset, converting from the datatype and
+dataspace of the dataset in memory to the datatype and dataspace of the dataset in the file. Specifying #H5S_ALL
+for both the memory and file dataspaces indicates that the entire dataspace of the dataset is to be written to.
+#H5S_ALL by itself does not necessarily have this meaning. See the \ref RM entry for #H5Dwrite for more information
+on using #H5S_ALL.
+
+#H5Dread reads raw data from the specified dataset to an application buffer, converting from the file datatype and
+dataspace to the memory datatype and dataspace. Specifying #H5S_ALL for both the memory and file dataspaces
+indicates that the entire dataspace of the dataset is to be read. #H5S_ALL by itself does not necessarily have
+this meaning. See the \ref RM entry for #H5Dread for more information on using #H5S_ALL.
+
+\subsection secLBDsetRWExCont File Contents
+
+Shown below is the contents of dset.h5 (created by the C program).
+
+dset.h5 in DDL
+\code
+ HDF5 "dset.h5" {
+ GROUP "/" {
+ DATASET "dset" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) }
+ DATA {
+ 1, 2, 3, 4, 5, 6,
+ 7, 8, 9, 10, 11, 12,
+ 13, 14, 15, 16, 17, 18,
+ 19, 20, 21, 22, 23, 24
+ }
+ }
+ }
+ }
+\endcode
+
+Shown below is the contents of dsetf.h5 (created by the FORTRAN program).
+
+dsetf.h5 in DDL
+\code
+ HDF5 "dsetf.h5" {
+ GROUP "/" {
+ DATASET "dset" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 6, 4 ) / ( 6, 4 ) }
+ DATA {
+ 1, 7, 13, 19,
+ 2, 8, 14, 20,
+ 3, 9, 15, 21,
+ 4, 10, 16, 22,
+ 5, 11, 17, 23,
+ 6, 12, 18, 24
+ }
+ }
+ }
+ }
+\endcode
+
++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBAttrCreate Creating an Attribute +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +Attributes are small datasets that can be used to describe the nature and/or the intended usage of +the object they are attached to. In this section, we show how to create, read, and write an attribute. + +\section secLBAttrCreate Creating an attribute +Creating an attribute is similar to creating a dataset. To create an attribute, the application must +specify the object which the attribute is attached to, the datatype and dataspace of the attribute +data, and the attribute creation property list. + +The steps to create an attribute are as follows: +
-
+
- Obtain the object identifier that the attribute is to be attached to. +
- Define the characteristics of the attribute and specify the attribute creation property list.
+
-
+
- Define the datatype. +
- Define the dataspace. +
- Specify the attribute creation property list. +
+ - Create the attribute. +
- Close the attribute and datatype, dataspace, and attribute creation property list, if necessary. +
-
+
- Obtain the attribute identifier. +
- Specify the attribute's memory datatype. +
- Perform the desired operation. +
- Close the memory datatype if necessary. +
dset.h5
+in C (dsetf.h5 in FORTRAN), obtains the identifier of the dataset /dset, defines
+the attribute's dataspace, creates the dataset attribute, writes the attribute, and then closes the attribute's
+dataspace, attribute, dataset, and file.
+
+For details on compiling an HDF5 application:
+[ \ref LBCompiling ]
+
+\subsection secLBAttrCreateRWExRem Remarks
+#H5Acreate creates an attribute which is attached to the object specified by the first parameter, and returns an identifier.
+
+#H5Awrite writes the entire attribute, and returns the status of the write.
+
+When an attribute is no longer accessed by a program, #H5Aclose must be called to release the attribute from use.
+An #H5Aclose/h5aclose_f call is mandatory.
+
+\subsection secLBAttrCreateRWExCont File Contents
+
+Shown below is the contents and the attribute definition of dset.h5 (created by the C program).
+
+dset.h5 in DDL
+\code
+HDF5 "dset.h5" {
+GROUP "/" {
+DATASET "dset" {
+DATATYPE { H5T_STD_I32BE }
+DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) }
+DATA {
+ 1, 2, 3, 4, 5, 6,
+ 7, 8, 9, 10, 11, 12,
+ 13, 14, 15, 16, 17, 18,
+ 19, 20, 21, 22, 23, 24
+}
+ATTRIBUTE "attr" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 2 ) / ( 2 ) }
+ DATA {
+ 100, 200
+ }
+}
+}
+}
+}
+\endcode
+
+Shown below is the contents and the attribute definition of dsetf.h5 (created by the FORTRAN program).
+
+dsetf.h5 in DDL
+\code
+HDF5 "dsetf.h5" {
+GROUP "/" {
+DATASET "dset" {
+DATATYPE { H5T_STD_I32BE }
+DATASPACE { SIMPLE ( 6, 4 ) / ( 6, 4 ) }
+DATA {
+ 1, 7, 13, 19,
+ 2, 8, 14, 20,
+ 3, 9, 15, 21,
+ 4, 10, 16, 22,
+ 5, 11, 17, 23,
+ 6, 12, 18, 24
+}
+ATTRIBUTE "attr" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 2 ) / ( 2 ) }
+ DATA {
+ 100, 200
+ }
+}
+}
+}
+}
+\endcode
+
+\subsection secLBAttrCreateRWExDDL Attribute Definition in DDL
+
+HDF5 Attribute Definition
+\code
++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnBasics2.dox b/doxygen/dox/LearnBasics2.dox new file mode 100644 index 0000000..ffcb971 --- /dev/null +++ b/doxygen/dox/LearnBasics2.dox @@ -0,0 +1,1159 @@ +/** @page LBGrpCreate Creating an Group +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBGrpCreate Creating an group +An HDF5 group is a structure containing zero or more HDF5 objects. The two primary HDF5 objects are groups and datasets. To create a group, the calling program must: +
-
+
- Obtain the location identifier where the group is to be created. +
- Create the group. +
- Close the group. +
group.h5 in C
+(groupf.h5 for FORTRAN), creates a group called MyGroup in the root group, and then closes the group and file.
+
+For details on compiling an HDF5 application:
+[ \ref LBCompiling ]
+
+\subsection secLBGrpCreateRWExCont File Contents
+
+Shown below is the contents and the definition of the group of group.h5 (created by the C program).
+(The FORTRAN program creates the HDF5 file groupf.h5 and the resulting DDL shows the filename
+groupf.h5 in the first line.)
+| +\image html imggrpcreate.gif + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBGrpCreateNames Creating Groups using Absolute and Relative Names +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +Recall that to create an HDF5 object, we have to specify the location where the object is to be created. +This location is determined by the identifier of an HDF5 object and the name of the object to be created. +The name of the created object can be either an absolute name or a name relative to the specified identifier. +In the previous example, we used the file identifier and the absolute name
/MyGroup to create a group.
+
+In this section, we discuss HDF5 names and show how to use absolute and relative names.
+
+\section secLBGrpCreateNames Names
+HDF5 object names are a slash-separated list of components. There are few restrictions on names: component
+names may be any length except zero and may contain any character except slash (/) and the null terminator.
+A full name may be composed of any number of component names separated by slashes, with any of the component
+names being the special name . (a dot or period). A name which begins with a slash is an absolute name which
+is accessed beginning with the root group of the file; all other names are relative names and and the named
+object is accessed beginning with the specified group. A special case is the name / (or equivalent) which
+refers to the root group.
+
+Functions which operate on names generally take a location identifier, which can be either a file identifier
+or a group identifier, and perform the lookup with respect to that location. Several possibilities are
+described in the following table:
+
+| Location Type | +Object Name | +Description | +
|---|---|---|
| File identifier | +/foo/bar | +The object bar in group foo in the root group. | +
| Group identifier | +/foo/bar | +The object bar in group foo in the root group of the file containing the specified group. +In other words, the group identifier's only purpose is to specify a file. | +
| File identifier | +/ | +The root group of the specified file. | +
| Group identifier | +/ | +The root group of the file containing the specified group. | +
| Group identifier | +foo/bar | +The object bar in group foo in the specified group. | +
| File identifier | +. | +The root group of the file. | +
| Group identifier | +. | +The specified group. | +
| Other identifier | +. | +The specified object. | +
MyGroup in the root group of the specified file.
+
+The second #H5Gcreate/h5gcreate_f creates the group Group_A in the group MyGroup in the root group of the specified
+file. Note that the parent group (MyGroup) already exists.
+
+The third #H5Gcreate/h5gcreate_f creates the group Group_B in the specified group.
+
+\subsection secLBGrpCreateNamesExCont File Contents
+
+Shown below is the contents and the definition of the group of groups.h5 (created by the C program).
+(The FORTRAN program creates the HDF5 file groupsf.h5 and the resulting DDL shows the filename
+groupsf.h5 in the first line.)
+| +\image html imggrps.gif + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBGrpDset Creating Datasets in Groups +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBGrpDset Datasets in Groups +We have shown how to create groups, datasets, and attributes. In this section, we show how to create +datasets in groups. Recall that #H5Dcreate creates a dataset at the location specified by a location +identifier and a name. Similar to #H5Gcreate, the location identifier can be a file identifier or a +group identifier and the name can be relative or absolute. The location identifier and the name +together determine the location where the dataset is to be created. If the location identifier and +name refer to a group, then the dataset is created in that group. + +\section secLBGrpDsetEx Programming Example + +\subsection secLBGrpDsetExDesc Description +See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. + +The example shows how to create a dataset in a particular group. It opens the file created in the previous example and creates two datasets: + +For details on compiling an HDF5 application: +[ \ref LBCompiling ] + +\subsection secLBGrpDsetExCont File Contents + +Shown below is the contents and the definition of the group of
groups.h5 (created by the C program).
+(The FORTRAN program creates the HDF5 file groupsf.h5 and the resulting DDL shows the filename
+groupsf.h5 in the first line.)
+| +\image html imggrpdsets.gif + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDsetSubRW Reading From or Writing To a Subset of a Dataset +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDsetSubRW Dataset Subsets +There are two ways that you can select a subset in an HDF5 dataset and read or write to it: +
- +Hyperslab Selection: The #H5Sselect_hyperslab call selects a logically contiguous +collection of points in a dataspace, or a regular pattern of points or blocks in a dataspace. +
- +Element Selection: The #H5Sselect_elements call selects elements in an array. +
| +\image html LBDsetSubRWProg.png + | +
| +\image html imgLBDsetSubRW11.png + | +
| +\image html imgLBDsetSubRW12.png + | +
| +\image html imgLBDsetSubRW31.png + | +
| +\image html imgLBDsetSubRW32.png + | +
| +\image html imgLBDsetSubRW33.png + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDatatypes Datatype Basics +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDtype What is a Datatype? +A datatype is a collection of datatype properties which provide complete information for data conversion to or from that datatype. + +Datatypes in HDF5 can be grouped as follows: +\li Pre-Defined Datatypes: These are datatypes that are created by HDF5. They are actually opened +(and closed) by HDF5, and can have a different value from one HDF5 session to the next. +\li Derived Datatypes: These are datatypes that are created or derived from the pre-defined datatypes. +Although created from pre-defined types, they represent a category unto themselves. An example of a commonly used derived +datatype is a string of more than one character. + +\section secLBDtypePre Pre-defined Datatypes +The properties of pre-defined datatypes are: +\li Pre-defined datatypes are opened and closed by HDF5. +\li A pre-defined datatype is a handle and is NOT PERSISTENT. Its value can be different from one HDF5 session to the next. +\li Pre-defined datatypes are Read-Only. +\li As mentioned, other datatypes can be derived from pre-defined datatypes. + +There are two types of pre-defined datatypes, standard (file) and native. + +
Standard
+A standard (or file) datatype can be: +-
+
- Atomic: A datatype which cannot be decomposed into smaller datatype units at the API level.
+The atomic datatypes are:
+
-
+
- integer +
- float +
- string (1-character) +
- date and time +
- bitfield +
- reference +
- opaque +
+ - Composite: An aggregation of one or more datatypes.
+Composite datatypes include:
+
-
+
- array +
- variable length +
- enumeration +
- compound datatypes +
+
| Notes | +
|---|
| +\li Standard pre-defined datatypes are the SAME on all platforms. +\li They are the datatypes that you see in an HDF5 file. +\li They are typically used when creating a dataset. + | +
Native
+Native pre-defined datatypes are used for memory operations, such as reading and writing. They are +NOT THE SAME on different platforms. They are similar to C type names, and are aliased to the +appropriate HDF5 standard pre-defined datatype for a given platform. + +For example, when on an Intel based PC, #H5T_NATIVE_INT is aliased to the standard pre-defined type, +#H5T_STD_I32LE. On a MIPS machine, it is aliased to #H5T_STD_I32BE. +| Notes | +
|---|
| +\li Native datatypes are NOT THE SAME on all platforms. +\li Native datatypes simplify memory operations (read/write). The HDF5 library automatically converts as needed. +\li Native datatypes are NOT in an HDF5 File. The standard pre-defined datatype that a native datatype corresponds +to is what you will see in the file. + | +
Pre-Defined
+The following table shows the native types and the standard pre-defined datatypes they correspond +to. (Keep in mind that HDF5 can convert between datatypes, so you can specify a buffer of a larger +type for a dataset of a given type. For example, you can read a dataset that has a short datatype +into a long integer buffer.) + +| C Type | +HDF5 Memory Type | +HDF5 File Type* | +
|---|---|---|
| Integer | +||
| int | +#H5T_NATIVE_INT | +#H5T_STD_I32BE or #H5T_STD_I32LE | +
| short | +#H5T_NATIVE_SHORT | +#H5T_STD_I16BE or #H5T_STD_I16LE | +
| long | +#H5T_NATIVE_LONG | +#H5T_STD_I32BE, #H5T_STD_I32LE, + #H5T_STD_I64BE or #H5T_STD_I64LE | +
| long long | +#H5T_NATIVE_LLONG | +#H5T_STD_I64BE or #H5T_STD_I64LE | +
| unsigned int | +#H5T_NATIVE_UINT | +#H5T_STD_U32BE or #H5T_STD_U32LE | +
| unsigned short | +#H5T_NATIVE_USHORT | +#H5T_STD_U16BE or #H5T_STD_U16LE | +
| unsigned long | +#H5T_NATIVE_ULONG | +#H5T_STD_U32BE, #H5T_STD_U32LE, + #H5T_STD_U64BE or #H5T_STD_U64LE | +
| unsigned long long | +#H5T_NATIVE_ULLONG | +#H5T_STD_U64BE or #H5T_STD_U64LE | +
| Float | +||
| float | +#H5T_NATIVE_FLOAT | +#H5T_IEEE_F32BE or #H5T_IEEE_F32LE | +
| double | +#H5T_NATIVE_DOUBLE | +#H5T_IEEE_F64BE or #H5T_IEEE_F64LE | +
| F90 Type | +HDF5 Memory Type | +HDF5 File Type* | +
|---|---|---|
| integer | +H5T_NATIVE_INTEGER | +#H5T_STD_I32BE(8,16) or #H5T_STD_I32LE(8,16) | +
| real | +H5T_NATIVE_REAL | +#H5T_IEEE_F32BE or #H5T_IEEE_F32LE | +
| double-precision | +#H5T_NATIVE_DOUBLE | +#H5T_IEEE_F64BE or #H5T_IEEE_F64LE | +
| * Note that the HDF5 File Types listed are those that are most commonly created. + The file type created depends on the compiler switches and platforms being + used. For example, on the Cray an integer is 64-bit, and using #H5T_NATIVE_INT (C) + or H5T_NATIVE_INTEGER (F90) would result in an #H5T_STD_I64BE file type. | +
-
+
- Make a copy of the pre-defined datatype: +\code + tid = H5Tcopy (H5T_STD_I32BE); +\endcode + +
- Change the datatype. +
Creating
+The function #H5Tarray_create creates a new array datatype object. Parameters specify +\li the base datatype of each element of the array, +\li the rank of the array, i.e., the number of dimensions, +\li the size of each dimension, and +\li the dimension permutation of the array, i.e., whether the elements of the array are listed in C or FORTRAN order. + +Working with existing array datatypes
+When working with existing arrays, one must first determine the the rank, or number of dimensions, of the array. + +The function #H5Tget_array_dims returns the rank of a specified array datatype. + +In many instances, one needs further information. The function #H5Tget_array_dims retrieves the +permutation of the array and the size of each dimension. + +\subsection subsecLBDtypeSpecCmpd Compound + +\subsubsection subsubsecLBDtypeSpecCmpdProp Properties of compound datatypes +A compound datatype is similar to a struct in C or a common block in Fortran. It is a collection of +one or more atomic types or small arrays of such types. To create and use of a compound datatype +you need to refer to various properties of the data compound datatype: +\li It is of class compound. +\li It has a fixed total size, in bytes. +\li It consists of zero or more members (defined in any order) with unique names and which occupy non-overlapping regions within the datum. +\li Each member has its own datatype. +\li Each member is referenced by an index number between zero and N-1, where N is the number of members in the compound datatype. +\li Each member has a name which is unique among its siblings in a compound datatype. +\li Each member has a fixed byte offset, which is the first byte (smallest byte address) of that member in a compound datatype. +\li Each member can be a small array of up to four dimensions. + +Properties of members of a compound datatype are defined when the member is added to the compound type and cannot be subsequently modified. + +\subsubsection subsubsecLBDtypeSpecCmpdDef Defining compound datatypes +Compound datatypes must be built out of other datatypes. First, one creates an empty compound +datatype and specifies its total size. Then members are added to the compound datatype in any order. + +Member names. Each member must have a descriptive name, which is the key used to uniquely identify +the member within the compound datatype. A member name in an HDF5 datatype does not necessarily +have to be the same as the name of the corresponding member in the C struct in memory, although +this is often the case. Nor does one need to define all members of the C struct in the HDF5 +compound datatype (or vice versa). + +Offsets. Usually a C struct will be defined to hold a data point in memory, and the offsets of the +members in memory will be the offsets of the struct members from the beginning of an instance of the +struct. The library defines the macro to compute the offset of a member within a struct: +\code + HOFFSET(s,m) +\endcode +This macro computes the offset of member m within a struct variable s. + +Here is an example in which a compound datatype is created to describe complex numbers whose type +is defined by the complex_t struct. +\code +typedef struct { + double re; /*real part */ + double im; /*imaginary part */ +} complex_t; + +complex_t tmp; /*used only to compute offsets */ +hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp); +H5Tinsert (complex_id, "real", HOFFSET(tmp,re), H5T_NATIVE_DOUBLE); +H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im), H5T_NATIVE_DOUBLE); +\endcode + +\subsection subsecLBDtypeSpecRef Reference +There are two types of Reference datatypes in HDF5: +\li \ref subsubsecLBDtypeSpecRefObj +\li \ref subsubsecLBDtypeSpecRefDset + +\subsubsection subsubsecLBDtypeSpecRefObj Reference to objects +In HDF5, objects (i.e. groups, datasets, and named datatypes) are usually accessed by name. +There is another way to access stored objects -- by reference. + +An object reference is based on the relative file address of the object header in the file +and is constant for the life of the object. Once a reference to an object is created and +stored in a dataset in the file, it can be used to dereference the object it points to. +References are handy for creating a file index or for grouping related objects by storing +references to them in one dataset. + +Creating and storing references to objects
+The following steps are involved in creating and storing file references to objects: +-
+
- Create the objects or open them if they already exist in the file. +
- Create a dataset to store the objects' references, by specifying #H5T_STD_REF_OBJ as the datatype +
- Create and store references to the objects in a buffer, using #H5Rcreate. +
- Write a buffer with the references to the dataset, using #H5Dwrite with the #H5T_STD_REF_OBJ datatype. +
Reading references and accessing objects using references
+The following steps are involved: +-
+
- Open the dataset with the references and read them. The #H5T_STD_REF_OBJ datatype must be used to describe the memory datatype. +
- Use the read reference to obtain the identifier of the object the reference points to using #H5Rdereference. +
- Open the dereferenced object and perform the desired operations. +
- Close all objects when the task is complete. +
Creating and storing references to dataset regions
+The following steps are involved in creating and storing references to a dataset region: +\li Create a dataset to store the dataset region (selection), by passing in #H5T_STD_REF_DSETREG for the datatype when calling #H5Dcreate. +\li Create selection(s) in existing dataset(s) using #H5Sselect_hyperslab and/or #H5Sselect_elements. +\li Create reference(s) to the selection(s) using #H5Rcreate and store them in a buffer. +\li Write the references to the dataset regions in the file. +\li Close all objects. + +Reading references to dataset regions
+The following steps are involved in reading references to dataset regions and referenced dataset regions (selections). +-
+
- Open and read the dataset containing references to the dataset regions. +The datatype #H5T_STD_REF_DSETREG must be used during read operation. +
- Use #H5Rdereference to obtain the dataset identifier from the read dataset region reference. + OR + Use #H5Rget_region to obtain the dataspace identifier for the dataset containing the selection from the read dataset region reference. + +
- With the dataspace identifier, the \ref H5S interface functions, H5Sget_select_*, +can be used to obtain information about the selection. +
- Close all objects when they are no longer needed. +
| Function | +Description | +
|---|---|
| #H5Sget_select_npoints | +Returns the number of elements in the hyperslab | +
| #H5Sget_select_hyper_nblocks | +Returns the number of blocks in the hyperslab | +
| #H5Sget_select_hyper_blocklist | +Returns the "lower left" and "upper right" coordinates of the blocks in the hyperslab selection | +
| #H5Sget_select_bounds | +Returns the coordinates of the "minimal" block containing a hyperslab selection | +
| #H5Sget_select_elem_npoints | +Returns the number of points in the element selection | +
| #H5Sget_select_elem_pointlist | +Returns the coordinates of points in the element selection | +
-
+
- Ragged arrays: Multi-dimensional ragged arrays can be implemented with the last (fastest changing) +dimension being ragged by using a VL datatype as the type of the element stored. (Or as a field in a compound datatype.) + +
- Fractal arrays: If a compound datatype has a VL field of another compound type with VL fields +(a nested VL datatype), this can be used to implement ragged arrays of ragged arrays, to whatever +nesting depth is required for the user. + +
- Polygon lists: A common storage requirement is to efficiently store arrays of polygons with +different numbers of vertices. VL datatypes can be used to efficiently and succinctly describe an +array of polygons with different numbers of vertices. + +
- Character strings: Perhaps the most common use of VL datatypes will be to store C-like VL character +strings in dataset elements or as attributes of objects. + +
- Indices: An array of VL object references could be used as an index to all the objects in a file
+which contain a particular sequence of dataset values. Perhaps an array something like the following:
+\code
+ Value1: Object1, Object3, Object9
+ Value2: Object0, Object12, Object14, Object21, Object22
+ Value3: Object2
+ Value4:
+ Value5: Object1, Object10, Object12 + . + . +\endcode +
+ - Object Tracking: An array of VL dataset region references can be used as a method of tracking
+objects or features appearing in a sequence of datasets. Perhaps an array of them would look like:
+\code
+ Feature1: Dataset1:Region, Dataset3:Region, Dataset9:Region
+ Feature2: Dataset0:Region, Dataset12:Region, Dataset14:Region,
+ Dataset21:Region, Dataset22:Region
+ Feature3: Dataset2:Region
+ Feature4:
+ Feature5: Dataset1:Region, Dataset10:Region, Dataset12:Region + . + . +\endcode +
+
Creation
+VL datatypes are created with the #H5Tvlen_create function as follows: +\code +type_id = H5Tvlen_create(hid_t base_type_id); +\endcode +The base datatype will be the datatype that the sequence is composed of, characters for character +strings, vertex coordinates for polygon lists, etc. The base datatype specified for the VL datatype +can be of any HDF5 datatype, including another VL datatype, a compound datatype, or an atomic datatype. + +Querying base datatype of VL datatype
+It may be necessary to know the base datatype of a VL datatype before memory is allocated, etc. +The base datatype is queried with the #H5Tget_super function, described in the \ref H5T documentation. + +Querying minimum memory required for VL information
+It order to predict the memory usage that #H5Dread may need to allocate to store VL data while +reading the data, the #H5Dvlen_get_buf_size function is provided: +\code +herr_t H5Dvlen_get_buf_size(hid_t dataset_id, hid_t type_id, hid_t space_id, hsize_t *size) +\endcode +This routine checks the number of bytes required to store the VL data from the dataset, using +the \em space_id for the selection in the dataset on disk and the \em type_id for the memory representation +of the VL data in memory. The *\em size value is modified according to how many bytes are required +to store the VL data in memory. + +Specifying how to manage memory for the VL datatype
+The memory management method is determined by dataset transfer properties passed into the +#H5Dread and #H5Dwrite functions with the dataset transfer property list. + +Default memory management is set by using #H5P_DEFAULT for the dataset transfer +property list identifier. If #H5P_DEFAULT is used with #H5Dread, the system \em malloc and \em free +calls will be used for allocating and freeing memory. In such a case, #H5P_DEFAULT should +also be passed as the property list identifier to #H5Dvlen_reclaim. + +The rest of this subsection is relevant only to those who choose not to use default memory management. + +The user can choose whether to use the system \em malloc and \em free calls or user-defined, or custom, +memory management functions. If user-defined memory management functions are to be used, the +memory allocation and free routines must be defined via #H5Pset_vlen_mem_manager(), as follows: +\code +herr_t H5Pset_vlen_mem_manager(hid_t plist_id, H5MM_allocate_t alloc, void *alloc_info, H5MM_free_t free, void *free_info) +\endcode +The \em alloc and \em free parameters identify the memory management routines to be used. If the user +has defined custom memory management routines, \em alloc and/or \em free should be set to make those +routine calls (i.e., the name of the routine is used as the value of the parameter); if the user +prefers to use the system's \em malloc and/or \em free, the \em alloc and \em free parameters, respectively, should be set to \em NULL + +The prototypes for the user-defined functions would appear as follows: +\code +typedef void *(*H5MM_allocate_t)(size_t size, void *info) ; typedef void (*H5MM_free_t)(void *mem, void *free_info) ; +\endcode +The \em alloc_info and \em free_info parameters can be used to pass along any required information to +the user's memory management routines. + +In summary, if the user has defined custom memory management routines, the name(s) of the routines +are passed in the \em alloc and \em free parameters and the custom routines' parameters are passed in the +\em alloc_info and \em free_info parameters. If the user wishes to use the system \em malloc and \em free functions, +the \em alloc and/or \em free parameters are set to \em NULL and the \em alloc_info and \em free_info parameters are ignored. + +Recovering memory from VL buffers read in
+The complex memory buffers created for a VL datatype may be reclaimed with the #H5Dvlen_reclaim +function call, as follows: +\code +herr_t H5Dvlen_reclaim(hid_t type_id, hid_t space_id, hid_t plist_id, void *buf); +\endcode + +The \em type_id must be the datatype stored in the buffer, \em space_id describes the selection for the +memory buffer to free the VL datatypes within, \em plist_id is the dataset transfer property list +which was used for the I/O transfer to create the buffer, and \em buf is the pointer to the buffer +to free the VL memory within. The VL structures (#hvl_t) in the user's buffer are modified to zero +out the VL information after it has been freed. + +If nested VL datatypes were used to create the buffer, this routine frees them from the bottom up, +releasing all the memory without creating memory leaks. + ++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnBasics3.dox b/doxygen/dox/LearnBasics3.dox new file mode 100644 index 0000000..2fe0f52 --- /dev/null +++ b/doxygen/dox/LearnBasics3.dox @@ -0,0 +1,1015 @@ +/** @page LBPropsList Property Lists Basics +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBPList What is a Property (or Property List)? +In HDF5, a property or property list is a characteristic or feature associated with an HDF5 object. +There are default properties which handle the most common needs. These default properties are +specified by passing in #H5P_DEFAULT for the Property List parameter of a function. Default properties +can be modified by use of the \ref H5P interface and function parameters. + +The \ref H5P API allows a user to take advantage of the more powerful features in HDF5. It typically +supports unusual cases when creating or accessing HDF5 objects. There is a programming model for +working with Property Lists in HDF5 (see below). + +For examples of modifying a property list, see these tutorial topics: +\li \see \ref LBDsetLayout +\li \see \ref LBExtDset +\li \see \ref LBComDset + +There are many Property Lists associated with creating and accessing objects in HDF5. See the +\ref H5P Interface documentation in the HDF5 \ref RM for a list of the different +properties associated with HDF5 interfaces. + +In summary: +\li Properties are features of HDF5 objects, that can be changed by use of the Property List API and function parameters. +\li Property lists provide a mechanism for adding functionality to HDF5 calls without increasing the number of arguments used for a given call. +\li The Property List API supports unusual cases when creating and accessing HDF5 objects. + +\section secLBPListProg Programming Model +Default properties are specified by simply passing in #H5P_DEFAULT (C) / H5P_DEFAULT_F (F90) for +the property list parameter in those functions for which properties can be changed. + +The programming model for changing a property list is as follows: +\li Create a copy or "instance" of the desired pre-defined property type, using the #H5Pcreate call. This +will return a property list identifier. Please see the \ref RM entry for #H5Pcreate, for a comprehensive +list of the property types. +\li With the property list identifier, modify the property, using the \ref H5P APIs. +\li Modify the object feature, by passing the property list identifier into the corresponding HDF5 object function. +\li Close the property list when done, using #H5Pclose. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDsetLayout Dataset Storage Layout +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDsetLayoutDesc Description of a Dataset + +\section secLBDsetLayout Dataset Storage Layout +The storage information, or storage layout, defines how the raw data values in the dataset are +physically stored on disk. There are three ways that a dataset can be stored: +\li contiguous +\li chunked +\li compact + +See the #H5Pset_layout/#H5Pget_layout APIs for details. + +\subsection subsecLBDsetLayoutCont Contiguous +If the storage layout is contiguous, then the raw data values will be stored physically adjacent +to each other in the HDF5 file (in one contiguous block). This is the default layout for a dataset. +In other words, if you do not explicitly change the storage layout for the dataset, then it will +be stored contiguously. +
| +\image html tutr-locons.png + | +
| +\image html tutr-lochk.png + | +
| +\image html tutr-lochks.png + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + + +@page LBExtDset Extendible Datasets +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBExtDsetCreate Creating an Extendible Dataset +An extendible dataset is one whose dimensions can grow. HDF5 allows you to define a dataset to have +certain initial dimensions, then to later increase the size of any of the initial dimensions. + +HDF5 requires you to use chunking to define extendible datasets. This makes it possible to extend +datasets efficiently without having to excessively reorganize storage. (To use chunking efficiently, +be sure to see the advanced topic, Chunking in HDF5.) + +The following operations are required in order to extend a dataset: +\li Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended. +\li Set dataset creation properties to enable chunking. +\li Create the dataset. +\li Extend the size of the dataset. + +\section secLBExtDsetProg Programming Example + +\subsection subsecLBExtDsetProgDesc Description +See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. + +The example shows how to create a 3 x 3 extendible dataset, write to that dataset, extend the dataset +to 10x3, and write to the dataset again. + +For details on compiling an HDF5 application: +[ \ref LBCompiling ] + +\subsection subsecLBExtDsetProgRem Remarks +\li An unlimited dimension dataspace is specified with the #H5Screate_simple call, by passing in +#H5S_UNLIMITED as an element of the maxdims array. +\li The #H5Pcreate call creates a new property as an instance of a property list class. For creating +an extendible array dataset, pass in #H5P_DATASET_CREATE for the property list class. +\li The #H5Pset_chunk call modifies a Dataset Creation Property List instance to store a chunked +layout dataset and sets the size of the chunks used. +\li To extend an unlimited dimension dataset use the the #H5Dset_extent call. Please be aware that +after this call, the dataset's dataspace must be refreshed with #H5Dget_space before more data can be accessed. +\li The #H5Pget_chunk call retrieves the size of chunks for the raw data of a chunked layout dataset. +\li Once there is no longer a need for a Property List instance, it should be closed with the #H5Pclose call. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBComDset Compressed Datasets +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBComDsetCreate Creating a Compressed Dataset +HDF5 requires you to use chunking to create a compressed dataset. (To use chunking efficiently, +be sure to see the advanced topic, Chunking in HDF5.) + +The following operations are required in order to create a compressed dataset: +\li Create a dataset creation property list. +\li Modify the dataset creation property list instance to enable chunking and to enable compression. +\li Create the dataset. +\li Close the dataset creation property list and dataset. + +For more information on compression, see the FAQ question on Using Compression in HDF5. + +\section secLBComDsetProg Programming Example + +\subsection subsecLBComDsetProgDesc Description +See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. + +The example creates a chunked and ZLIB compressed dataset. It also includes comments for what needs +to be done to create an SZIP compressed dataset. The example then reopens the dataset, prints the +filter information, and reads the dataset. + +For details on compiling an HDF5 application: +[ \ref LBCompiling ] + +\subsection subsecLBComDsetProgRem Remarks +\li The #H5Pset_chunk call modifies a Dataset Creation Property List instance to store a chunked layout +dataset and sets the size of the chunks used. +\li The #H5Pset_deflate call modifies the Dataset Creation Property List instance to use ZLIB or DEFLATE +compression. The #H5Pset_szip call modifies it to use SZIP compression. There are different compression +parameters required for each compression method. +\li SZIP compression can only be used with atomic datatypes that are integer, float, or char. It cannot be +applied to compound, array, variable-length, enumerations, or other user-defined datatypes. The call +to #H5Dcreate will fail if attempting to create an SZIP compressed dataset with a non-allowed datatype. +The conflict can only be detected when the property list is used. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBContents Discovering the Contents of an HDF5 File +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBContents Discovering what is in an HDF5 file +HDFView and h5dump are standalone tools which cannot be called within an application, and using +#H5Dopen and #H5Dread require that you know the name of the HDF5 dataset. How would an application +that has no prior knowledge of an HDF5 file be able to determine or discover the contents of it, +much like HDFView and h5dump? + +The answer is that there are ways to discover the contents of an HDF5 file, by using the +\ref H5G, \ref H5L and \ref H5O APIs: +\li The \ref H5G interface (covered earlier) consists of routines for working with groups. A group is +a structure that can be used to organize zero or more HDF5 objects, not unlike a Unix directory. +\li The \ref H5L interface consists of link routines. A link is a path between groups. The \ref H5L interface +allows objects to be accessed by use of these links. +\li The \ref H5O interface consists of routines for working with objects. Datasets, groups, and committed +datatypes are all objects in HDF5. + +Interface routines that simplify the process: +\li #H5Literate traverses the links in a specified group, in the order of the specified index, using a +user-defined callback routine. (A callback function is one that will be called when a certain condition +is met, at a certain point in the future.) +\li #H5Ovisit / #H5Lvisit recursively visit all objects/links accessible from a specified object/group. + + +\section secLBContentsProg Programming Example + +\subsection subsecLBContentsProgUsing Using #H5Literate, #H5Lvisit and #H5Ovisit +For example code, see the \ref HDF5Examples page. +Specifically look at the Examples by API. +There are examples for different languages, where examples of using #H5Literate and #H5Ovisit/#H5Lvisit are included. + +The h5ex_g_traverse example traverses a file using H5Literate: +\li C: h5ex_g_traverse.c +\li F90: h5ex_g_traverse_F03.f90 + +The h5ex_g_visit example traverses a file using H5Ovisit and H5Lvisit: +\li C: h5ex_g_visit.c +\li F90: h5ex_g_visit_F03.f90 + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBQuiz Learning the basics QUIZ +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\ref LBFileOrg +
-
+
- Name and describe the two primary objects that can be stored in an HDF5 file. + +
- What is an attribute? + +
- Give the path name for an object called
harrythat is a member of a group calleddick, which, in turn, is a member of the root group. +
+
-
+
- Describe the purpose of each of the following HDF5 APIs: +\code + H5A, H5D, H5E, H5F, H5G, H5T, H5Z +\endcode + +
-
+
- What two HDF5 routines must be called to create an HDF5 file? + +
- What include file must be included in any file that uses the HDF5 library? + +
- An HDF5 file is never completely empty because as soon as it is created, it automatically contains a certain primary object. What is that object? + +
-
+
- Name and describe two major datatype categories. + +
- List the HDF5 atomic datatypes. Give an example of a predefined datatype. How would you create a string dataset? + +
- What does the dataspace describe? What are the major characteristics of the simple dataspace? + +
- What information needs to be passed to the #H5Dcreate function, i.e., what information is needed to describe a dataset at creation time? + +
-
+
- What are six pieces of information which need to be specified for reading and writing a dataset? + +
- Why are both the memory dataspace and file dataspace needed for read/write operations, while only the memory datatype is required? + +
- In Figure 6.1, what does this line mean? +\code +DATASPACE { SIMPLE (4 , 6 ) / ( 4 , 6 ) } +\endcode + +
-
+
- What is an attribute? + +
- Can partial I/O operations be performed on attributes? + +
-
+
- What are the two primary objects that can be included in a group? + +
-
+
- Group names can be specified in two ways. What are these two types of group names? + +
- You have a dataset named
mooin the groupboo, which is in the groupfoo, which, in turn, +is in therootgroup. How would you specify an absolute name to access this dataset? +
+
-
+
- Describe a way to access the dataset moo described in the previous section +(question 2) using a relative name. Describe a way to access the same dataset using an absolute name. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBQuizAnswers Learning the basics QUIZ with Answers +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\ref LBFileOrg +
-
+
- Name and describe the two primary objects that can be stored in an HDF5 file.
+
+
++ +Answer + +Group: A grouping structure containing zero or more HDF5 objects, together with supporting metadata. +
+Dataset: A multidimensional array of data elements, together with supporting metadata. +
+ - What is an attribute?
+
+
++ +Answer + +An HDF5 attribute is a user-defined HDF5 structure that provides extra information about an HDF5 object. + +
+ - Give the path name for an object called
harrythat is a member of a group calleddick, which, in turn, is a member of the root group. ++
++ +Answer + +/dick/harry + +
+
-
+
- Describe the purpose of each of the following HDF5 APIs:
+\code
+ H5A, H5D, H5E, H5F, H5G, H5T, H5Z
+\endcode
+
+
++ +Answer + +H5A: Attribute access and manipulation routines + +
+H5D: Dataset access and manipulation routines +
+H5E: Error handling routines H5F: File access routines +
+H5G: Routines for creating and operating on groups +
+H5T: Routines for creating and manipulating the datatypes of dataset elements +
+H5Z: Data compression routines +
+
-
+
- What two HDF5 routines must be called to create an HDF5 file?
+
+
++ +Answer + +#H5Fcreate and #H5Fclose. + +
+ - What include file must be included in any file that uses the HDF5 library?
+
+
++ +Answer + +hdf5.h must be included because it contains definitions and declarations used by the library. + +
+ - An HDF5 file is never completely empty because as soon as it is created, it automatically contains a certain primary object. What is that object?
+
+
++ +Answer + +The root group. + +
+
-
+
- Name and describe two major datatype categories.
+
+
++ +Answer + +Atomic datatype: An atomic datatype cannot be decomposed into smaller units at the API level. + +
+Compound datatype: A compound datatype is a collection of atomic and compound datatypes, or small arrays of such types. +
+ - List the HDF5 atomic datatypes. Give an example of a predefined datatype. How would you create a string dataset?
+
+
++ +Answer + +There are six HDF5 atomic datatypes: integer, floating point, date and time, character string, bit field, and opaque. + +
+Examples of predefined datatypes include the following:
+\li #H5T_IEEE_F32LE - 4-byte little-endian, IEEE floating point +\li #H5T_NATIVE_INT - native integer + +You would create a string dataset with the #H5T_C_S1 datatype, and set the size of the string with the #H5Tset_size call. +
+ - What does the dataspace describe? What are the major characteristics of the simple dataspace?
+
+
++ +Answer + +The dataspace describes the dimensionality of the dataset. A simple dataspace is characterized by its rank and dimension sizes. + +
+ - What information needs to be passed to the #H5Dcreate function, i.e., what information is needed to describe a dataset at creation time?
+
+
++ +Answer + +The dataset location, name, dataspace, datatype, and dataset creation property list. + +
+
-
+
- What are six pieces of information which need to be specified for reading and writing a dataset?
+
+
++ +Answer + +The dataset identifier, the dataset's datatype and dataspace in memory, the dataspace in the file, +the dataset transfer property list, and a data buffer. + +
+ - Why are both the memory dataspace and file dataspace needed for read/write operations, while only the memory datatype is required?
+
+
++ +Answer + +A dataset's file datatype is not required for a read/write operation because the file datatype is specified +when the dataset is created and cannot be changed. Both file and memory dataspaces are required for dataset +subsetting and for performing partial I/O operations. + +
+ - In Figure 6.1, what does this line mean?
+\code
+DATASPACE { SIMPLE (4 , 6 ) / ( 4 , 6 ) }
+\endcode
+
+
++ +Answer + +It means that the dataset dset has a simple dataspace with the current dimensions (4,6) and the maximum size of the dimensions (4,6). + +
+
-
+
- What is an attribute?
+
+
++ +Answer + +An attribute is a dataset attached to an object. It describes the nature and/or the intended usage of the object. + +
+ - Can partial I/O operations be performed on attributes?
+
+
++ +Answer + +No. + +
+
-
+
- What are the two primary objects that can be included in a group?
+
+
++ +Answer + +A group and a dataset. + +
+
-
+
- Group names can be specified in two ways. What are these two types of group names?
+
+
++ +Answer + +Relative and absolute. + +
+ - You have a dataset named
mooin the groupboo, which is in the groupfoo, which, in turn, +is in therootgroup. How would you specify an absolute name to access this dataset? ++
++ +Answer + +/foo/boo/moo + +
+
-
+
- Describe a way to access the dataset moo described in the previous section
+(question 2) using a relative name. Describe a way to access the same dataset using an absolute name.
+
+
++ +Answer + +Access the group /foo and get the group ID. Access the group boo using the group ID obtained in Step 1. +Access the dataset moo using the group ID obtained in Step 2. +\code +gid = H5Gopen (file_id, "/foo", 0); /* absolute path */ +gid1 = H5Gopen (gid, "boo", 0); /* relative path */ +did = H5Dopen (gid1, "moo"); /* relative path */ +\endcode +Access the group /foo and get the group ID. Access the dataset boo/moo with the group ID just obtained. +\code +gid = H5Gopen (file_id, "/foo", 0); /* absolute path */ +did = H5Dopen (gid, "boo/moo"); /* relative path */ +\endcode +Access the dataset with an absolute path. +\code +did = H5Dopen (file_id, "/foo/boo/moo"); /* absolute path */ +\endcode + +
+
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBCompiling Compiling HDF5 Applications +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBCompiling Tools and Instructions on Compiling +Compiling applications to use the HDF5 Library can be as simple as executing: +\code +h5cc -o myprog myprog.c +\endcode + +As an application's file base evolves, there are better solutions using autotools and makefiles or +CMake and CMakeLists.txt files. Many tutorials and references can be found with a simple search. + +This tutorial section will discuss the use of compile scripts on Linux. +See the \ref secLBCompilingVS section for compiling with Visual Studio. + +\section secLBCompilingLinux Compile Scripts +When the library is built, the following compile scripts are included: +\li h5cc: compile script for HDF5 C programs +\li h5fc: compile script for HDF5 F90 programs +\li h5c++: compile script for HDF5 C++ programs + +These scripts are easilye used to compile single file applications, such as those included in the tutorial. +
| Warning + | +The h5cc/h5fc/h5c++ compile scripts are included when building with configure. Versions of +these compile scripts have also been added to CMake for Linux ONLY. The CMake versions rely on pkgconfig files. + | +
|---|
Examples of Using the Unix Compile Scripts:
+Following are examples of compiling and running an application with the Unix compile scripts: +\code + h5fc myprog.f90 + ./a.out + + h5cc -o myprog myprog.c + ./myprog +\endcode + +To see how the libraries linked in with a compile script were configured and built, use the +-showconfig option. For example, if using h5cc type: +\code + h5cc -showconfig +\endcode + +Detailed Description of Unix Compile Scripts:
+The h5cc, h5c++, and h5fc compile scripts come with the HDF5 binary distributions (include files, +libraries, and utilities) for the platforms we support. The h5c++ and h5fc utilities are ONLY present +if the library was built with C++ and Fortran. + +\section secLBCompilingVS Using Visual Studio + + 1. If you are building on 64-bit Windows, find the "Platform" dropdown + and select "x64". Also select the correct Configuration (Debug, Release, RelWithDebInfo, etc) + + 2. Set up path for external headers + + The HDF5 install path settings will need to be in the project property sheets per project. + Go to "Project" and select "Properties", find "Configuration Properties", + and then "C/C++". + + 2.1 Add the header path to the "Additional Include Directories" setting. Under "C/C++" + find "General" and select "Additional Include Directories". Select "Edit" from the dropdown + and add the HDF5 install/include path to the list. + (Ex: "C:\Program Files\HDF_Group\HDF5\1.10.9\include") + + 2.2 Building applications with the dynamic/shared hdf5 libraries requires + that the "H5_BUILT_AS_DYNAMIC_LIB" compile definition be used. Under "C/C++" + find "Preprocessor" and select "Preprocessor Definitions". Select "Edit" from the dropdown + and add "H5_BUILT_AS_DYNAMIC_LIB" to the list. + + 3. Set up path for external libraries + + The HDF5 install path/lib settings will need to be in the project property sheets per project. + Go to "Project" and select "Properties", find "Configuration Properties", + and then "Linker". + + 3.1 Add the libraries to the "Additional Dependencies" setting. Under "Linker" + find "Input" and select "Additional Dependencies". Select "Edit" from the dropdown + and add the required HDF5 install/lib path to the list. + (Ex: "C:\Program Files\HDF_Group\HDF5\1.10.9\lib\hdf5.lib") + + 3.2 For static builds, the external libraries should be added. + For example, to compile a C++ application, enter: + libhdf5_cpp.lib libhdf5.lib libz.lib libszaec.lib libaec.lib + +\section secLBCompilingLibs HDF5 Libraries +Following are the libraries included with HDF5. Whether you are using the Unix compile scripts or +Makefiles, or are compiling on Windows, these libraries are or may need to be specified. The order +they are specified is important on Linux: + +| Library | +Linux Name | +Mac Name | +Windows Name | +
|---|---|---|---|
| +\code +HDF5 High Level C++ APIs +HDF5 C++ Library +HDF5 High Level Fortran APIs +HDF5 Fortran Library +HDF5 High Level C APIs +HDF5 C Library +\endcode + | ++\code +libhdf5_hl_cpp.a +libhdf5_cpp.a +libhdf5hl_fortran.a +libhdf5_fortran.a +libhdf5_hl.a +libhdf5.a +\endcode + | ++\code +libhdf5_hl_cpp.a +libhdf5_cpp.a +libhdf5hl_fortran.a +libhdf5_fortran.a +libhdf5_hl.a +libhdf5.a +\endcode + | ++Windows +\code +libhdf5_hl_cpp.lib +libhdf5_cpp.lib +libhdf5hl_fortran.lib +libhdf5_fortran.lib +libhdf5_hl.lib +libhdf5.lib +\endcode + |
| Library | +Linux Name | +Mac Name | +Windows Name | +
|---|---|---|---|
| +\code +HDF5 High Level C++ APIs +HDF5 C++ Library +HDF5 High Level Fortran APIs +HDF5 Fortran Library +HDF5 High Level C APIs +HDF5 C Library +\endcode + | ++\code +libhdf5_hl_cpp.so +libhdf5_cpp.so +libhdf5hl_fortran.so +libhdf5_fortran.so +libhdf5_hl.so +libhdf5.so +\endcode + | ++\code +libhdf5_hl_cpp.dylib +libhdf5_cpp.dylib +libhdf5hl_fortran.dylib +libhdf5_fortran.dylib +libhdf5_hl.dylib +libhdf5.dylib +\endcode + | ++\code +hdf5_hl_cpp.lib +hdf5_cpp.lib +hdf5hl_fortran.lib +hdf5_fortran.lib +hdf5_hl.lib +hdf5.lib +\endcode + |
| Library | +Linux Name | +Mac Name | +Windows Name | +
|---|---|---|---|
| +\code +SZIP Compression Library +SZIP Compression Library +ZLIB or DEFLATE Compression Library +\endcode + | ++\code +libszaec.a +libaec.a +libz.a +\endcode + | ++\code +libszaec.a +libaec.a +libz.a +\endcode + | ++\code +libszaec.lib +libaec.lib +libz.lib +\endcode + | +
How do you use find_package with HDF5?
+To use find_package you will first need to make sure that HDF5_ROOT is set correctly. For setting this +environment variable see the Preconditions in the USING_HDF5_CMake.txt file in the share directory. + +See the CMakeLists.txt file provided with these examples for how to use find_package with HDF5. + +Please note that the find_package invocation changed to require "shared" or "static": +\code + FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED shared) + FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED static) +\endcode + +Previously, the find_package invocation was: +\code + FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED) +\endcode + +My platform/compiler is not included. Can I still use the configuration files?
+Yes, you can but you will have to edit the HDF5_Examples.cmake file and update the variable: +\code + CTEST_CMAKE_GENERATOR +\endcode + +The generators for your platform can be seen by typing: +\code + cmake --help +\endcode + +What do I do if the build fails?
+I received an error during the build and the application binary is not in the +build directory as I expected. How do I determine what the problem is? + +If the error is not clear, then the first thing you may want to do is replace the -V (Dash Uppercase Vee) +option for ctest in the build script to -VV (Dash Uppercase Vee Uppercase Vee). Then remove the build +directory and re-run the build script. The output should be more verbose. + +If the error is still not clear, then check the log files. You will find those in the build directory. +For example, on Unix the log files will be in: +\code + build/Testing/Temporary/ +\endcode +There are log files for the configure, test, and build. + ++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBTraining Training Videos +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +Training Videos + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnHDFView.dox b/doxygen/dox/LearnHDFView.dox new file mode 100644 index 0000000..b1f632c --- /dev/null +++ b/doxygen/dox/LearnHDFView.dox @@ -0,0 +1,472 @@ +/** @page LearnHDFView Learning HDF5 with HDFView + +Navigate back: \ref index "Main" / \ref GettingStarted +
+ +This tutorial enables you to get a feel for HDF5 by using the HDFView browser. It does NOT require +any programming experience. + +\section sec_learn_hv_install HDFView Installation +\li Download and install HDFView. It can be downloaded from the Download HDFView page. +\li Obtain the storm1.txt text file, used in the tutorial. + +\section sec_learn_hv_begin Begin Tutorial +Once you have HDFView installed, bring it up and you are ready to begin the tutorial. + +
+This tutorial requires that the default HDFView File Access Mode be Read / Write. If fields are greyed out so that you cannot select them, then the File Access Mode is Read Only.
+
+To change the File Access Mode follow these steps:
+
|
+
-
+
- @ref subsec_learn_hv_topics_file +
- @ref subsec_learn_hv_topics_image +
- @ref subsec_learn_hv_topics_attr +
- @ref subsec_learn_hv_topics_compress +
- @ref subsec_learn_hv_topics_subset +
- @ref subsec_learn_hv_topics_table +
-
+
- Select the File pull-down menu at the top left, and then select New -> HDF5. +
- Specify a location and type in storm.h5 for the name of your file, and click on the Save button.
+You will see the storm.h5 file in the TableView:
+
+
++ ++\image html storm.png + +
+ - Right click on storm.h5, and select New -> Group. +
- Enter Data for the name of the group and then click the Ok button. You will see the group Data in the TableView.
+
+
++ ++\image html DataGroup.png + +
+ - Right click on the group Data and select New -> Dataset. +
- A window pops up on the right. Fill in the information as follows, and then click Ok (leave the
+Datatype information as is):
+
+
++ +Dataset Name + +Storm + ++ +Under Dataspace, Current size + +57x57 + ++ +Layout + +Contiguous (default) + +
+ - Click to expand the Data group in the tree view to see the Storm dataset:
+
+
++ ++\image html StormDataset.png + +
+ - Double left click on the Storm dataset in the tree view. A window with an empty spreadsheet pops open. +
- Copy the data from the storm1.txt file into the dataset.
+
+If you downloaded storm1.txt,
+then click on the Import/Export Data menu and select Import Data from -> Text File.
+Specify a location, select storm1.txt
+and click on the Open button. Answer Yes in the dialog box that
+pops up (which asks if you wish to paste the selected data).
+
+Alternately, you can copy/paste directly. Select and copy the data in a separate window. Position your
+cursor at (0,0) in your table, and select Paste from the Table menu.
+
+The values will be entered into the spreadsheet.
+
+
++ ++\image html datasetwdata.png + +
+ - Table -> Close the dataset, and save the data. +
-
+
- Right click on Storm in the tree view, and select Open As. +
- Select the Image button under Display As (near the top) in the Dataset Selection window that pops
+up. Then click OK at the bottom of the window to display the image.
+
+
++ ++\image html showasimage.png + +
+ - The rainbow icon brings you to the Image Palette window. Click on that to play with the palette +(GrayWave probably is the best choice). Close. +
-
+
- Click on the /Data folder in the tree view. You will see two tabs, Object Attribute Info and
+General Object Info, in the pane on the right site of the HDFView window.
+
+
++ ++\image html noattrs.png + +
+ - With the left mouse button, select the Add Attribute button. +
- Select the Add Attribute button to add an attribute with these values: +
- Select the Ok button. The attribute will show up under the Object Attribute Info tab. +
- Double-click the BatchID attribute line to open the data table for BatchID. +
- Click in the first cell and enter 3343 followed by the enter key. +
- Table -> Close, answer Yes in the dialog box that +pops up (which asks if you wish to paste the selected data). +
| Name + | +BatchID + | +
|---|---|
| Type + | +INTEGER + | +
| Size (bits) + | +32 + | +
-
+
- Left mouse click on the /Storm dataset in the tree view. You will see the Object Attribute +Info and General Object Info tabs on the right +
- In the Object Attribute Info pane select the Add Attribute button and enter an attribute with +these values. (Be sure to add a String Length or the string will be truncated to one character!): +
- Select the Ok button. The attribute will show up under the Object Attribute Info tab. +
- Double-click the Units attribute line to open the data table for Units. +
- Click in the first cell and enter m/s followed by the enter key. +
- Table -> Close, answer Yes in the dialog box that
+pops up (which asks if you wish to paste the selected data).
+
+
++ ++\image html scarletletter.png + +
+
| Name + | +Units + | +
|---|---|
| Type + | +STRING + | +
| String Length + | +3 + | +
-
+
- Right click on storm.h5. Select New -> Group. +
- Enter Image for the name of the group, and click the OK button to create the group.
+
+
++ ++\image html newgroupimage.png + +
+ - Right click on the Image group, and select New -> Dataset. +
- Enter the following information for the dataset. Leave the Datatype as is (INTEGER):
+
+
+You will see the Another Storm dataset in the Image group: ++ +Dataset name + +Another Storm + ++ +Under Dataspace, Current size + +57x57 + ++ +Storage Layout + +Chunked + ++ +Chunk Size + +20x20 + ++ +Compression + +gzip + ++ Compression Level + +9 + ++
++ ++\image html hdfview-anthrstrm.png + +
+ - Double left-mouse click on the Another Storm dataset to display the spreadsheet:
+
+
++ ++\image html hdfview-anthrstrm-sprdsht.png + +
+ - Copy the data from the storm1.txt file into the dataset. (See the previous topic for copying +storm1.txt into a dataset.) +
- Table -> Close, and save the data. +
- Right click on Another Storm, and select Open As. +
- Select the Image button in the Dataset Selection window that pops up. Click the Ok button at the
+bottom of the window to view the dataset as an image.
+
+
++ ++\image html hdfview-anthrstrm-img.png + +
+
-
+
- Right click on the Data group and select New -> Image. +
- A window pops up on the right. Enter the following and then click Ok: +
- Close the dataset. +
- Expand the Data group to see its contents. You will see the Storm Image dataset.
+
+
++ ++\image html hdfview-imgicon.png + +
+ -
+Add data to the Storm Image dataset as was shown previously:
+
-
+
- Right click on Storm Image, and select Open As to open the Dataset Selection window. +
- Click on the Spreadsheet button at the top left of the Dataset Selection window to view the image +as a spreadsheet. +
- Copy the data from the storm1.txt file into the dataset. +
- Close the dataset and save the data. +
+ - Left double click on Storm Image to see the image. Close the dataset. +
- Right click on Storm Image and select Open As to bring up the Data Selection window. +
- Select a subset by clicking the left mouse on the image in the window and dragging the mouse.
+Notice that the Height and Width values change. Select to display it as an image. Click Ok.
+
+
++ ++\image html hdfview-imgsubset.png + +
+ - Position the cursor in the middle of the image. Press Shift+Left Mouse button and hold, and then +drag the mouse to select another subset. +
- Select Image->Write Selection to Image. Enter Subset for the new image name. Click Ok. The Subset +image will appear in the tree view on the left. +
- Left double click on the image Subset to bring it up on the right.
+
+
++ ++\image html hdfview-newimgsubset.png + +
+ - Close the Subset image. +
| Image name + | +Storm Image + | +
|---|---|
| Height + | +57 + | +
| Width + | +57 + | +
-
+
- Right button click on the group Data. Select New -> Compound DS. +
- A window pops up. Only fill in the following fields:
+
+
+ ++ +Dataset name + +Table + ++ +Dataspace (Current size only) + +4 + ++ +Compound Datatype Properties: + +
Number of Members +3 + ++ +Compound Datatype Properties: + +
Name / Datatype / Size +Description / string / 4 + +
Temperature / float / 1 +
Pressure / double / 1 ++
++ ++\image html hdfview-newcmpd.png + +
+ - Click Ok at the bottom. +
- Open the Data group (if it is not open) and double left click on the Table object.
+
+
++ ++\image html hdfview-table.png + +
+ - Close the dataset. +
+Navigate back: \ref index "Main" / \ref GettingStarted + +*/ diff --git a/doxygen/dox/ReferenceManual.dox b/doxygen/dox/ReferenceManual.dox index df0c747..7900925 100644 --- a/doxygen/dox/ReferenceManual.dox +++ b/doxygen/dox/ReferenceManual.dox @@ -1,53 +1,32 @@ /** \page RM HDF5 Reference Manual -The functions provided by the HDF5 C-API are grouped into the following +The functions provided by the HDF5 API are grouped into the following \Emph{modules}:
| Modules | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
-
+ +\section secToolsBasic Basic Facts about HDF5 +The following are basic facts about HDF5 files to keep in mind while completing these tutorial topics: +\li All HDF5 files contain a root group "/". +\li There are two primary objects in HDF5, a group and a dataset: + Groups allow objects to be organized into a group structure, such as a tree. + Datasets contain raw data values. +\li Additional information about an HDF5 object may optionally be stored in attributes attached to the object. + +\section secToolsTopics Tutorial Topics +
+Navigate back: \ref index "Main" / \ref GettingStarted + +@page ViewToolsCommand Command-line Tools +Navigate back: \ref index "Main" / \ref GettingStarted + + +\section secViewToolsCommandObtain Obtain Tools and Files (Optional) +Pre-built binaries for Linux and Windows are distributed within the respective HDF5 binary release +packages, which can be obtained from the Download HDF5 page. + +HDF5 files can be obtained from various places such as \ref HDF5Examples and HDF-EOS and Tools and +Information Center. Specifically, the following examples are used in this tutorial topic: +\li HDF5 Files created from compiling the \ref LBExamples +\li HDF5 Files on the Examples by API page +\li NPP JPSS files, SVM01_npp.. (gzipped) +and SVM09_npp.. (gzipped) +\li HDF-EOS OMI-Aura file + +\section secViewToolsCommandTutor Tutorial Topics +A variety of command-line tools are included in the HDF5 binary distribution. There are tools to view, +edit, convert and compare HDF5 files. This tutorial discusses the tools by their functionality. It +does not cover all of the HDF5 tools. + +
+Navigate back: \ref index "Main" / \ref GettingStarted + +@page ViewToolsView Command-line Tools For Viewing HDF5 Files +Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + + +\section secViewToolsViewTOC Contents +
-h
+or --help option:
+\code
+h5dump -h
+\endcode
+
+The following h5dump options can be helpful in viewing the content and structure of a file:
+
h5dump using the -d D option and specifying the entire
+path and name of the dataset for D. The path is important in identifying the correct dataset,
+as there can be multiple datasets with the same name. The path can be determined by looking at
+the objects in the file with h5dump -n.
+
+The following example uses the groups.h5 file that is created by the
+\ref LBExamples
+example h5_crtgrpar.c. To display dset1 in the groups.h5 file below, specify dataset
+/MyGroup/dset1. The -H option is used to suppress printing of the data values:
+
+Contents of groups.h5
+\code
+ $ h5dump -n groups.h5
+ HDF5 "groups.h5" {
+ FILE_CONTENTS {
+ group /
+ group /MyGroup
+ group /MyGroup/Group_A
+ dataset /MyGroup/Group_A/dset2
+ group /MyGroup/Group_B
+ dataset /MyGroup/dset1
+ }
+ }
+\endcode
+
+Display dataset "dset1"
+\code
+ $ h5dump -d "/MyGroup/dset1" -H groups.h5
+ HDF5 "groups.h5" {
+ DATASET "/MyGroup/dset1" {
+ DATATYPE H5T_STD_I32BE
+ DATASPACE SIMPLE { ( 3, 3 ) / ( 3, 3 ) }
+ }
+ }
+\endcode
+
+\subsubsection subsubsecViewToolsViewDset_h5dumpEx5 Example 5
+The -p option is used to examine the the dataset filters, storage layout, and fill value properties of a dataset.
+
+This option can be useful for checking how well compression works, or even for analyzing performance
+and dataset size issues related to chunking. (The smaller the chunk size, the more chunks that HDF5
+has to keep track of, which increases the size of the file and potentially affects performance.)
+
+In the file shown below the dataset /DS1 is both chunked and compressed:
+\code
+ $ h5dump -H -p -d "/DS1" h5ex_d_gzip.h5
+ HDF5 "h5ex_d_gzip.h5" {
+ DATASET "/DS1" {
+ DATATYPE H5T_STD_I32LE
+ DATASPACE SIMPLE { ( 32, 64 ) / ( 32, 64 ) }
+ STORAGE_LAYOUT {
+ CHUNKED ( 4, 8 )
+ SIZE 5278 (1.552:1 COMPRESSION)
+ }
+ FILTERS {
+ COMPRESSION DEFLATE { LEVEL 9 }
+ }
+ FILLVALUE {
+ FILL_TIME H5D_FILL_TIME_IFSET
+ VALUE 0
+ }
+ ALLOCATION_TIME {
+ H5D_ALLOC_TIME_INCR
+ }
+ }
+ }
+\endcode
+
+You can obtain the h5ex_d_gzip.c program that created this file, as well as the file created,
+from the Examples by API page.
+
+\subsection subsecViewToolsViewDset_h5ls h5ls
+Specific datasets can be specified with h5ls by simply adding the dataset path and dataset after the
+file name. As an example, this command displays dataset dset2 in the groups.h5
+file used in @ref subsubsecViewToolsViewDset_h5dumpEx4 :
+\code
+h5ls groups.h5/MyGroup/Group_A/dset2
+\endcode
+
+Just the dataspace information gets displayed:
+\code
+dset2 Dataset {2, 10}
+\endcode
+
+The following options can be used to see detailed information about a dataset.
+
-v is shown below:
+\code
+ $ h5ls -v groups.h5/MyGroup/Group_A/dset2
+ Opened "groups.h5" with sec2 driver.
+ dset2 Dataset {2/2, 10/10}
+ Location: 1:3840
+ Links: 1
+ Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
+ Type: 32-bit big-endian integer
+\endcode
+
+The output of using -d is shown below:
+\code
+ $ h5ls -d groups.h5/MyGroup/Group_A/dset2
+ dset2 Dataset {2, 10}
+ Data:
+ (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
+\endcode
+
+\section secViewToolsViewGrps Groups
+Both h5dump and h5ls can be used to view specific groups in a file.
+
h5dump options that are useful for examining groups are:
+
HDFEOS group in the OMI file mentioned previously, you can specify the path and name of the group as follows:
+\code
+h5dump -g "/HDFEOS" -H -A 0 OMI-Aura.he5
+\endcode
+
+The -A 0 option suppresses attributes and -H suppresses printing of data values:
+\code
+ HDF5 "OMI-Aura.he5" {
+ GROUP "/HDFEOS" {
+ GROUP "ADDITIONAL" {
+ GROUP "FILE_ATTRIBUTES" {
+ }
+ }
+ GROUP "GRIDS" {
+ GROUP "OMI Column Amount O3" {
+ GROUP "Data Fields" {
+ DATASET "ColumnAmountO3" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ DATASET "RadiativeCloudFraction" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ DATASET "SolarZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ DATASET "ViewingZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ }
+ }
+ }
+ }
+ }
+\endcode
+
+\subsection subsecViewToolsViewGrps_h5ls h5ls
+You can view the contents of a group with h5ls/ by specifying the group after the file name.
+To use h5ls to view the contents of the /HDFEOS group in the OMI-Aura.he5 file, type:
+\code
+h5ls -r OMI-Aura.he5/HDFEOS
+\endcode
+
+The output of this command is:
+\code
+ /ADDITIONAL Group
+ /ADDITIONAL/FILE_ATTRIBUTES Group
+ /GRIDS Group
+ /GRIDS/OMI\ Column\ Amount\ O3 Group
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields Group
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/ColumnAmountO3 Dataset {720, 1440}
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/RadiativeCloudFraction Dataset {720, 1440}
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/SolarZenithAngle Dataset {720, 1440}
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/ViewingZenithAngle Dataset {720, 1440}
+\endcode
+
+If you specify the -v option, you can also see the attributes and properties of the datasets.
+
+\section secViewToolsViewAttr Attributes
+
+\subsection subsecViewToolsViewAttr_h5dump h5dump
+Attributes are displayed by default if using h5dump. Some files contain many attributes, which
+can make it difficult to examine the objects in the file. Shown below are options that can help
+when using h5dump to work with files that have attributes.
+
+\subsubsection subsubsecViewToolsViewAttr_h5dumpEx6 Example 6
+The -a A option will display an attribute. However, the path to the attribute must be included
+when specifying this option. For example, to see the ScaleFactor attribute in the OMI-Aura.he5 file, type:
+\code
+h5dump -a "/HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle/ScaleFactor" OMI-Aura.he5
+\endcode
+
+This command displays:
+\code
+ HDF5 "OMI-Aura.he5" {
+ ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+ }
+ }
+\endcode
+
+How can you determine the path to the attribute? This can be done by looking at the file contents with the -n 1 option:
+\code
+h5dump -n 1 OMI-Aura.he5
+\endcode
+
+Below is a portion of the output for this command:
+\code
+ HDF5 "OMI-Aura.he5" {
+ FILE_CONTENTS {
+ group /
+ group /HDFEOS
+ group /HDFEOS/ADDITIONAL
+ group /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/EndUTC
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleDay
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleDayOfYear
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleMonth
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleYear
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/InstrumentName
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/OrbitNumber
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/OrbitPeriod
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/PGEVersion
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/Period
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/ProcessLevel
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/StartUTC
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/TAI93At0zOfGranule
+
+ ...
+\endcode
+
+There can be multiple objects or attributes with the same name in a file. How can you make sure
+you are finding the correct object or attribute? You can first determine how many attributes
+there are with a specified name, and then examine the paths to them.
+
+The -N option can be used to display all objects or attributes with a given name.
+For example, there are four attributes with the name ScaleFactor in the OMI-Aura.he5 file,
+as can be seen below with the -N option:
+\code
+h5dump -N ScaleFactor OMI-Aura.he5
+\endcode
+
+It outputs:
+\code
+HDF5 "OMI-Aura.he5" {
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+}
+\endcode
+
+\subsection subsecViewToolsViewAttr_h5ls h5ls
+If you include the -v (verbose) option for h5ls, you will see all of the attributes for the
+specified file, dataset or group. You cannot display individual attributes.
+
+\section secViewToolsViewSub Dataset Subset
+
+\subsection subsecViewToolsViewSub_h5dump h5dump
+If you have a very large dataset, you may wish to subset or see just a portion of the dataset.
+This can be done with the following h5dump options.
+
START (s), STRIDE (S), COUNT (c), and BLOCK (k) options
+define the shape and size of the selection. They are arrays with the same number of dimensions as the rank
+of the dataset's dataspace, and they all work together to define the selection. A change to one of
+these arrays can affect the others.
+
+When specifying these h5dump options, a comma is used as the delimiter for each dimension in the
+option value. For example, with a 2-dimensional dataset, the option value is specified as "H,W",
+where H is the height and W is the width. If the offset is 0 for both dimensions, then
+START would be specified as follows:
+\code
+-s "0,0"
+\endcode
+
+There is also a shorthand way to specify these options with brackets at the end of the dataset name:
+\code
+-d DATASETNAME[s;S;c;k]
+\endcode
+
+Multiple dimensions are separated by commas. For example, a subset for a 2-dimensional dataset would be specified as follows:
+\code
+-d DATASETNAME[s,s;S,S;c,c;k,k]
+\endcode
+
+For a detailed understanding of how selections works, see the #H5Sselect_hyperslab API in the \ref RM.
+
+The dataset SolarZenithAngle in the OMI-Aura.he5 file can be used to illustrate these options. This
+dataset is a 2-dimensional dataset of size 720 (height) x 1440 (width). Too much data will be displayed
+by simply viewing the specified dataset with the -d option:
+\code
+h5dump -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" OMI-Aura.he5
+\endcode
+Subsetting narrows down the output that is displayed. In the following example, the first
+15x10 elements (-c "15,10") are specified, beginning with position (0,0) (-s "0,0"):
+\code
+ h5dump -A 0 -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" -s "0,0" -c "15,10" -w 0 OMI-Aura.he5
+\endcode
+
+If using the shorthand method, specify:
+\code
+ h5dump -A 0 -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle[0,0;;15,10;]" -w 0 OMI-Aura.he5
+\endcode
+
+Where,
+\par The -d option must be specified
+
+before
+\par subsetting options (if not using the shorthand method).
+
+The -A 0 option suppresses the printing of attributes.
+
+The -w 0 option sets the number of columns of output to the maximum allowed value (65535).
+This ensures that there are enough columns specified for displaying the data.
+
+Either command displays:
+\code
+ HDF5 "OMI-Aura.he5" {
+ DATASET "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ SUBSET {
+ START ( 0, 0 );
+ STRIDE ( 1, 1 );
+ COUNT ( 15, 10 );
+ BLOCK ( 1, 1 );
+ DATA {
+ (0,0): 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403,
+ (1,0): 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071,
+ (2,0): 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867,
+ (3,0): 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632,
+ (4,0): 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429,
+ (5,0): 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225,
+ (6,0): 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021,
+ (7,0): 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715,
+ (8,0): 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511,
+ (9,0): 77.658, 77.658, 77.658, 77.307, 77.307, 77.307, 77.307, 77.307, 77.307, 77.307,
+ (10,0): 77.556, 77.556, 77.556, 77.556, 77.556, 77.556, 77.556, 77.556, 77.102, 77.102,
+ (11,0): 78.408, 78.408, 78.408, 78.408, 78.408, 78.408, 78.408, 78.408, 77.102, 77.102,
+ (12,0): 76.34, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413,
+ (13,0): 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 77.195,
+ (14,0): 78.005, 78.005, 78.005, 78.005, 78.005, 78.005, 76.991, 76.991, 76.991, 76.991
+ }
+ }
+ }
+ }
+\endcode
+
+What if we wish to read three rows of three elements at a time (-c "3,3"), where each element
+is a 2 x 3 block (-k "2,3") and we wish to begin reading from the second row (-s "1,0")?
+
+You can do that with the following command:
+\code
+ h5dump -A 0 -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle"
+ -s "1,0" -S "2,3" -c "3,3" -k "2,3" -w 0 OMI-Aura.he5
+\endcode
+
+In this case, the stride must be specified as 2 by 3 (or larger) to accommodate the reading of 2 by 3 blocks.
+If it is smaller, the command will fail with the error,
+\code
+h5dump error: wrong subset selection; blocks overlap.
+\endcode
+
+The output of the above command is shown below:
+\code
+ HDF5 "OMI-Aura.he5" {
+ DATASET "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ SUBSET {
+ START ( 1, 0 );
+ STRIDE ( 2, 3 );
+ COUNT ( 3, 3 );
+ BLOCK ( 2, 3 );
+ DATA {
+ (1,0): 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071,
+ (2,0): 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867,
+ (3,0): 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632,
+ (4,0): 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429,
+ (5,0): 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225,
+ (6,0): 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021
+ }
+ }
+ }
+ }
+\endcode
+
+\section secViewToolsViewDtypes Datatypes
+
+\subsection subsecViewToolsViewDtypes_h5dump h5dump
+The following datatypes are discussed, using the output of h5dump with HDF5 files from the
+Examples by API page:
+
/DS1. It has a datatype of #H5T_STD_I32LE (32-bit Little-Endian Integer)
+and is a 4 by 7 array:
+\code
+$ h5dump h5ex_d_rdwr.h5
+HDF5 "h5ex_d_rdwr.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_STD_I32LE
+ DATASPACE SIMPLE { ( 4, 7 ) / ( 4, 7 ) }
+ DATA {
+ (0,0): 0, -1, -2, -3, -4, -5, -6,
+ (1,0): 0, 0, 0, 0, 0, 0, 0,
+ (2,0): 0, 1, 2, 3, 4, 5, 6,
+ (3,0): 0, 2, 4, 6, 8, 10, 12
+ }
+ }
+}
+}
+\endcode
+
+Contrast that with the following dataset that has both an Array datatype and is an array:
+\code
+$ h5dump h5ex_t_array.h5
+HDF5 "h5ex_t_array.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_ARRAY { [3][5] H5T_STD_I64LE }
+ DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
+ DATA {
+ (0): [ 0, 0, 0, 0, 0,
+ 0, -1, -2, -3, -4,
+ 0, -2, -4, -6, -8 ],
+ (1): [ 0, 1, 2, 3, 4,
+ 1, 1, 1, 1, 1,
+ 2, 1, 0, -1, -2 ],
+ (2): [ 0, 2, 4, 6, 8,
+ 2, 3, 4, 5, 6,
+ 4, 4, 4, 4, 4 ],
+ (3): [ 0, 3, 6, 9, 12,
+ 3, 5, 7, 9, 11,
+ 6, 7, 8, 9, 10 ]
+ }
+ }
+}
+}
+\endcode
+
+In this file, dataset /DS1 has a datatype of
+\code
+H5T_ARRAY { [3][5] H5T_STD_I64LE }
+\endcode
+and it also has a dataspace of
+\code
+SIMPLE { ( 4 ) / ( 4 ) }
+\endcode
+In other words, it is an array of four elements, in which each element is a 3 by 5 array of #H5T_STD_I64LE.
+
+This dataset is much more complex. Also note that subsetting cannot be done on Array datatypes.
+
+See this FAQ for more information on the Array datatype.
+
+\subsubsection subsubsecViewToolsViewDtypes_objref Object Reference
+An Object Reference is a reference to an entire object (dataset, group, or named datatype).
+A dataset with an Object Reference datatype consists of one or more Object References.
+An Object Reference dataset can be used as an index to an HDF5 file.
+
+The /DS1 dataset in the following file (h5ex_t_objref.h5) is an Object Reference dataset.
+It contains two references, one to group /G1 and the other to dataset /DS2:
+\code
+$ h5dump h5ex_t_objref.h5
+HDF5 "h5ex_t_objref.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_REFERENCE { H5T_STD_REF_OBJECT }
+ DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
+ DATA {
+ (0): GROUP 1400 /G1 , DATASET 800 /DS2
+ }
+ }
+ DATASET "DS2" {
+ DATATYPE H5T_STD_I32LE
+ DATASPACE NULL
+ DATA {
+ }
+ }
+ GROUP "G1" {
+ }
+}
+}
+\endcode
+
+\subsubsection subsubsecViewToolsViewDtypes_regref Region Reference
+A Region Reference is a reference to a selection within a dataset. A selection can be either
+individual elements or a hyperslab. In h5dump you will see the name of the dataset along with
+the elements or slab that is selected. A dataset with a Region Reference datatype consists of
+one or more Region References.
+
+An example of a Region Reference dataset (h5ex_t_regref.h5) can be found on the
+Examples by API page,
+under Datatypes. If you examine this dataset with h5dump you will see that /DS1 is a
+Region Reference dataset as indicated by its datatype, highlighted in bold below:
+\code
+$ h5dump h5ex_t_regref.h5
+HDF5 "h5ex_t_regref.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
+ DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
+ DATA {
+ DATASET /DS2 {(0,1), (2,11), (1,0), (2,4)},
+ DATASET /DS2 {(0,0)-(0,2), (0,11)-(0,13), (2,0)-(2,2), (2,11)-(2,13)}
+ }
+ }
+ DATASET "DS2" {
+ DATATYPE H5T_STD_I8LE
+ DATASPACE SIMPLE { ( 3, 16 ) / ( 3, 16 ) }
+ DATA {
+ (0,0): 84, 104, 101, 32, 113, 117, 105, 99, 107, 32, 98, 114, 111, 119,
+ (0,14): 110, 0,
+ (1,0): 102, 111, 120, 32, 106, 117, 109, 112, 115, 32, 111, 118, 101,
+ (1,13): 114, 32, 0,
+ (2,0): 116, 104, 101, 32, 53, 32, 108, 97, 122, 121, 32, 100, 111, 103,
+ (2,14): 115, 0
+ }
+ }
+}
+}
+\endcode
+
+It contains two Region References:
+\li A selection of four individual elements in dataset /DS2 : (0,1), (2,11), (1,0), (2,4)
+See the #H5Sselect_elements API in the \ref UG for information on selecting individual elements.
+\li A selection of these blocks in dataset /DS2 : (0,0)-(0,2), (0,11)-(0,13), (2,0)-(2,2), (2,11)-(2,13)
+See the #H5Sselect_hyperslab API in the \ref UG for how to do hyperslab selection.
+
+
+If you look at the code that creates the dataset (h5ex_t_regref.c) you will see that the
+first reference is created with these calls:
+\code
+ status = H5Sselect_elements (space, H5S_SELECT_SET, 4, coords[0]);
+ status = H5Rcreate (&wdata[0], file, DATASET2, H5R_DATASET_REGION, space);
+\endcode
+
+where the buffer containing the coordinates to select is:
+\code
+ coords[4][2] = { {0, 1},
+ {2, 11},
+ {1, 0},
+ {2, 4} },
+\endcode
+
+The second reference is created by calling,
+\code
+ status = H5Sselect_hyperslab (space, H5S_SELECT_SET, start, stride, count, block);
+ status = H5Rcreate (&wdata[1], file, DATASET2, H5R_DATASET_REGION, space);
+\endcode
+where start, stride, count, and block have these values:
+\code
+ start[2] = {0, 0},
+ stride[2] = {2, 11},
+ count[2] = {2, 2},
+ block[2] = {1, 3};
+\endcode
+
+These start, stride, count, and block values will select the elements shown in bold in the dataset:
+\code
+84 104 101 32 113 117 105 99 107 32 98 114 111 119 110 0
+102 111 120 32 106 117 109 112 115 32 111 118 101 114 32 0
+116 104 101 32 53 32 108 97 122 121 32 100 111 103 115 0
+\endcode
+
+If you use h5dump to select a subset of dataset
+/DS2 with these start, stride, count, and block values, you will see that the same elements are selected:
+\code
+$ h5dump -d "/DS2" -s "0,0" -S "2,11" -c "2,2" -k "1,3" h5ex_t_regref.h5
+HDF5 "h5ex_t_regref.h5" {
+DATASET "/DS2" {
+ DATATYPE H5T_STD_I8LE
+ DATASPACE SIMPLE { ( 3, 16 ) / ( 3, 16 ) }
+ SUBSET {
+ START ( 0, 0 );
+ STRIDE ( 2, 11 );
+ COUNT ( 2, 2 );
+ BLOCK ( 1, 3 );
+ DATA {
+ (0,0): 84, 104, 101, 114, 111, 119,
+ (2,0): 116, 104, 101, 100, 111, 103
+ }
+ }
+}
+}
+\endcode
+
+For more information on selections, see the tutorial topic on
+@ref LBDsetSubRW. Also see the
+\ref secViewToolsViewSub tutorial topic on using h5dump to view a subset.
+
+\subsubsection subsubsecViewToolsViewDtypes_string String
+There are two types of string data, fixed length strings and variable length strings.
+
+Below is the h5dump output for two files that have the same strings written to them. In one file,
+the strings are fixed in length, and in the other, the strings have different sizes (and are variable in size).
+
+Dataset of Fixed Length Strings
+\code
+HDF5 "h5ex_t_string.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_STRING {
+ STRSIZE 7;
+ STRPAD H5T_STR_SPACEPAD;
+ CSET H5T_CSET_ASCII;
+ CTYPE H5T_C_S1;
+ }
+ DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
+ DATA {
+ (0): "Parting", "is such", "sweet ", "sorrow."
+ }
+ }
+}
+}
+\endcode
+
+Dataset of Variable Length Strings
+\code
+HDF5 "h5ex_t_vlstring.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_STRING {
+ STRSIZE H5T_VARIABLE;
+ STRPAD H5T_STR_SPACEPAD;
+ CSET H5T_CSET_ASCII;
+ CTYPE H5T_C_S1;
+ }
+ DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
+ DATA {
+ (0): "Parting", "is such", "sweet", "sorrow."
+ }
+ }
+}
+}
+\endcode
+
+You might wonder which to use. Some comments to consider are included below.
+\li In general, a variable length string dataset is more complex than a fixed length string. If you don't
+specifically need a variable length type, then just use the fixed length string.
+\li A variable length dataset consists of pointers to heaps in different locations in the file. For this
+reason, a variable length dataset cannot be compressed. (Basically, the pointers get compressed and
+not the actual data!) If compression is needed, then do not use variable length types.
+\li If you need to create an array of of different length strings, you can either use fixed length strings
+along with compression, or use a variable length string.
+
++Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + +*/ diff --git a/doxygen/dox/ViewTools2.dox b/doxygen/dox/ViewTools2.dox new file mode 100644 index 0000000..4d8788a --- /dev/null +++ b/doxygen/dox/ViewTools2.dox @@ -0,0 +1,786 @@ +/** @page ViewToolsEdit Command-line Tools For Editing HDF5 Files +Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + + +\section secViewToolsEditTOC Contents +
h5repack tool can be used to remove unused space in an HDF5
+file. If no options other than the input and output HDF5 files are specified on the
+h5repack command line, it will write the file to the new
+file, getting rid of the unused space:
+\code
+h5repack I. Introduction-
The format of an HDF5 file on disk encompasses several - key ideas of the HDF4 and AIO file formats as well as - addressing some shortcomings therein. The new format is - more self-describing than the HDF4 format and is more - uniformly applied to data objects in the file. - -An HDF5 file appears to the user as a directed graph. - The nodes of this graph are the higher-level HDF5 objects - that are exposed by the HDF5 APIs: - -
At the lowest level, as information is actually written to the disk, - an HDF5 file is made up of the following objects: -
The HDF5 Library uses these low-level objects to represent the - higher-level objects that are then presented to the user or - to applications through the APIs. For instance, a group is an - object header that contains a message that points to a local - heap (for storing the links to objects in the group) and to a - B-tree (which indexes the links). A dataset is an object header - that contains messages that describe datatype, dataspace, layout, - filters, external files, fill value, and other elements with the - layout message pointing to either a raw data chunk or to a - B-tree that points to raw data chunks. +
The format of an HDF5 file on disk encompasses several key ideas + of the HDF4 and AIO file formats as well as addressing some + shortcomings therein. The new format is more self-describing than the + HDF4 format and is more uniformly applied to data objects in the file. + +An HDF5 file appears to the user as a directed graph. The nodes + of this graph are the higher-level HDF5 objects that are exposed by the + HDF5 APIs: + +
At the lowest level, as information is actually written to the + disk, an HDF5 file is made up of the following objects: +
The HDF5 Library uses these low-level objects to represent the + higher-level objects that are then presented to the user or to + applications through the APIs. For instance, a group is an object + header that contains a message that points to a local heap (for storing + the links to objects in the group) and to a B-tree (which indexes the + links). A dataset is an object header that contains messages that + describe datatype, dataspace, layout, filters, external files, fill + value, and other elements with the layout message pointing to either a + raw data chunk or to a B-tree that points to raw data chunks. I.A. This Document-This document describes the lower-level data objects; - the higher-level objects and their properties are described - in the HDF5 User’s Guide. - -Three levels of information comprise the file format. - Level 0 contains basic information for identifying and - defining information about the file. Level 1 information contains - the information about the pieces of a file shared by many objects - in the file (such as a B-trees and heaps). Level 2 is the rest - of the file and contains all of the data objects, with each object - partitioned into header information, also known as - metadata, and data. - -The sizes of various fields in the following layout tables are - determined by looking at the number of columns the field spans - in the table. There are three exceptions: (1) The size may be - overridden by specifying a size in parentheses, (2) the size of - addresses is determined by the Size of Offsets field - in the superblock and is indicated in this document with a - superscripted ‘O’, and (3) the size of length fields is determined - by the Size of Lengths field in the superblock and is - indicated in this document with a superscripted ‘L’. - -Values for all fields in this document should be treated as unsigned - integers, unless otherwise noted in the description of a field. - Additionally, all metadata fields are stored in little-endian byte - order. - - -All checksums used in the format are computed with the - Jenkins’ - lookup3 algorithm. - - -Whenever a bit flag or field is mentioned for an entry, bits are - numbered from the lowest bit position in the entry. - - -Various tables in this document aligned with “This space inserted - only to align table nicely”. These entries in the table are just - to make the table presentation nicer and do not represent any values - or padding in the file. - ++ This document describes the lower-level data objects; the higher-level + objects and their properties are described in the HDF5 + User Guide. + + ++ Three levels of information comprise the file format. Level 0 contains + basic information for identifying and defining information about the + file. Level 1 information contains the information about the pieces of + a file shared by many objects in the file (such as a B-trees and + heaps). Level 2 is the rest of the file and contains all of the data + objects, with each object partitioned into header information, also + known as metadata, and data. + + ++ The sizes of various fields in the following layout tables are + determined by looking at the number of columns the field spans in the + table. There are three exceptions: (1) The size may be overridden by + specifying a size in parentheses, (2) the size of addresses is + determined by the Size of Offsets field in the superblock and + is indicated in this document with a superscripted ‘O’, and + (3) the size of length fields is determined by the Size of + Lengths field in the superblock and is indicated in this document with + a superscripted ‘L’. + + +Values for all fields in this document should be treated as + unsigned integers, unless otherwise noted in the description of a + field. Additionally, all metadata fields are stored in little-endian + byte order. + ++ All checksums used in the format are computed with the Jenkins’ + lookup3 algorithm. + + +Whenever a bit flag or field is mentioned for an entry, bits are + numbered from the lowest bit position in the entry. + +Various tables in this document aligned with “This space + inserted only to align table nicely”. These entries in the table + are just to make the table presentation nicer and do not represent any + values or padding in the file. I.B. Changes for HDF5 1.10-As of October 2015, changes in the file format for HDF5 1.10 - have not yet been finalized. +As of October 2015, changes in the file format for HDF5 1.10 have + not yet been finalized. - -II. Disk Format: Level 0 - File Metadata- -- -II.A. Disk Format: Level 0A - Format Signature and Superblock- -The superblock may begin at certain predefined offsets within - the HDF5 file, allowing a block of unspecified content for - users to place additional information at the beginning (and - end) of the HDF5 file without limiting the HDF5 Library’s - ability to manage the objects within the file itself. This - feature was designed to accommodate wrapping an HDF5 file in - another file format or adding descriptive information to an HDF5 - file without requiring the modification of the actual file’s - information. The superblock is located by searching for the - HDF5 format signature at byte offset 0, byte offset 512, and at - successive locations in the file, each a multiple of two of - the previous location; in other words, at these byte offsets: - 0, 512, 1024, 2048, and so on. - -The superblock is composed of the format signature, followed by a - superblock version number and information that is specific to each - version of the superblock. - Currently, there are three versions of the superblock format. - Version 0 is the default format, while version 1 is basically the same - as version 0 with additional information when a non-default B-tree ‘K’ - value is stored. Version 2 is the latest format, with some fields - eliminated or compressed and with superblock extension and checksum - support. - -Version 0 and 1 of the superblock are described below: - - -
-
-
-
-
-
+
+ II. Disk Format: Level 0 - File Metadata +-- Version 2 of the superblock is described below: - -
-
+
+ + II.A. Disk Format: Level 0A - Format + Signature and Superblock +-
The superblock may begin at certain predefined offsets within the + HDF5 file, allowing a block of unspecified content for users to place + additional information at the beginning (and end) of the HDF5 file + without limiting the HDF5 Library’s ability to manage the objects + within the file itself. This feature was designed to accommodate + wrapping an HDF5 file in another file format or adding descriptive + information to an HDF5 file without requiring the modification of the + actual file’s information. The superblock is located by searching + for the HDF5 format signature at byte offset 0, byte offset 512, and at + successive locations in the file, each a multiple of two of the + previous location; in other words, at these byte offsets: 0, 512, 1024, + 2048, and so on. -The superblock is composed of the format signature, followed by a + superblock version number and information that is specific to each + version of the superblock. Currently, there are three versions of the + superblock format. Version 0 is the default format, while version 1 is + basically the same as version 0 with additional information when a + non-default B-tree ‘K’ value is stored. Version 2 is the + latest format, with some fields eliminated or compressed and with + superblock extension and checksum support. --
-
-
- -II.B. Disk Format: Level 0B - File Driver Info+
+
-
-
-
|
- Driver Information Driver information is stored in a format defined by the
- file driver (see description below). |
+
|
+ (Items marked with a ‘1’ in the above table are
+ new in version 1 of the superblock) |
+ - The two drivers encoded in the Driver Identification field are as follows: -
The format of the Driver Information field for the - above two drivers are described below: - -
-
-
-
|
+ Version Number of the Superblock |
+ This value is used to determine the format of the + information in the superblock. When the format of the information + in the superblock is changed, the version number is incremented to + the next integer and can be used to determine how the information + in the superblock is formatted. --
-
+
+ This field is present in version 0+ of the superblock. + -
-
+
|
+ Version Number of the File’s Free Space + Information
+ |
+ This value is used to determine the format of the + file’s free space information. ++ The only value currently valid in this field is ‘0’, + which indicates that the file’s free space is as described below. + -- -II.C. Disk Format: Level 0C - Superblock Extension++ This field is present in version 0 and 1 of the + superblock. + +The superblock extension is used to store superblock metadata - which is either optional, or added after the version of the superblock - was defined. Superblock extensions may only exist when version 2+ of - superblock is used. A superblock extension is an object header which may - hold the following messages: -
|
+ Version Number of the Root Group Symbol Table Entry |
+ This value is used to determine the format of the + information in the Root Group Symbol Table Entry. When the format + of the information in that field is changed, the version number is + incremented to the next integer and can be used to determine how + the information in the field is formatted. ++ The only value currently valid in this field is ‘0’, + which indicates that the root group symbol table entry is formatted + as described below. + ++ This field is present in version 0 and 1 of the + superblock. + |
+ Version Number of the Shared Header Message Format |
+ This value is used to determine the format of the + information in a shared object header message. Since the format of + the shared header messages differs from the other private header + messages, a version number is used to identify changes in the + format. ++ The only value currently valid in this field is ‘0’, + which indicates that shared header messages are formatted as + described below. + ++ This field is present in version 0 and 1 of the + superblock. + |
+ Size of Offsets This value contains the number of bytes used to store + addresses in the file. The values for the addresses of objects in + the file are offsets relative to a base address, usually the + address of the superblock signature. This allows a wrapper to be + added after the file is created without invalidating the internal + offset locations. -- - - -III. Disk Format: Level 1 - File Infrastructure- -- -III.A. Disk Format: Level 1A - B-trees and B-tree Nodes- -B-trees allow flexible storage for objects which tend to grow - in ways that cause the object to be stored discontiguously. B-trees - are described in various algorithms books including “Introduction to - Algorithms” by Thomas H. Cormen, Charles E. Leiserson, and Ronald - L. Rivest. B-trees are used in several places in the HDF5 file format, - when an index is needed for another data structure. - -The version 1 B-tree structure described below is the original index - structure, but are limited by some bugs in our implementation (mainly in - how they handle deleting records). The version 1 B-trees are being phased - out in favor of the version 2 B-trees described below, although both - types of structures may be found in the same file, depending on - application settings when creating the file. - -- -III.A.1. Disk Format: Level 1A1 - Version 1 B-trees (B-link Trees)- -Version 1 B-trees in HDF5 files an implementation of the B-link tree, - in which the sibling nodes at a particular level in the tree are stored - in a doubly-linked list, is described in the “Efficient Locking for - Concurrent Operations on B-trees” paper by Phillip Lehman and S. Bing Yao - as published in the ACM Transactions on Database Systems, - Vol. 6, No. 4, December 1981. - -The B-link trees implemented by the file format contain one more - key than the number of children. In other words, each child - pointer out of a B-tree node has a left key and a right key. - The pointers out of internal nodes point to sub-trees while - the pointers out of leaf nodes point to symbol nodes and - raw data chunks. - Aside from that difference, internal nodes and leaf nodes - are identical. - -
-
+
+
+ Version 2 of the superblock is described below: + +
+
-
-
-
-
|
- Key
- |
+ The format and size of the key values is determined by - the type of data to which this tree points. The keys are - ordered and are boundaries for the contents of the child - pointer; that is, the key values represented by child - N fall between Key N and Key - N+1. Whether the interval is open or closed on - each end is determined by the type of data to which the - tree points. - - -- The format of the key depends on the node type. - For nodes of node type 0 (group nodes), the key is formatted as - follows: - -
- For nodes of node type 1 (chunked raw data nodes), the key is - formatted as follows: - -
|
- Child Pointer
- |
- The tree node contains file addresses of subtrees or - data depending on the node level. Nodes at Level 0 point - to data addresses, either raw data chunks or group nodes. - Nodes at non-zero levels point to other nodes of the - same B-tree. - -For raw data chunk nodes, the child pointer is the address - of a single raw data chunk. For group nodes, the child pointer - points to a symbol table, which contains - information for multiple symbol table entries. - -- Conceptually, each B-tree node looks like this: -
- - where child[i] is a pointer to a sub-tree (at a level - above Level 0) or to data (at Level 0). - Each key[i] describes an item stored by the B-tree - (a chunk or an object of a group node). The range of values - represented by child[i] is indicated by key[i] - and key[i+1]. - - - The following question must next be answered: - “Is the value described by key[i] contained in - child[i-1] or in child[i]?” - The answer depends on the type of tree. - In trees for groups (node type 0) the object described by - key[i] is the greatest object contained in - child[i-1] while in chunk trees (node type 1) the - chunk described by key[i] is the least chunk in - child[i]. - -That means that key[0] for group trees is sometimes unused; - it points to offset zero in the heap, which is always the - empty string and compares as “less-than” any valid object name. - -And key[N] for chunk trees is sometimes unused; - it contains a chunk offset which compares as “greater-than” - any other chunk offset and has a chunk byte size of zero - to indicate that it is not actually allocated. - -- -III.A.2. Disk Format: Level 1A2 - Version 2 B-trees- -Version 2 B-trees are “traditional” B-trees, with one major difference. - Instead of just using a simple pointer (or address in the file) to a - child of an internal node, the pointer to the child node contains two - additional pieces of information: the number of records in the child - node itself, and the total number of records in the child node and - all its descendants. Storing this additional information allows fast - array-like indexing to locate the nth record in the B-tree. - -The entry into a version 2 B-tree is a header which contains global - information about the structure of the B-tree. The root node - address - field in the header points to the B-tree root node, which is either an - internal or leaf node, depending on the value in the header’s - depth field. An internal node consists of records plus - pointers to further leaf or internal nodes in the tree. A leaf node - consists of solely of records. The format of the records depends on - the B-tree type (stored in the header). - -
-
+
Field Name |
+ Description |
+
-
-
-
-
-
- -
-
+
+ + II.B. Disk Format: Level 0B - File Driver + Info ++ ++ The driver information block is an optional region of the file + which contains information needed by the file driver to reopen a file. + The format is described below: + --
-
-
-
- -
-
-
-
-
+
+
The two drivers encoded in the +Driver Identification field are as follows: +
+ The format of the Driver Information field for the above two + drivers are described below: + - |
- Type
- |
- This field is the type of the B-tree node. It should always - be the same as the B-tree type in the header. - -
+
-
-
- The record layout for each stored (in other words, non-testing) - B-tree type is as follows: - -
-
-
-
-
-
-
-
- -
-
+
|
+
- Name of Member File N (variable + size) + -
-
-
-
- -
-
-
|
+
- Size of Member File + -
-
-
|
- Huge Object Length
- |
- The length of the huge object in the file. - -+ + II.C. Disk Format: Level 0C - Superblock + Extension ++ ++ The superblock extension is used to store superblock metadata + which is either optional, or added after the version of the superblock + was defined. Superblock extensions may only exist when version 2+ of + superblock is used. A superblock extension is an object header which + may hold the following messages: + +
- -
-
+
+ + III.A. Disk Format: Level 1A - B-trees and B-tree + Nodes ++ +B-trees allow flexible storage for objects which tend to grow in + ways that cause the object to be stored discontiguously. B-trees are + described in various algorithms books including “Introduction to + Algorithms” by Thomas H. Cormen, Charles E. Leiserson, and Ronald + L. Rivest. B-trees are used in several places in the HDF5 file format, + when an index is needed for another data structure. + +The version 1 B-tree structure described below is the original + index structure, but are limited by some bugs in our implementation + (mainly in how they handle deleting records). The version 1 B-trees are + being phased out in favor of the version 2 B-trees described below, + although both types of structures may be found in the same file, + depending on application settings when creating the file. -
+ + III.A.1. Disk Format: Level 1A1 - Version 1 + B-trees (B-link Trees) ++ ++ Version 1 B-trees in HDF5 files an implementation of the B-link tree, + in which the sibling nodes at a particular level in the tree are stored + in a doubly-linked list, is described in the “Efficient Locking + for Concurrent Operations on B-trees” paper by Phillip Lehman and + S. Bing Yao as published in the ACM Transactions on + Database Systems, Vol. 6, No. 4, December 1981. + -The B-link trees implemented by the file format contain one more + key than the number of children. In other words, each child pointer out + of a B-tree node has a left key and a right key. The pointers out of + internal nodes point to sub-trees while the pointers out of leaf nodes + point to symbol nodes and raw data chunks. Aside from that difference, + internal nodes and leaf nodes are identical. --
-
-
-
- -
-
-
-
-
-
-
-
- -
-
-
-
-
- -
-
+
-
-
+
|
+ Keys and Child Pointers
+ |
+ + Each tree has 2K+1 keys with 2K child pointers + interleaved between the keys. The number of keys and child pointers + actually containing valid values is determined by the node’s + Entries Used field. If that field is N then the + B-link tree contains N child pointers and N+1 + keys. + +- -
-
+
+
Message Location |
- This space inserted only to align table nicely |
- Hash |
+
+ For nodes of node type 1 (chunked raw data nodes), the key is + formatted as follows: +
Reserved (zero) |
- Message Type |
- Object Header Index |
+
+ |
+ Child Pointer
+ |
The tree node contains file addresses of subtrees or data + depending on the node level. Nodes at Level 0 point to data + addresses, either raw data chunks or group nodes. Nodes at non-zero + levels point to other nodes of the same B-tree. ++ For raw data chunk nodes, the child pointer is the address of a + single raw data chunk. For group nodes, the child pointer points to + a symbol table, which contains + information for multiple symbol table entries. + + |
+ Object Header AddressO Conceptually, each B-tree node looks like this: +
where child[ +i] is a pointer to a sub-tree (at a level above Level 0) or to +data (at Level 0). Each key[ +i] describes an +item stored by the B-tree (a chunk or an object of a group node). +The range of values represented by child[ +i] is indicated by key[ +i] and key[ +i+1]. + + + + The following question must next be answered: “Is the value + described by key[i] contained in child[i-1] or in child[i]?” + The answer depends on the type of tree. In trees for groups (node type + 0) the object described by key[i] is the greatest object + contained in child[i-1] while in chunk trees (node type 1) the + chunk described by key[i] is the least chunk in child[i]. + -
That means that key[0] for group trees is sometimes unused; it + points to offset zero in the heap, which is always the empty string and + compares as “less-than” any valid object name. -+ And key[N] for chunk trees is sometimes unused; it contains a + chunk offset which compares as “greater-than” any other + chunk offset and has a chunk byte size of zero to indicate that it is + not actually allocated. + --
-
-
-
- -
-
-
-
-
-
- -
-
-
-
-
+
|
+ Split Percent
+ |
+ The percent full that a node needs to increase above before + it is split. + |
+ Merge Percent
+ |
+ The percent full that a node needs to be decrease below + before it is split. +- -III.B. Disk Format: Level 1B - Group Symbol Table Nodes+ |
+ Root Node Address
+ |
+ + This is the address of the root B-tree node. A B-tree with no + records will have the undefined + address in this field. + +A group is an object internal to the file that allows - arbitrary nesting of objects within the file (including other groups). - A group maps a set of link names in the group to a set of relative - file addresses of objects in the file. Certain metadata for an object to - which the group points can be cached in the group’s symbol table entry in - addition to being in the object’s header. + |
+ Number of Records in Root Node
+ |
+ This is the number of records in the root node. +An HDF5 object name space can be stored hierarchically by - partitioning the name into components and storing each - component as a link in a group. The link for a - non-ultimate component points to the group containing - the next component. The link for the last - component points to the object being named. + |
+ Total Number of Records in B-tree
+ |
+ This is the total number of records in the entire B-tree. +One implementation of a group is a collection of symbol table nodes - indexed by a B-link tree. Each symbol table node contains entries - for one or more links. If an attempt is made to add a link to an already - full symbol table node containing 2K entries, then the node is - split and one node contains K symbols and the other contains - K+1 symbols. + |
+ Checksum
+ |
+ This is the checksum for the B-tree header. +
-
-
-
-
-
-
- -III.C. Disk Format: Level 1C - Symbol Table Entry- -Each symbol table entry in a symbol table node is designed - to allow for very fast browsing of stored objects. - Toward that design goal, the symbol table entries - include space for caching certain constant metadata from the - object header. - -
-
+
+
+
+
-
-
-
-
-
-
- Format of the Scratch-pad Space- -The symbol table entry scratch-pad space is formatted - according to the value in the Cache Type field. - -If the Cache Type field contains the value zero
- If the Cache Type field contains the value one
-
-
+
|
+ Child Node Pointer
+ |
+
- This field is the address of the child node pointed to by the + internal node. +-
-
+
- If the Cache Type field contains the value two
-
-
-
-
+
- -III.D. Disk Format: Level 1D - Local Heaps- -A local heap is a collection of small pieces of data that are particular - to a single object in the HDF5 file. Objects can be - inserted and removed from the heap at any time. - The address of a heap does not change once the heap is created. - For example, a group stores addresses of objects in symbol table nodes - with the names of links stored in the group’s local heap. - - -
-
+
+
+ The record layout for each stored (in other words, non-testing) + B-tree type is as follows: - +
+
-
+
Objects within a local heap should be aligned on an 8-byte boundary. - -- -III.E. Disk Format: Level 1E - Global Heap- -Each HDF5 file has a global heap which stores various types of - information which is typically shared between datasets. The - global heap was designed to satisfy these goals: - -
The implementation of the heap makes use of the memory management - already available at the file level and combines that with a new - object called a collection to achieve goal B. The global heap - is the set of all collections. Each global heap object belongs to - exactly one collection and each collection contains one or more global - heap objects. For the purposes of disk I/O and caching, a collection is - treated as an atomic object, addressing goal A. - - -When a global heap object is deleted from a collection (which occurs - when its reference count falls to zero), objects located after the - deleted object in the collection are packed down toward the beginning - of the collection and the collection’s global heap object 0 is created - (if possible) or its size is increased to account for the recently - freed space. There are no gaps between objects in each collection, - with the possible exception of the final space in the collection, if - it is not large enough to hold the header for the collection’s global - heap object 0. These features address goal C. - - -The HDF5 Library creates global heap collections as needed, so there may - be multiple collections throughout the file. The set of all of them is - abstractly called the “global heap”, although they do not actually link - to each other, and there is no global place in the file where you can - discover all of the collections. The collections are found simply by - finding a reference to one through another object in the file. For - example, data of variable-length datatype elements is stored in the - global heap and is accessed via a global heap ID. The format for - global heap IDs is described at the end of this section. - - -
-
+
++
+
-
|
- Global Heap Object 2 + +
+
-
-
-
-
+
+
+
-
-
- -
-
+ +
+
-
+
+
+
|
+ Huge Object Length
+ |
+ The length of the huge object in the file. +- - The format for the ID used to locate an object in the global heap is - described here: +
-
-
-
-
-
-
- -III.F. Disk Format: Level 1F - Fractal Heap- -- Each fractal heap consists of a header and zero or more direct and - indirect blocks (described below). The header contains general - information as well as - initialization parameters for the doubling table. The Root - Block Address in the header points to the first direct or - indirect block in the heap. - - -- Fractal heaps are based on a data structure called a doubling - table. A doubling table provides a mechanism for quickly - extending an array-like data structure that minimizes the number of - empty blocks in the heap, while retaining very fast lookup of any - element within the array. More information on fractal heaps and - doubling tables can be found in the RFC - “Private - Heaps in HDF5.” - - -- The fractal heap implements the doubling table structure with - indirect and direct blocks. - Indirect blocks in the heap do not actually contain data for - objects in the heap, their “size” is abstract - - they represent the indexing structure for locating the - direct blocks in the doubling table. - Direct blocks - contain the actual data for objects stored in the heap. - - -- All indirect blocks have a constant number of block entries in each - row, called the width of the doubling table (stored in - the heap header). - - The number - of rows for each indirect block in the heap is determined by the - size of the block that the indirect block represents in the - doubling table (calculation of this is shown below) and is - constant, except for the “root” - indirect block, which expands and shrinks its number of rows as - needed. - - -- Blocks in the first two rows of an indirect block - are Starting Block Size number of bytes in size, - and the blocks in each subsequent row are twice the size of - the blocks in the previous row. In other words, blocks in - the third row are twice the Starting Block Size, - blocks in the fourth row are four times the - Starting Block Size, and so on. Entries for - blocks up to the Maximum Direct Block Size point to - direct blocks, and entries for blocks greater than that size - point to further indirect blocks (which have their own - entries for direct and indirect blocks). - - -
- The number of rows of blocks, nrows, in an
- indirect block of size iblock_size is given by the
- following expression:
-
- The maximum number of rows of direct blocks, max_dblock_rows,
- in any indirect block of a fractal heap is given by the
- following expression:
-
- Using the computed values for nrows and
- max_dblock_rows, along with the Width of the
- doubling table, the number of direct and indirect block entries
- (K and N in the indirect block description, below)
- in an indirect block can be computed:
- - The size indirect blocks on disk is determined by the number - of rows in the indirect block (computed above). The size of direct - blocks on disk is exactly the size of the block in the doubling - table. - - -
-
+
+
+ +
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+ +
+
+
+
+
+
|
+ Heap ID
+ |
+
- This is an 8-byte sequence of bytes and is the heap ID for + the shared message in the shared message index’s fractal + heap. ++ +
+
-
+
+
+
- -
-
-
+
+
-
-
-
-
+
+
+ +
+
-
-
- -
-
+
+
+
-
|
- Child Direct Block #1 AddressO |
- Size of Filtered Direct Block #1 (optional)L Filter Mask for Direct Block #1 (optional) |
- ... |
- + + III.B. Disk Format: Level 1B - Group Symbol + Table Nodes ++ +A group is an object internal to the file that allows arbitrary + nesting of objects within the file (including other groups). A group + maps a set of link names in the group to a set of relative file + addresses of objects in the file. Certain metadata for an object to + which the group points can be cached in the group’s symbol table + entry in addition to being in the object’s header. + +An HDF5 object name space can be stored hierarchically by + partitioning the name into components and storing each component as a + link in a group. The link for a non-ultimate component points to the + group containing the next component. The link for the last component + points to the object being named. + ++ One implementation of a group is a collection of symbol table nodes + indexed by a B-link tree. Each symbol table node contains entries for + one or more links. If an attempt is made to add a link to an already + full symbol table node containing 2K entries, then the node is + split and one node contains K symbols and the other contains K+1 + symbols. + - |
- Child Direct Block #K-1 AddressO |
- Size of Filtered Direct Block #K-1 (optional)L Filter Mask for Direct Block #K-1 (optional) |
-
+
-
-
+
+
+ + III.C. Disk Format: Level 1C - Symbol + Table Entry ++ +Each symbol table entry in a symbol table node is designed to + allow for very fast browsing of stored objects. Toward that design + goal, the symbol table entries include space for caching certain + constant metadata from the object header. + +
+
-
-
- An object in the fractal heap is identified by means of a fractal heap ID, - which encodes information to locate the object in the heap. - Currently, the fractal heap stores an object in one of three ways, - depending on the object’s size: - -
-
+
|
+ + Scratch-pad Space (16 bytes) + + The specific format for each type of heap ID is described below: - +
-
+
-
-
-
-
- -
-
-
+ Format of the Scratch-pad Space--
-
+
+ If the Cache Type field contains the value zero
+
+ If the Cache Type field contains the value one
+ - -
-
|
+ (Items marked with an ‘O’ in the
+ above table are of the size specified in “Size of
+ Offsets” field in the superblock.) |
+
+ -
-
-
- -
-
+
+
+
-
+ + III.D. Disk Format: Level 1D - Local Heaps +-
A local heap is a collection of small pieces of data that are + particular to a single object in the HDF5 file. Objects can be inserted + and removed from the heap at any time. The address of a heap does not + change once the heap is created. For example, a group stores addresses + of objects in symbol table nodes with the names of links stored in the + group’s local heap. -
+
+
- -
-
+
+
+
-
-
-
-
-
Objects within a local heap should be aligned on an 8-byte + boundary. -- -
-
-
-
-
-
- -III.G. Disk Format: Level 1G - Free-space Manager- -- Free-space managers are used to describe space within a heap or - the entire HDF5 file that is not currently used for that heap or - file. - - -- The free-space manager header contains metadata information - about the space being tracked, along with the address of the list - of free space sections which actually describes the free - space. The header records information about free-space sections being - tracked, creation parameters for handling free-space sections of a - client, and section information used to locate the collection of - free-space sections. - - -- The free-space section list stores a collection of - free-space sections that is specific to each client of the - free-space manager. - - For example, the fractal heap is a client of the free space manager - and uses it to track unused space within the heap. There are 4 - types of section records for the fractal heap, each of which has - its own format, listed below. - - -
-
+
Number of Section Classes |
- This space inserted only to align table nicely |
- +
+
-
Checksum |
- + +
+
+
-
-
+
|
- Shrink Percent
- |
- This is the percent of current size to shrink the allocated - serialized free-space section list. - -+
+
-
|
- Checksum
- |
- This is the checksum for the free-space manager header. -+ The format for the ID used to locate an object in the global heap + is described here: -- The free-space sections being managed are stored in a - free-space section list, described below. The sections - in the free-space section list are stored in the following way: - a count of the number of sections describing a particular size of - free space and the size of the free-space described (in bytes), - followed by a list of section description records; then another - section count and size, followed by the list of section - descriptions for that size; and so on. - - -
-
-
+
+
-
Record Set #0 Section Record #K-1 Offset(variable size) |
- Record Set #0 Section Record #K-1 Type |
- This space inserted only to align table nicely |
- + + III.F. Disk Format: Level 1F - Fractal Heap ++ ++ Each fractal heap consists of a header and zero or more direct and + indirect blocks (described below). The header contains general + information as well as initialization parameters for the doubling + table. The Root Block Address in the header points to the + first direct or indirect block in the heap. + -Record Set #0 Section Record #K-1 Data (variable size) |
- + Fractal heaps are based on a data structure called a doubling + table. A doubling table provides a mechanism for quickly extending an + array-like data structure that minimizes the number of empty blocks in + the heap, while retaining very fast lookup of any element within the + array. More information on fractal heaps and doubling tables can be + found in the RFC “Private Heaps in + HDF5.” + -Number of Section Records in Set #1 (variable size) |
- The fractal heap implements the doubling table structure with + indirect and direct blocks. Indirect blocks in the heap do not actually + contain data for objects in the heap, their “size” is + abstract - they represent the indexing structure for locating the + direct blocks in the doubling table. Direct blocks contain the actual + data for objects stored in the heap. + ++ All indirect blocks have a constant number of block entries in each + row, called the width of the doubling table (stored in the + heap header). The number of rows for each indirect block in the heap is + determined by the size of the block that the indirect block represents + in the doubling table (calculation of this is shown below) and is + constant, except for the “root” indirect block, which + expands and shrinks its number of rows as needed. + -Size of Free-space Section Described in Record Set #1 (variable size) |
- + Blocks in the first two rows of an indirect block are Starting + Block Size number of bytes in size, and the blocks in each subsequent + row are twice the size of the blocks in the previous row. In other + words, blocks in the third row are twice the Starting Block + Size, blocks in the fourth row are four times the Starting + Block Size, and so on. Entries for blocks up to the Maximum + Direct Block Size point to direct blocks, and entries for blocks + greater than that size point to further indirect blocks (which have + their own entries for direct and indirect blocks). + -Record Set #1 Section Record #0 Offset(variable size) |
-
+ The number of rows of blocks, nrows, in an indirect block of
+ size iblock_size is given by the following expression: Record Set #1 Section Record #0 Type |
- This space inserted only to align table nicely |
-
+ The maximum number of rows of direct blocks, max_dblock_rows,
+ in any indirect block of a fractal heap is given by the following
+ expression: Record Set #1 Section Record #0 Data (variable size) |
-
+ Using the computed values for nrows and max_dblock_rows,
+ along with the Width of the doubling table, the number of
+ direct and indirect block entries (K and N in the
+ indirect block description, below) in an indirect block can be
+ computed: ... |
- The size indirect blocks on disk is determined by the number of + rows in the indirect block (computed above). The size of direct blocks + on disk is exactly the size of the block in the doubling table. -Record Set #1 Section Record #K-1 Offset(variable size) |
-
+
+
|
+ Number of Managed Objects in HeapL +
|
+
- Size of Huge Objects in HeapL + -
-
-
-
Checksum |
+ The defined record types for a fractal heap client are: +
|
- 0Fractal heap “single” section
- |
- +
+
|
- Record Set #N Section #K Data
- |
+ This is the section-type specific information for each record - in the record set, described below. - - |
+ Maximum Size of Managed Objects
+ |
This is the maximum size of managed objects allowed in the + heap. Objects greater than this this are ‘huge’ objects + and will be stored in the file directly, rather than in a direct + block for the heap. + |
- Checksum
- |
+ This is the checksum for the Free-space Section List. - - |
+ Next Huge Object ID
+ |
This is the next ID value to use for a huge object in the + heap. +- - The section-type specific data for each free-space section record is - described below: - - -
-
-
-
- -
-
-
-
- -
-
-
-
- -
-
-
-
- -
-
-
-
-
-
-
- -III.H. Disk Format: Level 1H - Shared Object Header Message Table- -- The shared object header message table is used to locate - object - header messages that are shared between two or more object headers - in the file. Shared object header messages are stored and indexed - in the file in one of two ways: indexed sequentially in a - shared header message list or indexed with a v2 B-tree. - The shared messages themselves are either stored in a fractal - heap (when two or more objects share the message), or remain in an - object’s header (when only one object uses the message currently, - but the message can be shared in the future). - - -- The shared object header message table - contains a list of shared message index headers. Each index header - records information about the version of the index format, the index - storage type, flags for the message types indexed, the number of - messages in the index, the address where the index resides, - and the fractal heap address if shared messages are stored there. - - -- Each index can be either a list or a v2 B-tree and may transition - between those two forms as the number of messages in the index - varies. Each shared message record contains information used to - locate the shared message from either a fractal heap or an object - header. The types of messages that can be shared are: Dataspace, - Datatype, Fill Value, Filter Pipeline and Attribute. - - -- The shared object header message table is pointed to - from a shared message table message - in the superblock extension for a file. This message stores the - version of the table format, along with the number of index headers - in the table. - - -
-
+
+
+ +
+
-
-
Version |
+ This space inserted
+ only to align table nicely |
+
-
-
-
-
- An index can track more than one type of message, but each type - of message can only by in one index. - - - ++
+
+
- - Shared messages are indexed either with a shared message record - list, described below, or using a v2 B-tree (using record type 7). - The number of records in the shared message record list is - determined in the index’s entry in the shared object header message - table. - - -
-
-
-
-
-
- - The record for each shared message in an index is stored in one of the - following forms: - - -
-
-
-
-
-
-
- -
-
+
+
+
-
-
-
-
+
- - - -IV. Disk Format: Level 2 - Data Objects- -Data objects contain the “real” user-visible information in the file. - These objects compose the scientific data and other information which - are generally thought of as “data” by the end-user. All the - other information in the file is provided as a framework for - storing and accessing these data objects. - - -A data object is composed of header and data - information. The header information contains the information - needed to interpret the data information for the object as - well as additional “metadata” or pointers to additional - “metadata” used to describe or annotate each object. - - -- -IV.A. Disk Format: Level 2A - Data Object Headers- -The header information of an object is designed to encompass - all of the information about an object, except for the data itself. - This information includes the dataspace, the datatype, information - about how the data is stored on disk (in external files, compressed, - broken up in blocks, and so on), as well as other information used - by the library to speed up access to the data objects or maintain - a file’s integrity. Information stored by user applications - as attributes is also stored in the object’s header. The header - of each object is not necessarily located immediately prior to the - object’s data in the file and in fact may be located in any - position in the file. The order of the messages in an object header - is not significant. - -Object headers are composed of a prefix and a set of messages. The - prefix contains the information needed to interpret the messages and - a small amount of metadata about the object, and the messages contain - the majority of the metadata about the object. - - -- -IV.A.1. Disk Format: Level 2A1 - Data Object Header Prefix- -- -IV.A.1.a. Version 1 Data Object Header Prefix- -Header messages are aligned on 8-byte boundaries for version 1 - object headers. - - -
-
+
An object in the fractal heap is identified by means of a fractal + heap ID, which encodes information to locate the object in the heap. + Currently, the fractal heap stores an object in one of three ways, + depending on the object’s size: + +
+
-
-
- -IV.A.1.b. Version 2 Data Object Header Prefix- -Note that the “total number of messages” field has been dropped from - the data object header prefix in this version. The number of messages - in the data object header is just determined by the messages encountered - in all the object header blocks. - -Note also that the fields and messages in this version of data object - headers have no alignment or padding bytes inserted - they are - stored packed together. - -
-
+
Huge |
+
+ |
+ When the size of an object is larger than Maximum Size of + Managed Objects in the Fractal Heap Header, the + object’s data is stored on its own in the file and the object + is tracked/indexed via a version 2 B-tree. All huge objects for a + particular fractal heap use the same v2 B-tree. All huge objects + for a particular fractal heap use the same format for their huge + object IDs. + --
-
+
+
The header message types and the message data associated with - them compose the critical “metadata” about each object. Some - header messages are required for each object while others are - optional. Some optional header messages may also be repeated - several times in the header itself, the requirements and number - of times allowed in the header will be noted in each header - message description below. - - - -- -IV.A.2. Disk Format: Level 2A2 - Data Object Header Messages- -Data object header messages are small pieces of metadata that are - stored in the data object header for each object in an HDF5 file. - Data object header messages provide the metadata required to describe - an object and its contents, as well as optional pieces of metadata - that annotate the meaning or purpose of the object. - - -Data object header messages are either stored directly in the data - object header for the object or are shared between multiple objects - in the file. When a message is shared, a flag in the Message Flags - indicates that the actual Message Data - portion of that message is stored in another location (such as another - data object header, or a heap in the file) and the Message Data - field contains the information needed to locate the actual information - for the message. - - -- The format of shared message data is described here: - -
-
Managed |
+
+ |
+ When the size of an object does not meet the above two + conditions, the object is stored and managed via the direct and + indirect blocks based on the doubling table. + | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
- 
- -
| Field Name | -Description | -||||||
|---|---|---|---|---|---|---|---|
Version |
- The version number is used when there are changes in the format - of a shared object message and is described here: -
|
- ||||||
Type |
- The type of shared message location: -
|
- ||||||
Address |
- The address of the object header - containing the message to be shared. - |
-
The specific format for each type of heap ID is described below: +
--
-
| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Type | -This space inserted only to align table nicely | -|
AddressO |
- |||
| - | - (Items marked with an ‘O’ in the above table are of the size - specified in “Size of Offsets” field in the superblock.) - |
-
| Field Name | -Description | -||||
|---|---|---|---|---|---|
Version |
- The version number is used when there are changes in the format - of a shared object message and is described here: -
|
- ||||
Type |
- The type of shared message location: -
|
- ||||
Address |
- The address of the object header - containing the message to be shared. |
-
-
-
| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Type | -This space inserted only to align table nicely | -|
| Location (variable size) | -|||
-
| Field Name | -Description | -||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Version |
- The version number indicates changes in the format of shared - object message and is described here: -
|
- ||||||||||
Type |
- The type of shared message location: -
|
- ||||||||||
Location |
- This field contains either a Size of Offsets-bytes - address of the object header - containing the message to be shared, or an 8-byte fractal heap ID - for the message in the file’s shared object header message - heap. - - |
-
The following is a list of currently defined header messages: -
- --
IV.A.2.a. The NIL Message
- - -| Header Message Name: NIL | |
| Header Message Type: 0x0000 | |
| Length: Varies | |
| Status: Optional; may be repeated. | |
| Description: | -The NIL message is used to indicate a message which is to be - ignored when reading the header messages for a data object. - [Possibly one which has been deleted for some reason.] - |
| Format of Data: Unspecified | |
Data (variable size)
-
IV.A.2.b. The Dataspace Message
- - -| Header Message Name: Dataspace | |
| Header Message Type: 0x0001 | |
| Length: Varies according to the number of - dimensions, as described in the following table. | |
| Status: Required for dataset objects; - may not be repeated. | |
| Description: | -The dataspace message describes the number of dimensions (in - other words, “rank”) and size of each dimension that - the data object has. This message is only used for datasets which - have a simple, rectilinear, array-like layout; datasets requiring - a more complex layout are not yet supported. - | -
| Format of Data: See the tables - below. | |
| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Dimensionality | -Flags | -Reserved | -
| Reserved | -|||
Dimension #1 SizeL |
- |||
| . . . |
- |||
Dimension #n SizeL |
- |||
Dimension #1 Maximum SizeL (optional) |
- |||
| . . . |
- |||
Dimension #n Maximum SizeL (optional) |
- |||
Permutation Index #1L (optional) |
- |||
| . . . |
- |||
Permutation Index #nL (optional) |
- |||
| - | - (Items marked with an ‘L’ in the above table are of the size - specified in “Size of Lengths” field in the superblock.) - |
-
| Field Name | -Description | -
|---|---|
Version |
-
- This value is used to determine the format of the - Dataspace Message. When the format of the - information in the message is changed, the version number - is incremented and can be used to determine how the - information in the object header is formatted. This - document describes version one (1) (there was no version - zero (0)). - - |
-
Dimensionality |
-
- This value is the number of dimensions that the data - object has. - - |
-
Flags |
-
- This field is used to store flags to indicate the - presence of parts of this message. Bit 0 (the least - significant bit) is used to indicate that maximum - dimensions are present. Bit 1 is used to indicate that - permutation indices are present. - - |
-
Dimension #n Size |
-
- This value is the current size of the dimension of the - data as stored in the file. The first dimension stored in - the list of dimensions is the slowest changing dimension - and the last dimension stored is the fastest changing - dimension. - - |
-
Dimension #n Maximum Size |
-
- This value is the maximum size of the dimension of the - data as stored in the file. This value may be the special - “unlimited” size which indicates - that the data may expand along this dimension indefinitely. - If these values are not stored, the maximum size of each - dimension is assumed to be the dimension’s current size. - - |
-
Permutation Index #n |
-
- This value is the index permutation used to map - each dimension from the canonical representation to an - alternate axis for each dimension. If these values are - not stored, the first dimension stored in the list of - dimensions is the slowest changing dimension and the last - dimension stored is the fastest changing dimension. - - |
-
-
Version 2 of the dataspace message dropped the optional - permutation index value support, as it was never implemented in the - HDF5 Library:
- -| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Dimensionality | -Flags | -Type | -
Dimension #1 SizeL |
- |||
| . . . |
- |||
Dimension #n SizeL |
- |||
Dimension #1 Maximum SizeL (optional) |
- |||
| . . . |
- |||
Dimension #n Maximum SizeL (optional) |
- |||
| - | - (Items marked with an ‘L’ in the above table are of the size - specified in “Size of Lengths” field in the superblock.) - |
Version, Type & Length
This is a bit field with the following definition:
+| Bit | +Description | +
|---|
| Field Name | -Description | -||||||||
|---|---|---|---|---|---|---|---|---|---|
Version |
-
- This value is used to determine the format of the - Dataspace Message. This field should be ‘2’ for version 2 - format messages. - - |
- ||||||||
Dimensionality |
-
- This value is the number of dimensions that the data object has. - - |
- ||||||||
Flags |
-
- This field is used to store flags to indicate the - presence of parts of this message. Bit 0 (the least - significant bit) is used to indicate that maximum - dimensions are present. - - |
- ||||||||
Type |
-
- This field indicates the type of the dataspace: -
|
- ||||||||
Dimension #n Size |
-
- This value is the current size of the dimension of the - data as stored in the file. The first dimension stored in - the list of dimensions is the slowest changing dimension - and the last dimension stored is the fastest changing - dimension. - - |
- ||||||||
Dimension #n Maximum Size |
-
- This value is the maximum size of the dimension of the - data as stored in the file. This value may be the special - “unlimited” size which indicates - that the data may expand along this dimension indefinitely. - If these values are not stored, the maximum size of each - dimension is assumed to be the dimension’s current size. - - |
- ||||||||
6-7 |
+ The current version of ID format. This document describes + version 0. | +||||||||
4-5 |
+ The ID type. Tiny objects have a value of 2.
+ |
+ ||||||||
0-3 |
+ The length of the tiny object. The value stored is one + less than the actual length (since zero-length objects are not + allowed to be stored in the heap). For example, an object of + actual length 1 has an encoded length of 0, an object of actual + length 2 has an encoded length of 1, and so on. | +
Data
This is the data for the object.
+-
IV.A.2.c. The Link Info Message
- - -| Header Message Name: Link Info | |
| Header Message Type: 0x002 | |
| Length: Varies | |
| Status: Optional; may not be - repeated. | |
| Description: | -The link info message tracks variable information about the - current state of the links for a “new style” - group’s behavior. Variable information will be stored in - this message and constant information will be stored in the - Group Info message. - |
| Format of Data: See the tables - below. | |
| byte | -byte | -byte | -byte | -||||||
|---|---|---|---|---|---|---|---|---|---|
| Version | -Flags | -This space inserted only to align table nicely | -|||||||
Maximum Creation Index (8 bytes, optional) |
- |||||||||
Fractal Heap AddressO |
- |||||||||
Address of v2 B-tree for Name IndexO |
- |||||||||
Address of v2 B-tree for Creation Order IndexO (optional) |
- |||||||||
| - | - (Items marked with an ‘O’ in the above table are of the size - specified in “Size of Offsets” field in the superblock.) - |
-
| Field Name | -Description | -||||||||
|---|---|---|---|---|---|---|---|---|---|
Version |
-
- The version number for this message. This document describes - version 0. - |
- ||||||||
Flags |
- This field determines various optional aspects of the link - info message: - -
|
- ||||||||
Maximum Creation Index |
- This 64-bit value is the maximum creation order index value - stored for a link in this group. -This field is present if bit 0 of flags is set. - |
- ||||||||
Fractal Heap Address |
-
- - This is the address of the fractal heap to store dense links. - Each link stored in the fractal heap is stored as a - Link Message. - -- If there are no links in the group, or the group’s links - are stored “compactly” (as object header messages), this - value will be the undefined - address. - - |
- ||||||||
Address of v2 B-tree for Name Index |
- This is the address of the version 2 B-tree to index names of links. -If there are no links in the group, or the group’s links - are stored “compactly” (as object header messages), this - value will be the undefined - address. - - |
- ||||||||
Address of v2 B-tree for Creation Order Index |
- This is the address of the version 2 B-tree to index creation order of links. -If there are no links in the group, or the group’s links - are stored “compactly” (as object header messages), this - value will be the undefined - address. - -This field exists if bit 1 of flags is set. - |
- ||||||||
| Version, Type & Length | +Extended Length | +This space inserted + only to align table nicely | +|||||||
-
IV.A.2.d. The Datatype Message
- - -| Header Message Name: Datatype | |
| Header Message Type: 0x0003 - | |
| Length: Variable | |
| Status: Required for dataset or committed - datatype (formerly named datatype) objects; may not be repeated. - | |
| Description: | -The datatype message defines the datatype for each element - of a dataset or a common datatype for sharing between multiple - datasets. A datatype can describe an atomic type like a fixed- - or floating-point type or more complex types like a C struct - (compound datatype), array (array datatype) or C++ vector - (variable-length datatype). -Datatype messages that are part of a dataset object do not - describe how elements are related to one another; the dataspace - message is used for that purpose. Datatype messages that are part of - a committed datatype (formerly named datatype) message describe - a common datatype that can be shared by multiple datasets in the - file. - |
| Format of Data: See the tables - below. | |
| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Class and Version | -Class Bit Field, Bits 0-7 | -Class Bit Field, Bits 8-15 | -Class Bit Field, Bits 16-23 | -
| Size | -|||
Properties |
- |||
| Field Name | +Description | +||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Field Name | -Description | -||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Class and Version |
-
- The version of the datatype message and the datatype’s class - information are packed together in this field. The version - number is packed in the top 4 bits of the field and the class - is contained in the bottom 4 bits. - -The version number information is used for changes in the - format of the datatype message and is described here: + | ||||||||||||||||||
Version, Type & Length |
+
+ This is a bit field with the following definition:
|
The class of the datatype determines the format for the class - bit field and properties portion of the datatype message, which - are described below. The - following classes are currently defined: + + -
| Value | -Description | -
|---|---|
Extended Length |
+
+ This byte, together with the 4 bits in the previous byte, + forms an unsigned 12-bit integer for holding the length of the tiny + object. These 8 bits are bits 0-7 of the 12-bit integer formed. The + value stored is one less than the actual length (since zero-length + objects are not allowed to be stored in the heap). For example, an + object of actual length 1 has an encoded length of 0, an object of + actual length 2 has an encoded length of 1, and so on. + |
+
0 |
- Fixed-Point | -
Data |
+
+ This is the data for the object. + |
+
1 |
- Floating-Point | -
23+
+
4 |
- Bit field | -||
| byte | +byte | +byte | +byte | +
|---|---|---|---|
5 |
- Opaque | -||
| Version & Type | +This space inserted + only to align table nicely | +||
6 |
- Compound | -||
v2 B-tree KeyL + (variable size) + |
+ |||
| + | (Items marked with an ‘L’ in the + above table are of the size specified in “Size of + Lengths” field in the superblock.) | +
+
| Field Name | +Description | +||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Version & Type |
+
+ This is a bit field with the following definition: +
|
v2 B-tree Key
+ This field is the B-tree key for retrieving the information from + the version 2 B-tree for huge objects needed to access the object. + See the description of v2 B-tree records + sub-type 1 & 2 for a description of the fields. New key values are + derived from Next Huge Object ID in the Fractal + Heap Header. +
Class Bit Fields
The information in these bit fields is specific to each datatype - class and is described below. All bits not defined for a - datatype class are set to zero. -
-Size
The size of a datatype element in bytes. -
-+
+
Properties |
-
- This variable-sized sequence of bytes encodes information - specific to each datatype class and is described for each class - below. If there is no property information specified for a - datatype class, the size of this field is zero bytes. - - |
- ||
| byte | +byte | +byte | +byte | +
|---|
Address O
+
-
Class specific information for Fixed-Point Numbers (Class 0):
- -| Bits | -Meaning | -
|---|---|
0 |
- Byte Order. If zero, byte order is little-endian; - otherwise, byte order is big endian. |
-
1, 2 |
- Padding type. Bit 1 is the lo_pad bit and bit 2 - is the hi_pad bit. If a datum has unused bits at either - end, then the lo_pad or hi_pad bit is copied to those - locations. |
-
3 |
- Signed. If this bit is set then the fixed-point - number is in 2’s complement form. |
-
4-23 |
- Reserved (zero). |
-
Length L
+
-
| Byte | -Byte | -Byte | -Byte | -
|---|---|---|---|
| Bit Offset | -Bit Precision | -||
.
+ You will be asked to provide the following information:
+