diff options
Diffstat (limited to 'doc/html/TechNotes/VFL.html')
| -rw-r--r-- | doc/html/TechNotes/VFL.html | 1543 |
1 files changed, 0 insertions, 1543 deletions
diff --git a/doc/html/TechNotes/VFL.html b/doc/html/TechNotes/VFL.html deleted file mode 100644 index 5674cdb..0000000 --- a/doc/html/TechNotes/VFL.html +++ /dev/null @@ -1,1543 +0,0 @@ -<HTML> -<HEAD> -<!-- This HTML file has been created by texi2html 1.51 - from VFL.texi on 18 November 1999 --> - -<TITLE>HDF5 Virtual File Layer</TITLE> -</HEAD> -<BODY> -<H1>HDF5</H1> -<H2>Virtual File Layer</H2> -<H2>Proposal 1999-08-11</H2> -<ADDRESS>Robb Matzke</ADDRESS> -<P> -<P><HR><P> -<H1>Table of Contents</H1> -<UL> -<LI><A NAME="TOC1" HREF="VFL.html#SEC1">Introduction</A> -<LI><A NAME="TOC2" HREF="VFL.html#SEC2">Using a File Driver</A> -<UL> -<LI><A NAME="TOC3" HREF="VFL.html#SEC3">Driver Header Files</A> -<LI><A NAME="TOC4" HREF="VFL.html#SEC4">Creating and Opening Files</A> -<LI><A NAME="TOC5" HREF="VFL.html#SEC5">Performing I/O</A> -<LI><A NAME="TOC6" HREF="VFL.html#SEC6">File Driver Interchangeability</A> -</UL> -<LI><A NAME="TOC7" HREF="VFL.html#SEC7">Implementation of a Driver</A> -<UL> -<LI><A NAME="TOC8" HREF="VFL.html#SEC8">Mode Functions</A> -<LI><A NAME="TOC9" HREF="VFL.html#SEC9">File Functions</A> -<UL> -<LI><A NAME="TOC10" HREF="VFL.html#SEC10">Opening Files</A> -<LI><A NAME="TOC11" HREF="VFL.html#SEC11">Closing Files</A> -<LI><A NAME="TOC12" HREF="VFL.html#SEC12">File Keys</A> -<LI><A NAME="TOC13" HREF="VFL.html#SEC13">Saving Modes Across Opens</A> -</UL> -<LI><A NAME="TOC14" HREF="VFL.html#SEC14">Address Space Functions</A> -<UL> -<LI><A NAME="TOC15" HREF="VFL.html#SEC15">Userblock and Superblock</A> -<LI><A NAME="TOC16" HREF="VFL.html#SEC16">Allocation of Format Regions</A> -<LI><A NAME="TOC17" HREF="VFL.html#SEC17">Freeing Format Regions</A> -<LI><A NAME="TOC18" HREF="VFL.html#SEC18">Querying Address Range</A> -</UL> -<LI><A NAME="TOC19" HREF="VFL.html#SEC19">Data Functions</A> -<UL> -<LI><A NAME="TOC20" HREF="VFL.html#SEC20">Contiguous I/O Functions</A> -<LI><A NAME="TOC21" HREF="VFL.html#SEC21">Flushing Cached Data</A> -</UL> -<LI><A NAME="TOC22" HREF="VFL.html#SEC22">Optimization Functions</A> -<LI><A NAME="TOC23" HREF="VFL.html#SEC23">Registration of a Driver</A> -<LI><A NAME="TOC24" HREF="VFL.html#SEC24">Querying Driver Information</A> -</UL> -<LI><A NAME="TOC25" HREF="VFL.html#SEC25">Miscellaneous</A> -</UL> -<P><HR><P> - - -<H1><A NAME="SEC1" HREF="VFL.html#TOC1">Introduction</A></H1> - -<P> -The HDF5 file format describes how HDF5 data structures and dataset raw -data are mapped to a linear <STRONG>format address space</STRONG> and the HDF5 -library implements that bidirectional mapping in terms of an -API. However, the HDF5 format specifications do <EM>not</EM> indicate how -the format address space is mapped onto storage and HDF (version 5 and -earlier) simply mapped the format address space directly onto a single -file by convention. - -</P> -<P> -Since early versions of HDF5 it became apparent that users want the ability to -map the format address space onto different types of storage (a single file, -multiple files, local memory, global memory, network distributed global -memory, a network protocol, <I>etc</I>.) with various types of maps. For -instance, some users want to be able to handle very large format address -spaces on operating systems that support only 2GB files by partitioning the -format address space into equal-sized parts each served by a separate -file. Other users want the same multi-file storage capability but want to -partition the address space according to purpose (raw data in one file, object -headers in another, global heap in a third, <I>etc.</I>) in order to improve I/O -speeds. - -</P> -<P> -In fact, the number of storage variations is probably larger than the -number of methods that the HDF5 team is capable of implementing and -supporting. Therefore, a <STRONG>Virtual File Layer</STRONG> API is being -implemented which will allow application teams or departments to design -and implement their own mapping between the HDF5 format address space -and storage, with each mapping being a separate <STRONG>file driver</STRONG> -(possibly written in terms of other file drivers). The HDF5 team will -provide a small set of useful file drivers which will also serve as -examples for those who which to write their own: - -</P> -<DL COMPACT> - -<DT><CODE>H5FD_SEC2</CODE> -<DD> -This is the default driver which uses Posix file-system functions like -<CODE>read</CODE> and <CODE>write</CODE> to perform I/O to a single file. All I/O -requests are unbuffered although the driver does optimize file seeking -operations to some extent. - -<DT><CODE>H5FD_STDIO</CODE> -<DD> -This driver uses functions from <TT>`stdio.h'</TT> to perform buffered I/O -to a single file. - -<DT><CODE>H5FD_CORE</CODE> -<DD> -This driver performs I/O directly to memory and can be used to create small -temporary files that never exist on permanent storage. This type of storage is -generally very fast since the I/O consists only of memory-to-memory copy -operations. - -<DT><CODE>H5FD_MPIIO</CODE> -<DD> -This is the driver of choice for accessing files in parallel using MPI and -MPI-IO. It is only predefined if the library is compiled with parallel I/O -support. - -<DT><CODE>H5FD_FAMILY</CODE> -<DD> -Large format address spaces are partitioned into more manageable pieces and -sent to separate storage locations using an underlying driver of the user's -choice. The <CODE>h5repart</CODE> tool can be used to change the sizes of the -family members when stored as files or to convert a family of files to a -single file or vice versa. - -<DT><CODE>H5FD_SPLIT</CODE> -<DD> -The format address space is split into meta data and raw data and each is -mapped onto separate storage using underlying drivers of the user's -choice. The meta data storage can be read by itself (for limited -functionality) or both files can be accessed together. -</DL> - - - -<H1><A NAME="SEC2" HREF="VFL.html#TOC2">Using a File Driver</A></H1> - -<P> -Most application writers will use a driver defined by the HDF5 library or -contributed by another programming team. This chapter describes how existing -drivers are used. - -</P> - - - -<H2><A NAME="SEC3" HREF="VFL.html#TOC3">Driver Header Files</A></H2> - -<P> -Each file driver is defined in its own public header file which should -be included by any application which plans to use that driver. The -predefined drivers are in header files whose names begin with -<SAMP>`H5FD'</SAMP> followed by the driver name and <SAMP>`.h'</SAMP>. The <TT>`hdf5.h'</TT> -header file includes all the predefined driver header files. - -</P> -<P> -Once the appropriate header file is included a symbol of the form -<SAMP>`H5FD_'</SAMP> followed by the upper-case driver name will be the driver -identification number.<A NAME="DOCF1" HREF="VFL.html#FOOT1">(1)</A> However, the -value may change if the library is closed (<I>e.g.</I>, by calling -<CODE>H5close</CODE>) and the symbol is referenced again. - -</P> - - -<H2><A NAME="SEC4" HREF="VFL.html#TOC4">Creating and Opening Files</A></H2> - -<P> -In order to create or open a file one must define the method by which the -storage is accessed<A NAME="DOCF2" HREF="VFL.html#FOOT2">(2)</A> and does so by creating a file access property list<A NAME="DOCF3" HREF="VFL.html#FOOT3">(3)</A> which is passed to the <CODE>H5Fcreate</CODE> or -<CODE>H5Fopen</CODE> function. A default file access property list is created by -calling <CODE>H5Pcreate</CODE> and then the file driver information is inserted by -calling a driver initialization function such as <CODE>H5Pset_fapl_family</CODE>: - -</P> - -<PRE> -hid_t fapl = H5Pcreate(H5P_FILE_ACCESS); -size_t member_size = 100*1024*1024; /*100MB*/ -H5Pset_fapl_family(fapl, member_size, H5P_DEFAULT); -hid_t file = H5Fcreate("foo%05d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl); -H5Pclose(fapl); -</PRE> - -<P> -Each file driver will have its own initialization function -whose name is <CODE>H5Pset_fapl_</CODE> followed by the driver name and which -takes a file access property list as the first argument followed by -additional driver-dependent arguments. - -</P> -<P> -An alternative to using the driver initialization function is to set the -driver directly using the <CODE>H5Pset_driver</CODE> function.<A NAME="DOCF4" HREF="VFL.html#FOOT4">(4)</A> Its second argument is the file driver identifier, which may -have a different numeric value from run to run depending on the order in which -the file drivers are registered with the library. The third argument -encapsulates the additional arguments of the driver initialization -function. This method only works if the file driver writer has made the -driver-specific property list structure a public datatype, which is -often not the case. - -</P> - -<PRE> -hid_t fapl = H5Pcreate(H5P_FILE_ACCESS); -static H5FD_family_fapl_t fa = {100*1024*1024, H5P_DEFAULT}; -H5Pset_driver(fapl, H5FD_FAMILY, &fa); -hid_t file = H5Fcreate("foo.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl); -H5Pclose(fapl); -</PRE> - -<P> -It is also possible to query the file driver information from a file access -property list by calling <CODE>H5Pget_driver</CODE> to determine the driver and then -calling a driver-defined query function to obtain the driver information: - -</P> - -<PRE> -hid_t driver = H5Pget_driver(fapl); -if (H5FD_SEC2==driver) { - /*nothing further to get*/ -} else if (H5FD_FAMILY==driver) { - hid_t member_fapl; - haddr_t member_size; - H5Pget_fapl_family(fapl, &member_size, &member_fapl); -} else if (....) { - .... -} -</PRE> - - - -<H2><A NAME="SEC5" HREF="VFL.html#TOC5">Performing I/O</A></H2> - -<P> -The <CODE>H5Dread</CODE> and <CODE>H5Dwrite</CODE> functions transfer data between -application memory and the file. They both take an optional data transfer -property list which has some general driver-independent properties and -optional driver-defined properties. An application will typically perform I/O -in one of three styles via the <CODE>H5Dread</CODE> or <CODE>H5Dwrite</CODE> function: - -</P> -<P> -Like file access properties in the previous section, data transfer properties -can be set using a driver initialization function or a general purpose -function. For example, to set the MPI-IO driver to use independent access for -I/O operations one would say: - -</P> - -<PRE> -hid_t dxpl = H5Pcreate(H5P_DATA_XFER); -H5Pset_dxpl_mpio(dxpl, H5FD_MPIO_INDEPENDENT); -H5Dread(dataset, type, mspace, fspace, buffer, dxpl); -H5Pclose(dxpl); -</PRE> - -<P> -The alternative is to initialize a driver defined C <CODE>struct</CODE> and pass it -to the <CODE>H5Pset_driver</CODE> function: - -</P> - -<PRE> -hid_t dxpl = H5Pcreate(H5P_DATA_XFER); -static H5FD_mpio_dxpl_t dx = {H5FD_MPIO_INDEPENDENT}; -H5Pset_driver(dxpl, H5FD_MPIO, &dx); -H5Dread(dataset, type, mspace, fspace, buffer, dxpl); -</PRE> - -<P> -The transfer propery list can be queried in a manner similar to the file -access property list: the driver provides a function (or functions) to return -various information about the transfer property list: - -</P> - -<PRE> -hid_t driver = H5Pget_driver(dxpl); -if (H5FD_MPIO==driver) { - H5FD_mpio_xfer_t xfer_mode; - H5Pget_dxpl_mpio(dxpl, &xfer_mode); -} else { - .... -} -</PRE> - - - -<H2><A NAME="SEC6" HREF="VFL.html#TOC6">File Driver Interchangeability</A></H2> - -<P> -The HDF5 specifications describe two things: the mapping of data onto a linear -<STRONG>format address space</STRONG> and the C API which performs the mapping. -However, the mapping of the format address space onto storage intentionally -falls outside the scope of the HDF5 specs. This is a direct result of the fact -that it is not generally possible to store information about how to access -storage inside the storage itself. For instance, given only the file name -<TT>`/arborea/1225/work/f%03d'</TT> the HDF5 library is unable to tell whether the -name refers to a file on the local file system, a family of files on the local -file system, a file on host <SAMP>`arborea'</SAMP> port 1225, a family of files on a -remote system, <I>etc</I>. - -</P> -<P> -Two ways which library could figure out where the storage is located are: -storage access information can be provided by the user, or the library can try -all known file access methods. This implementation uses the former method. - -</P> -<P> -In general, if a file was created with one driver then it isn't possible to -open it with another driver. There are of course exceptions: a file created -with MPIO could probably be opened with the sec2 driver, any file created -by the sec2 driver could be opened as a family of files with one member, -<I>etc</I>. In fact, sometimes a file must not only be opened with the same -driver but also with the same driver properties. The predefined drivers are -written in such a way that specifying the correct driver is sufficient for -opening a file. - -</P> - - -<H1><A NAME="SEC7" HREF="VFL.html#TOC7">Implementation of a Driver</A></H1> - -<P> -A driver is simply a collection of functions and data structures which are -registered with the HDF5 library at runtime. The functions fall into these -categories: - -</P> - -<UL> -<LI>Functions which operate on modes - -<LI>Functions which operate on files - -<LI>Functions which operate on the address space - -<LI>Functions which operate on data - -<LI>Functions for driver initialization - -<LI>Optimization functions - -</UL> - - - -<H2><A NAME="SEC8" HREF="VFL.html#TOC8">Mode Functions</A></H2> - -<P> -Some drivers need information about file access and data transfers which are -very specific to the driver. The information is usually implemented as a pair -of pointers to C structs which are allocated and initialized as part of an -HDF5 property list and passed down to various driver functions. There are two -classes of settings: file access modes that describe how to access the file -through the driver, and data transfer modes which are settings that control -I/O operations. Each file opened by a particular driver may have a different -access mode; each dataset I/O request for a particular file may have a -different data transfer mode. - -</P> -<P> -Since each driver has its own particular requirements for various settings, -each driver is responsible for defining the mode structures that it -needs. Higher layers of the library treat the structures as opaque but must be -able to copy and free them. Thus, the driver provides either the size of the -structure or a pair of function pointers for each of the mode types. - -</P> -<P> -<STRONG>Example:</STRONG> The family driver needs to know how the format address -space is partitioned and the file access property list to use for the -family members. - -</P> - -<PRE> -/* Driver-specific file access properties */ -typedef struct H5FD_family_fapl_t { - hsize_t memb_size; /*size of each member */ - hid_t memb_fapl_id; /*file access property list of each memb*/ -} H5FD_family_fapl_t; - -/* Driver specific data transfer properties */ -typedef struct H5FD_family_dxpl_t { - hid_t memb_dxpl_id; /*data xfer property list of each memb */ -} H5FD_family_dxpl_t; -</PRE> - -<P> -In order to copy or free one of these structures the member file access -or data transfer properties must also be copied or freed. This is done -by providing a copy and close function for each structure: - -</P> -<P> -<STRONG>Example:</STRONG> The file access property list copy and close functions -for the family driver: - -</P> - -<PRE> -static void * -H5FD_family_fapl_copy(const void *_old_fa) -{ - const H5FD_family_fapl_t *old_fa = (const H5FD_family_fapl_t*)_old_fa; - H5FD_family_fapl_t *new_fa = malloc(sizeof(H5FD_family_fapl_t)); - assert(new_fa); - - memcpy(new_fa, old_fa, sizeof(H5FD_family_fapl_t)); - new_fa->memb_fapl_id = H5Pcopy(old_fa->memb_fapl_id); - return new_fa; -} - -static herr_t -H5FD_family_fapl_free(void *_fa) -{ - H5FD_family_fapl_t *fa = (H5FD_family_fapl_t*)_fa; - H5Pclose(fa->memb_fapl_id); - free(fa); - return 0; -} -</PRE> - -<P> -Generally when a file is created or opened the file access properties -for the driver are copied into the file pointer which is returned and -they may be modified from their original value (for instance, the file -family driver modifies the member size property when opening an existing -family). In order to support the <CODE>H5Fget_access_plist</CODE> function the -driver must provide a <CODE>fapl_get</CODE> callback which creates a copy of -the driver-specific properties based on a particular file. - -</P> -<P> -<STRONG>Example:</STRONG> The file family driver copies the member size file -access property list into the return value: - -</P> - -<PRE> -static void * -H5FD_family_fapl_get(H5FD_t *_file) -{ - H5FD_family_t *file = (H5FD_family_t*)_file; - H5FD_family_fapl_t *fa = calloc(1, sizeof(H5FD_family_fapl_t*)); - - fa->memb_size = file->memb_size; - fa->memb_fapl_id = H5Pcopy(file->memb_fapl_id); - return fa; -} -</PRE> - - - -<H2><A NAME="SEC9" HREF="VFL.html#TOC9">File Functions</A></H2> - -<P> -The higher layers of the library expect files to have a name and allow the -file to be accessed in various modes. The driver must be able to create a new -file, replace an existing file, or open an existing file. Opening or creating -a file should return a handle, a pointer to a specialization of the -<CODE>H5FD_t</CODE> struct, which allows read-only or read-write access and which -will be passed to the other driver functions as they are -called.<A NAME="DOCF5" HREF="VFL.html#FOOT5">(5)</A> - -</P> - -<PRE> -typedef struct { - /* Public fields */ - H5FD_class_t *cls; /*class data defined below*/ - - /* Private fields -- driver-defined */ - -} H5FD_t; -</PRE> - -<P> -<STRONG>Example:</STRONG> The family driver requires handles to the underlying -storage, the size of the members for this particular file (which might be -different than the member size specified in the file access property list if -an existing file family is being opened), the name used to open the file in -case additional members must be created, and the flags to use for creating -those additional members. The <CODE>eoa</CODE> member caches the size of the format -address space so the family members don't have to be queried in order to find -it. - -</P> - -<PRE> -/* The description of a file belonging to this driver. */ -typedef struct H5FD_family_t { - H5FD_t pub; /*public stuff, must be first */ - hid_t memb_fapl_id; /*file access property list for members */ - hsize_t memb_size; /*maximum size of each member file */ - int nmembs; /*number of family members */ - int amembs; /*number of member slots allocated */ - H5FD_t **memb; /*dynamic array of member pointers */ - haddr_t eoa; /*end of allocated addresses */ - char *name; /*name generator printf format */ - unsigned flags; /*flags for opening additional members */ -} H5FD_family_t; -</PRE> - -<P> -<STRONG>Example:</STRONG> The sec2 driver needs to keep track of the underlying Unix -file descriptor and also the end of format address space and current Unix file -size. It also keeps track of the current file position and last operation -(read, write, or unknown) in order to optimize calls to <CODE>lseek</CODE>. The -<CODE>device</CODE> and <CODE>inode</CODE> fields are defined on Unix in order to uniquely -identify the file and will be discussed below. - -</P> - -<PRE> -typedef struct H5FD_sec2_t { - H5FD_t pub; /*public stuff, must be first */ - int fd; /*the unix file */ - haddr_t eoa; /*end of allocated region */ - haddr_t eof; /*end of file; current file size*/ - haddr_t pos; /*current file I/O position */ - int op; /*last operation */ - dev_t device; /*file device number */ - ino_t inode; /*file i-node number */ -} H5FD_sec2_t; -</PRE> - - - -<H3><A NAME="SEC10" HREF="VFL.html#TOC10">Opening Files</A></H3> - -<P> -All drivers must define a function for opening/creating a file. This -function should have a prototype which is: - -</P> -<P> -<DL> -<DT><U>Function:</U> static H5FD_t * <B>open</B> <I>(const char *<VAR>name</VAR>, unsigned <VAR>flags</VAR>, hid_t <VAR>fapl</VAR>, haddr_t <VAR>maxaddr</VAR>)</I> -<DD><A NAME="IDX1"></A> - -</P> -<P> -The file name <VAR>name</VAR> and file access property list <VAR>fapl</VAR> are -the same as were specified in the <CODE>H5Fcreate</CODE> or <CODE>H5Fopen</CODE> -call. The <VAR>flags</VAR> are the same as in those calls also except the -flag <CODE>H5F_ACC_CREATE</CODE> is also present if the call was to -<CODE>H5Fcreate</CODE> and they are documented in the <TT>`H5Fpublic.h'</TT> -file. The <VAR>maxaddr</VAR> argument is the maximum format address that the -driver should be prepared to handle (the minimum address is always -zero). -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver opens a Unix file with the requested name -and saves information which uniquely identifies the file (the Unix device -number and inode). - -</P> - -<PRE> -static H5FD_t * -H5FD_sec2_open(const char *name, unsigned flags, hid_t fapl_id/*unused*/, - haddr_t maxaddr) -{ - unsigned o_flags; - int fd; - struct stat sb; - H5FD_sec2_t *file=NULL; - - /* Check arguments */ - if (!name || !*name) return NULL; - if (0==maxaddr || HADDR_UNDEF==maxaddr) return NULL; - if (ADDR_OVERFLOW(maxaddr)) return NULL; - - /* Build the open flags */ - o_flags = (H5F_ACC_RDWR & flags) ? O_RDWR : O_RDONLY; - if (H5F_ACC_TRUNC & flags) o_flags |= O_TRUNC; - if (H5F_ACC_CREAT & flags) o_flags |= O_CREAT; - if (H5F_ACC_EXCL & flags) o_flags |= O_EXCL; - - /* Open the file */ - if ((fd=open(name, o_flags, 0666))<0) return NULL; - if (fstat(fd, &sb)<0) { - close(fd); - return NULL; - } - - /* Create the new file struct */ - file = calloc(1, sizeof(H5FD_sec2_t)); - file->fd = fd; - file->eof = sb.st_size; - file->pos = HADDR_UNDEF; - file->op = OP_UNKNOWN; - file->device = sb.st_dev; - file->inode = sb.st_ino; - - return (H5FD_t*)file; -} -</PRE> - - - -<H3><A NAME="SEC11" HREF="VFL.html#TOC11">Closing Files</A></H3> - -<P> -Closing a file simply means that all cached data should be flushed to the next -lower layer, the file should be closed at the next lower layer, and all -file-related data structures should be freed. All information needed by the -close function is already present in the file handle. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>close</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX2"></A> - -</P> -<P> -The <VAR>file</VAR> argument is the handle which was returned by the <CODE>open</CODE> -function, and the <CODE>close</CODE> should free only memory associated with the -driver-specific part of the handle (the public parts will have already been released by HDF5's virtual file layer). -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver just closes the underlying Unix file, -making sure that the actual file size is the same as that known to the -library by writing a zero to the last file position it hasn't been -written by some previous operation (which happens in the same code which -flushes the file contents and is shown below). - -</P> - -<PRE> -static herr_t -H5FD_sec2_close(H5FD_t *_file) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - - if (H5FD_sec2_flush(_file)<0) return -1; - if (close(file->fd)<0) return -1; - free(file); - return 0; -} -</PRE> - - - -<H3><A NAME="SEC12" HREF="VFL.html#TOC12">File Keys</A></H3> - -<P> -Occasionally an application will attempt to open a single file more than one -time in order to obtain multiple handles to the file. HDF5 allows the files to -share information<A NAME="DOCF6" HREF="VFL.html#FOOT6">(6)</A> but in order to -accomplish this HDF5 must be able to tell when two names refer to the same -file. It does this by associating a driver-defined key with each file opened -by a driver and comparing the key for an open request with the keys for all -other files currently open by the same driver. - -</P> -<P> -<DL> -<DT><U>Function:</U> const int <B>cmp</B> <I>(const H5FD_t *<VAR>f1</VAR>, const H5FD_t *<VAR>f2</VAR>)</I> -<DD><A NAME="IDX3"></A> - -</P> -<P> -The driver may provide a function which compares two files <VAR>f1</VAR> and -<VAR>f2</VAR> belonging to the same driver and returns a negative, positive, or -zero value <I>a la</I> the <CODE>strcmp</CODE> function.<A NAME="DOCF7" HREF="VFL.html#FOOT7">(7)</A> If this -function is not provided then HDF5 assumes that all calls to the <CODE>open</CODE> -callback return unique files regardless of the arguments and it is up to the -application to avoid doing this if that assumption is incorrect. -</DL> - -</P> -<P> -Each time a file is opened the library calls the <CODE>cmp</CODE> function to -compare that file with all other files currently open by the same driver and -if one of them matches (at most one can match) then the file which was just -opened is closed and the previously opened file is used instead. - -</P> -<P> -Opening a file twice with incompatible flags will result in failure. For -instance, opening a file with the truncate flag is a two step process which -first opens the file without truncation so keys can be compared, and if no -matching file is found already open then the file is closed and immediately -reopened with the truncation flag set (if a matching file is already open then -the truncating open will fail). - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver uses the Unix device and i-node as the -key. They were initialized when the file was opened. - -</P> - -<PRE> -static int -H5FD_sec2_cmp(const H5FD_t *_f1, const H5FD_t *_f2) -{ - const H5FD_sec2_t *f1 = (const H5FD_sec2_t*)_f1; - const H5FD_sec2_t *f2 = (const H5FD_sec2_t*)_f2; - - if (f1->device < f2->device) return -1; - if (f1->device > f2->device) return 1; - - if (f1->inode < f2->inode) return -1; - if (f1->inode > f2->inode) return 1; - - return 0; -} -</PRE> - - - -<H3><A NAME="SEC13" HREF="VFL.html#TOC13">Saving Modes Across Opens</A></H3> - -<P> -Some drivers may also need to store certain information in the file superblock -in order to be able to reliably open the file at a later date. This is done by -three functions: one to determine how much space will be necessary to store -the information in the superblock, one to encode the information, and one to -decode the information. These functions are optional, but if any one is -defined then the other two must also be defined. - -</P> -<P> -<DL> -<DT><U>Function:</U> static hsize_t <B>sb_size</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX4"></A> -<DT><U>Function:</U> static herr_t <B>sb_encode</B> <I>(H5FD_t *<VAR>file</VAR>, char *<VAR>name</VAR>, unsigned char *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX5"></A> -<DT><U>Function:</U> static herr_t <B>sb_decode</B> <I>(H5FD_t *<VAR>file</VAR>, const char *<VAR>name</VAR>, const unsigned char *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX6"></A> - -</P> -<P> -The <CODE>sb_size</CODE> function returns the number of bytes necessary to encode -information needed later if the file is reopened. The <CODE>sb_encode</CODE> -function encodes information from the file into buffer <VAR>buf</VAR> -allocated by the caller. It also writes an 8-character (plus null -termination) into the <CODE>name</CODE> argument, which should be a unique -identification for the driver. The <CODE>sb_decode</CODE> function looks at -the <VAR>name</VAR> - -</P> -<P> - decodes -data from the buffer <VAR>buf</VAR> and updates the <VAR>file</VAR> argument with the new information, -advancing <VAR>*p</VAR> in the process. -</DL> - -</P> -<P> -The part of this which is somewhat tricky is that the file must be readable -before the superblock information is decoded. File access modes fall outside -the scope of the HDF5 file format, but they are placed inside the boot block -for convenience.<A NAME="DOCF8" HREF="VFL.html#FOOT8">(8)</A> - -</P> -<P> -<STRONG>Example:</STRONG> <EM>To be written later.</EM> - -</P> - - -<H2><A NAME="SEC14" HREF="VFL.html#TOC14">Address Space Functions</A></H2> - -<P> -HDF5 does not assume that a file is a linear address space of bytes. Instead, -the library will call functions to allocate and free portions of the HDF5 -format address space, which in turn map onto functions in the file driver to -allocate and free portions of file address space. The library tells the file -driver how much format address space it wants to allocate and the driver -decides what format address to use and how that format address is mapped onto -the file address space. Usually the format address is chosen so that the file -address can be calculated in constant time for data I/O operations (which are -always specified by format addresses). - -</P> - - - -<H3><A NAME="SEC15" HREF="VFL.html#TOC15">Userblock and Superblock</A></H3> - -<P> -The HDF5 format allows an optional userblock to appear before the actual HDF5 -data in such a way that if the userblock is <STRONG>sucked out</STRONG> of the file and -everything remaining is shifted downward in the file address space, then the -file is still a valid HDF5 file. The userblock size can be zero or any -multiple of two greater than or equal to 512 and the file superblock begins -immediately after the userblock. - -</P> -<P> -HDF5 allocates space for the userblock and superblock by calling an -allocation function defined below, which must return a chunk of memory at -format address zero on the first call. - -</P> - - -<H3><A NAME="SEC16" HREF="VFL.html#TOC16">Allocation of Format Regions</A></H3> - -<P> -The library makes many types of allocation requests: - -</P> -<DL COMPACT> - -<DT><CODE>H5FD_MEM_SUPER</CODE> -<DD> -An allocation request for the userblock and/or superblock. -<DT><CODE>H5FD_MEM_BTREE</CODE> -<DD> -An allocation request for a node of a B-tree. -<DT><CODE>H5FD_MEM_DRAW</CODE> -<DD> -An allocation request for the raw data of a dataset. -<DT><CODE>H5FD_MEM_META</CODE> -<DD> -An allocation request for the raw data of a dataset which -the user has indicated will be relatively small. -<DT><CODE>H5FD_MEM_GROUP</CODE> -<DD> -An allocation request for a group leaf node (internal nodes of the group tree -are allocated as H5MF_BTREE). -<DT><CODE>H5FD_MEM_GHEAP</CODE> -<DD> -An allocation request for a global heap collection. Global heaps are used to -store certain types of references such as dataset region references. The set -of all global heap collections can become quite large. -<DT><CODE>H5FD_MEM_LHEAP</CODE> -<DD> -An allocation request for a local heap. Local heaps are used to store the -names which are members of a group. The combined size of all local heaps is a -function of the number of object names in the file. -<DT><CODE>H5FD_MEM_OHDR</CODE> -<DD> -An allocation request for (part of) an object header. Object headers are -relatively small and include meta information about objects (like the data -space and type of a dataset) and attributes. -</DL> - -<P> -When a chunk of memory is freed the library adds it to a free list and -allocation requests are satisfied from the free list before requesting memory -from the file driver. Each type of allocation request enumerated above has its -own free list, but the file driver can specify that certain object types can -share a free list. It does so by providing an array which maps a request type -to a free list. If any value of the map is <CODE>H5MF_DEFAULT</CODE> (zero) then the -object's own free list is used. The special value <CODE>H5MF_NOLIST</CODE> indicates -that the library should not attempt to maintain a free list for that -particular object type, instead calling the file driver each time an object of -that type is freed. - -</P> -<P> -Mappings predefined in the <TT>`H5FDpublic.h'</TT> file are: -<DL COMPACT> - -<DT><CODE>H5FD_FLMAP_SINGLE</CODE> -<DD> -All memory usage types are mapped to a single free list. -<DT><CODE>H5FD_FLMAP_DICHOTOMY</CODE> -<DD> -Memory usage is segregated into meta data and raw data for the purposes of -memory management. -<DT><CODE>H5FD_FLMAP_DEFAULT</CODE> -<DD> -Each memory usage type has its own free list. -</DL> - -<P> -<STRONG>Example:</STRONG> To make a map that manages object headers on one free list -and everything else on another free list one might initialize the map with the -following code: (the use of <CODE>H5FD_MEM_SUPER</CODE> is arbitrary) - -</P> - -<PRE> -H5FD_mem_t mt, map[H5FD_MEM_NTYPES]; - -for (mt=0; mt<H5FD_MEM_NTYPES; mt++) { - map[mt] = (H5FD_MEM_OHDR==mt) ? mt : H5FD_MEM_SUPER; -} -</PRE> - -<P> -If an allocation request cannot be satisfied from the free list then one of -two things happen. If the driver defines an allocation callback then it is -used to allocate space; otherwise new memory is allocated from the end of the -format address space by incrementing the end-of-address marker. - -</P> -<P> -<DL> -<DT><U>Function:</U> static haddr_t <B>alloc</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, hsize_t <VAR>size</VAR>)</I> -<DD><A NAME="IDX7"></A> - -</P> -<P> -The <VAR>file</VAR> argument is the file from which space is to be allocated, -<VAR>type</VAR> is the type of memory being requested (from the list above) without -being mapped according to the freelist map and <VAR>size</VAR> is the number of -bytes being requested. The library is allowed to allocate large chunks of -storage and manage them in a layer above the file driver (although the current -library doesn't do that). The allocation function should return a format -address for the first byte allocated. The allocated region extends from that -address for <VAR>size</VAR> bytes. If the request cannot be honored then the -undefined address value is returned (<CODE>HADDR_UNDEF</CODE>). The first call to -this function for a file which has never had memory allocated <EM>must</EM> -return a format address of zero or <CODE>HADDR_UNDEF</CODE> since this is how the -library allocates space for the userblock and/or superblock. -</DL> - -</P> - -<P> -<STRONG>Example:</STRONG> <EM>To be written later.</EM> - -</P> - - -<H3><A NAME="SEC17" HREF="VFL.html#TOC17">Freeing Format Regions</A></H3> - -<P> -When the library is finished using a certain region of the format address -space it will return the space to the free list according to the type of -memory being freed and the free list map described above. If the free list has -been disabled for a particular memory usage type (according to the free list -map) and the driver defines a <CODE>free</CODE> callback then it will be -invoked. The <CODE>free</CODE> callback is also invoked for all entries on the free -list when the file is closed. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>free</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>)</I> -<DD><A NAME="IDX8"></A> - -</P> -<P> -The <VAR>file</VAR> argument is the file for which space is being freed; <VAR>type</VAR> -is the type of object being freed (from the list above) without being mapped -according to the freelist map; <VAR>addr</VAR> is the first format address to free; -and <VAR>size</VAR> is the size in bytes of the region being freed. The region -being freed may refer to just part of the region originally allocated and/or -may cross allocation boundaries provided all regions being freed have the same -usage type. However, the library will never attempt to free regions which have -already been freed or which have never been allocated. -</DL> - -</P> -<P> -A driver may choose to not define the <CODE>free</CODE> function, in which case -format addresses will be leaked. This isn't normally a huge problem since the -library contains a simple free list of its own and freeing parts of the format -address space is not a common occurrence. - -</P> -<P> -<STRONG>Example:</STRONG> <EM>To be written later.</EM> - -</P> - - -<H3><A NAME="SEC18" HREF="VFL.html#TOC18">Querying Address Range</A></H3> - -<P> -Each file driver must have some mechanism for setting and querying the end of -address, or <STRONG>EOA</STRONG>, marker. The EOA marker is the first format address -after the last format address ever allocated. If the last part of the -allocated address range is freed then the driver may optionally decrease the -eoa marker. - -</P> -<P> -<DL> -<DT><U>Function:</U> static haddr_t <B>get_eoa</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX9"></A> - -</P> -<P> -This function returns the current value of the EOA marker for the specified -file. -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver just returns the current eoa marker value -which is cached in the file structure: - -</P> - -<PRE> -static haddr_t -H5FD_sec2_get_eoa(H5FD_t *_file) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - return file->eoa; -} -</PRE> - -<P> -The eoa marker is initially zero when a file is opened and the library may set -it to some other value shortly after the file is opened (after the superblock -is read and the saved eoa marker is determined) or when allocating additional -memory in the absence of an <CODE>alloc</CODE> callback (described above). - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver simply caches the eoa marker in the file -structure and does not extend the underlying Unix file. When the file is -flushed or closed then the Unix file size is extended to match the eoa marker. - -</P> - -<PRE> -static herr_t -H5FD_sec2_set_eoa(H5FD_t *_file, haddr_t addr) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - file->eoa = addr; - return 0; -} -</PRE> - - - -<H2><A NAME="SEC19" HREF="VFL.html#TOC19">Data Functions</A></H2> - -<P> -These functions operate on data, transferring a region of the format address -space between memory and files. - -</P> - - - -<H3><A NAME="SEC20" HREF="VFL.html#TOC20">Contiguous I/O Functions</A></H3> - -<P> -A driver must specify two functions to transfer data from the library to the -file and vice versa. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>read</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, void *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX10"></A> -<DT><U>Function:</U> static herr_t <B>write</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, const void *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX11"></A> - -</P> -<P> -The <CODE>read</CODE> function reads data from file <VAR>file</VAR> beginning at address -<VAR>addr</VAR> and continuing for <VAR>size</VAR> bytes into the buffer <VAR>buf</VAR> -supplied by the caller. The <CODE>write</CODE> function transfers data in the -opposite direction. Both functions take a data transfer property list -<VAR>dxpl</VAR> which indicates the fine points of how the data is to be -transferred and which comes directly from the <CODE>H5Dread</CODE> or -<CODE>H5Dwrite</CODE> function. Both functions receive <VAR>type</VAR> of -data being written, which may allow a driver to tune it's behavior for -different kinds of data. -</DL> - -</P> -<P> -Both functions should return a negative value if they fail to transfer the -requested data, or non-negative if they succeed. The library will never -attempt to read from unallocated regions of the format address space. - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver just makes system calls. It tries not to -call <CODE>lseek</CODE> if the current operation is the same as the previous -operation and the file position is correct. It also fills the output buffer -with zeros when reading between the current EOF and EOA markers and restarts -system calls which were interrupted. - -</P> - -<PRE> -static herr_t -H5FD_sec2_read(H5FD_t *_file, H5FD_mem_t type/*unused*/, hid_t dxpl_id/*unused*/, - haddr_t addr, hsize_t size, void *buf/*out*/) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - ssize_t nbytes; - - assert(file && file->pub.cls); - assert(buf); - - /* Check for overflow conditions */ - if (REGION_OVERFLOW(addr, size)) return -1; - if (addr+size>file->eoa) return -1; - - /* Seek to the correct location */ - if ((addr!=file->pos || OP_READ!=file->op) && - file_seek(file->fd, (file_offset_t)addr, SEEK_SET)<0) { - file->pos = HADDR_UNDEF; - file->op = OP_UNKNOWN; - return -1; - } - - /* - * Read data, being careful of interrupted system calls, partial results, - * and the end of the file. - */ - while (size>0) { - do nbytes = read(file->fd, buf, size); - while (-1==nbytes && EINTR==errno); - if (-1==nbytes) { - /* error */ - file->pos = HADDR_UNDEF; - file->op = OP_UNKNOWN; - return -1; - } - if (0==nbytes) { - /* end of file but not end of format address space */ - memset(buf, 0, size); - size = 0; - } - assert(nbytes>=0); - assert((hsize_t)nbytes<=size); - size -= (hsize_t)nbytes; - addr += (haddr_t)nbytes; - buf = (char*)buf + nbytes; - } - - /* Update current position */ - file->pos = addr; - file->op = OP_READ; - return 0; -} -</PRE> - -<P> -<STRONG>Example:</STRONG> The sec2 <CODE>write</CODE> callback is similar except it updates -the file EOF marker when extending the file. - -</P> - - -<H3><A NAME="SEC21" HREF="VFL.html#TOC21">Flushing Cached Data</A></H3> - -<P> -Some drivers may desire to cache data in memory in order to make larger I/O -requests to the underlying file and thus improving bandwidth. Such drivers -should register a cache flushing function so that the library can insure that -data has been flushed out of the drivers in response to the application -calling <CODE>H5Fflush</CODE>. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>flush</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX12"></A> - -</P> -<P> -Flush all data for file <VAR>file</VAR> to storage. -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver doesn't cache any data but it also doesn't -extend the Unix file as agressively as it should. Therefore, when finalizing a -file it should write a zero to the last byte of the allocated region so that -when reopening the file later the EOF marker will be at least as large as the -EOA marker saved in the superblock (otherwise HDF5 will refuse to open the -file, claiming that the data appears to be truncated). - -</P> - -<PRE> -static herr_t -H5FD_sec2_flush(H5FD_t *_file) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - - if (file->eoa>file->eof) { - if (-1==file_seek(file->fd, file->eoa-1, SEEK_SET)) return -1; - if (write(file->fd, "", 1)!=1) return -1; - file->eof = file->eoa; - file->pos = file->eoa; - file->op = OP_WRITE; - } - - return 0; -} -</PRE> - - - -<H2><A NAME="SEC22" HREF="VFL.html#TOC22">Optimization Functions</A></H2> - -<P> -The library is capable of performing several generic optimizations on I/O, but -these types of optimizations may not be appropriate for a given VFL driver. -</P> - -<P> -Each driver may provide a query function to allow the library to query whether -to enable these optimizations. If a driver lacks a query function, the library -will disable all types of optimizations which can be queried. -</P> - -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>query</B> <I>(const H5FD_t *<VAR>file</VAR>, unsigned long *<VAR>flags</VAR>)</I> -<DD><A NAME="IDX17"></A> -</P> -<P> -This function is called by the library to query which optimizations to enable -for I/O to this driver. These are the flags which are currently defined: - -<UL> -<DL> -<DT>H5FD_FEAT_AGGREGATE_METADATA (0x00000001) -<DD>Defining the H5FD_FEAT_AGGREGATE_METADATA for a VFL driver means that -the library will attempt to allocate a larger block for metadata and -then sub-allocate each metadata request from that larger block. -<DT>H5FD_FEAT_ACCUMULATE_METADATA (0x00000002) -<DD>Defining the H5FD_FEAT_ACCUMULATE_METADATA for a VFL driver means that -the library will attempt to cache metadata as it is written to the file -and build up a larger block of metadata to eventually pass to the VFL -'write' routine. -<DT>H5FD_FEAT_DATA_SIEVE (0x00000004) -<DD>Defining the H5FD_FEAT_DATA_SIEVE for a VFL driver means that -the library will attempt to cache raw data as it is read from/written to -a file in a "data sieve" buffer. See Rajeev Thakur's papers: - <UL> - <DL> - <DT>http://www.mcs.anl.gov/~thakur/papers/romio-coll.ps.gz - <DT>http://www.mcs.anl.gov/~thakur/papers/mpio-high-perf.ps.gz - </DL> - </UL> -</DL> -</UL> -</P> - -</DL> -</P> - -<H2><A NAME="SEC23" HREF="VFL.html#TOC23">Registration of a Driver</A></H2> - -<P> -Before a driver can be used the HDF5 library needs to be told of its -existence. This is done by registering the driver, which results in a driver -identification number. Instead of passing many arguments to the registration -function, the driver information is entered into a structure and the address -of the structure is passed to the registration function where it is -copied. This allows the HDF5 API to be extended while providing backward -compatibility at the source level. - -</P> -<P> -<DL> -<DT><U>Function:</U> hid_t <B>H5FDregister</B> <I>(H5FD_class_t *<VAR>cls</VAR>)</I> -<DD><A NAME="IDX13"></A> - -</P> -<P> -The driver described by struct <VAR>cls</VAR> is registered with the library and an -ID number for the driver is returned. -</DL> - -</P> -<P> -The <CODE>H5FD_class_t</CODE> type is a struct with the following fields: - -</P> -<DL COMPACT> - -<DT><CODE>const char *name</CODE> -<DD> -A pointer to a constant, null-terminated driver name to be used for debugging -purposes. -<DT><CODE>size_t fapl_size</CODE> -<DD> -The size in bytes of the file access mode structure or zero if the driver -supplies a copy function or doesn't define the structure. -<DT><CODE>void *(*fapl_copy)(const void *fapl)</CODE> -<DD> -An optional function which copies a driver-defined file access mode structure. -This field takes precedence over <CODE>fm_size</CODE> when both are defined. -<DT><CODE>void (*fapl_free)(void *fapl)</CODE> -<DD> -An optional function to free the driver-defined file access mode structure. If -null, then the library calls the C <CODE>free</CODE> function to free the -structure. -<DT><CODE>size_t dxpl_size</CODE> -<DD> -The size in bytes of the data transfer mode structure or zero if the driver -supplies a copy function or doesn't define the structure. -<DT><CODE>void *(*dxpl_copy)(const void *dxpl)</CODE> -<DD> -An optional function which copies a driver-defined data transfer mode -structure. This field takes precedence over <CODE>xm_size</CODE> when both are -defined. -<DT><CODE>void (*dxpl_free)(void *dxpl)</CODE> -<DD> -An optional function to free the driver-defined data transfer mode -structure. If null, then the library calls the C <CODE>free</CODE> function to -free the structure. -<DT><CODE>H5FD_t *(*open)(const char *name, unsigned flags, hid_t fapl, haddr_t maxaddr)</CODE> -<DD> -The function which opens or creates a new file. -<DT><CODE>herr_t (*close)(H5FD_t *file)</CODE> -<DD> -The function which ends access to a file. -<DT><CODE>int (*cmp)(const H5FD_t *f1, const H5FD_t *f2)</CODE> -<DD> -An optional function to determine whether two open files have the same key. If -this function is not present then the library assumes that two files will -never be the same. -<DT><CODE>int (*query)(const H5FD_t *f, unsigned long *flags)</CODE> -<DD> -An optional function to determine which library optimizations a driver can -support. -<DT><CODE>haddr_t (*alloc)(H5FD_t *file, H5FD_mem_t type, hsize_t size)</CODE> -<DD> -An optional function to allocate space in the file. -<DT><CODE>herr_t (*free)(H5FD_t *file, H5FD_mem_t type, haddr_t addr, hsize_t size)</CODE> -<DD> -An optional function to free space in the file. -<DT><CODE>haddr_t (*get_eoa)(H5FD_t *file)</CODE> -<DD> -A function to query how much of the format address space has been allocated. -<DT><CODE>herr_t (*set_eoa)(H5FD_t *file, haddr_t)</CODE> -<DD> -A function to set the end of address space. -<DT><CODE>haddr_t (*get_eof)(H5FD_t *file)</CODE> -<DD> -A function to return the current end-of-file marker value. -<DT><CODE>herr_t (*read)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, void *buffer)</CODE> -<DD> -A function to read data from a file. -<DT><CODE>herr_t (*write)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, const void *buffer)</CODE> -<DD> -A function to write data to a file. -<DT><CODE>herr_t (*flush)(H5FD_t *file)</CODE> -<DD> -A function which flushes cached data to the file. -<DT><CODE>H5FD_mem_t fl_map[H5FD_MEM_NTYPES]</CODE> -<DD> -An array which maps a file allocation request type to a free list. -</DL> - -<P> -<STRONG>Example:</STRONG> The sec2 driver would be registered as: - -</P> - -<PRE> -static const H5FD_class_t H5FD_sec2_g = { - "sec2", /*name */ - MAXADDR, /*maxaddr */ - NULL, /*sb_size */ - NULL, /*sb_encode */ - NULL, /*sb_decode */ - 0, /*fapl_size */ - NULL, /*fapl_get */ - NULL, /*fapl_copy */ - NULL, /*fapl_free */ - 0, /*dxpl_size */ - NULL, /*dxpl_copy */ - NULL, /*dxpl_free */ - H5FD_sec2_open, /*open */ - H5FD_sec2_close, /*close */ - H5FD_sec2_cmp, /*cmp */ - H5FD_sec2_query, /*query */ - NULL, /*alloc */ - NULL, /*free */ - H5FD_sec2_get_eoa, /*get_eoa */ - H5FD_sec2_set_eoa, /*set_eoa */ - H5FD_sec2_get_eof, /*get_eof */ - H5FD_sec2_read, /*read */ - H5FD_sec2_write, /*write */ - H5FD_sec2_flush, /*flush */ - H5FD_FLMAP_SINGLE, /*fl_map */ -}; - -hid_t -H5FD_sec2_init(void) -{ - if (!H5FD_SEC2_g) { - H5FD_SEC2_g = H5FDregister(&H5FD_sec2_g); - } - return H5FD_SEC2_g; -} -</PRE> - -<P> -A driver can be removed from the library by unregistering it - -</P> -<P> -<DL> -<DT><U>Function:</U> herr_t <B>H5Dunregister</B> <I>(hid_t <VAR>driver</VAR>)</I> -<DD><A NAME="IDX14"></A> -Where <VAR>driver</VAR> is the ID number returned when the driver was registered. -</DL> - -</P> -<P> -Unregistering a driver makes it unusable for creating new file access or data -transfer property lists but doesn't affect any property lists or files that -already use that driver. - -</P> - - - -<H2><A NAME="SEC24" HREF="VFL.html#TOC24">Querying Driver Information</A></H2> - -<P> -<DL> -<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fapl</VAR>)</I> -<DD><A NAME="IDX15"></A> -<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fxpl</VAR>)</I> -<DD><A NAME="IDX16"></A> - -</P> -<P> -This function is intended to be used by driver functions, not applications. -It returns a pointer directly into the file access property list -<CODE><VAR>fapl</VAR></CODE> which is a copy of the driver's file access mode originally -provided to the <CODE>H5Pset_driver</CODE> function. If its argument is a data -transfer property list <CODE>fxpl</CODE> then it returns a pointer to the -driver-specific data transfer information instead. -</DL> - -</P> - - -<H1><A NAME="SEC25" HREF="VFL.html#TOC25">Miscellaneous</A></H1> - -<P> -The various private <CODE>H5F_low_*</CODE> functions will be replaced by public -<CODE>H5FD*</CODE> functions so they can be called from drivers. - -</P> -<P> -All private functions <CODE>H5F_addr_*</CODE> which operate on addresses will be -renamed as public functions by removing the first underscore so they can be -called by drivers. - -</P> -<P> -The <CODE>haddr_t</CODE> address data type will be passed by value throughout the -library. The original intent was that this type would eventually be a union of -file address types for the various drivers and may become quite large, but -that was back when drivers were part of HDF5. It will become an alias for an -unsigned integer type (32 or 64 bits depending on how the library was -configured). - -</P> -<P> -The various <CODE>H5F*.c</CODE> driver files will be renamed <CODE>H5FD*.c</CODE> and each -will have a corresponding header file. All driver functions except the -initializer and API will be declared static. - -</P> -<P> -This documentation didn't cover optimization functions which would be useful -to drivers like MPI-IO. Some drivers may be able to perform data pipeline -operations more efficiently than HDF5 and need to be given a chance to -override those parts of the pipeline. The pipeline would be designed to call -various H5FD optimization functions at various points which return one of -three values: the operation is not implemented by the driver, the operation is -implemented but failed in a non-recoverable manner, the operation is -implemented and succeeded. - -</P> -<P> -Various parts of HDF5 check the only the top-level file driver and do -something special if it is the MPI-IO driver. However, we might want to be -able to put the MPI-IO driver under other drivers such as the raw part of a -split driver or under a debug driver whose sole purpose is to accumulate -statistics as it passes all requests through to the MPI-IO driver. Therefore -we will probably need a function which takes a format address and or object -type and returns the driver which would have been used at the lowest level to -process the request. - -</P> - -<P><HR><P> -<H1>Footnotes</H1> -<H3><A NAME="FOOT1" HREF="VFL.html#DOCF1">(1)</A></H3> -<P>The driver name is by convention and might -not apply to drivers which are not distributed with HDF5. -<H3><A NAME="FOOT2" HREF="VFL.html#DOCF2">(2)</A></H3> -<P>The access method also indicates how to translate -the storage name to a storage server such as a file, network protocol, or -memory. -<H3><A NAME="FOOT3" HREF="VFL.html#DOCF3">(3)</A></H3> -<P>The term -"<EM>file</EM> access property list" is a misnomer since storage isn't -required to be a file. -<H3><A NAME="FOOT4" HREF="VFL.html#DOCF4">(4)</A></H3> -<P>This -function is overloaded to operate on data transfer property lists also, as -described below. -<H3><A NAME="FOOT5" HREF="VFL.html#DOCF5">(5)</A></H3> -<P>Read-only access is only appropriate when opening an existing -file. -<H3><A NAME="FOOT6" HREF="VFL.html#DOCF6">(6)</A></H3> -<P>For instance, writing data to one handle will cause -the data to be immediately visible on the other handle. -<H3><A NAME="FOOT7" HREF="VFL.html#DOCF7">(7)</A></H3> -<P>The ordering is -arbitrary as long as it's consistent within a particular file driver. -<H3><A NAME="FOOT8" HREF="VFL.html#DOCF8">(8)</A></H3> -<P>File access modes do not describe data, but rather -describe how the HDF5 format address space is mapped to the underlying -file(s). Thus, in general the mapping must be known before the file superblock -can be read. However, the user usually knows enough about the mapping for the -superblock to be readable and once the superblock is read the library can fill -in the missing parts of the mapping. -<P><HR><P> -This document was generated on 18 November 1999 using the -<A HREF="http://wwwcn.cern.ch/dci/texi2html/">texi2html</A> -translator version 1.51.</P> -<P> -Updated on 10/24/00 by hand, Quincey Koziol -</P> -</BODY> -</HTML> |