diff options
Diffstat (limited to 'doc/html/TechNotes')
53 files changed, 0 insertions, 10589 deletions
diff --git a/doc/html/TechNotes/Automake.html b/doc/html/TechNotes/Automake.html deleted file mode 100644 index c6dfc41..0000000 --- a/doc/html/TechNotes/Automake.html +++ /dev/null @@ -1,223 +0,0 @@ -<html> - -<head> -<meta http-equiv=Content-Type content="text/html; charset=utf-8"> -<title>An Automake Primer for HDF5</title> -</head> - -<h2>An Automake Primer for HDF5</h2> -<h4>James Laird - May 2005</h4><br> - -<p><h2><u>How to:</u><h2></p> - -<p><h3>Change a Makefile</h3></p> - -<p><h3>Add a source file to an existing program or library</h3></p> - -<p><h3>Add a simple test</h3></p> - -<p><h3>Add a test with multiple sources</h3></p> - -<p><h3>Add a new directory</h3></p> - -<p><h3>Add a program that is only compiled in parallel</h3></p> - -<p><h3>Change a program's name when it is compiled in parallel</h3></p> - -<p><h3>Add a new library</h3></p> - -<p><h3>Change the library's API</h3></p> -<br> - -<p><h4>Changing a Makefile</h4></p> - -<p>Suppose you need to make a minor change to the Makefile in the test directory -(<code>hdf5/test/Makefile</code>). You have checked out hdf5 from the CVS repository into -<code>~/scratch/hdf5</code>. You want to build the library in a directory named -<code>~/scratch/build</code>.<br> -First, edit the Makefile.am in the source tree. You must make any changes in the Makefile.am, -not the Makefile, since the Makefile is automatically generated.</p> - -<p><code>cd ~/scratch/hdf5/test<br> -vi Makefile.am</code></p> - -<p>Now, go to the root of the source tree and run the reconfigure script, which updates the -source tree. It will create a new Makefile.in in the test directory with your changes.</p> - -<p><code>cd ~/scratch/hdf5<br> -./bin/reconfigure</code></p> - -<p>After running <code>bin/reconfigure</code>, you will want to test your change. Go to -<code>~/scratch/build</code> and run <code>configure</code>.</p> - -<p><code>cd ~/scratch/build<br> - -../hdf5/configure<br> - -make check</code></p> - -<p>Configure generates Makefiles from the Makefiles.in in the source tree. The dependencies are:</p> - -<p><code>Makefile.am -> (bin/reconfigure) -> Makefile.in -> (configure) -> Makefile</code></p> - -<p>Reconfigure should also be used when any change is made to configure.in.</p> -<br> - -<p><h4>Adding a source file to an existing program or library</h4></p> - -<p>Suppose you want to add the source file <code>h5testfoo.c</code> to the HDF5 test -library in the test directory. You open up <code>test/Makefile.am</code> in your -favorite text editor and scroll down until you see the line:</p> - -<p><code>libh5test_la_SOURCES=h5test.c testframe.c</code></p> - -<p>Just add <code>h5testfoo.c</code> to the list of sources. You're done!<br> -Now run <code>bin/reconfigure</code> to create a new Makefile.in from the Makefile.am you just -edited.</p> -<br> - -<p><h4>Adding a simple test</h4></p> - -<p>Suppose you want to create a new test executable named <code>newtest</code> with one -source file, <code>newtest.c</code>. You open up <code>test/Makefile.am</code> and find -the line</p> - -<p><code>TEST_PROG=testhdf5 lheap ohdr ...</code></p> - -<p>Just add <code>newtest</code> to the list of programs. That's it! Automake will by -default guess that your program <code>newtest</code> has one source file named -<code>newtest.c</code>.<br> -Now run <code>bin/reconfigure</code> to update the Makefile.in.</p> -<br> - -<p><h4>Adding a slightly more complicated test</h4></p> - -<p>Suppose you want to create a new test executable named <code>newertest</code> with -several source files. You open up <code>test/Makefile.am</code> as before and find the line</p> - -<p><code>TEST_PROG=testhdf5 lheap ohdr ...</code></p> - -<p>Add <code>newertest</code> to the list of programs.<br> -Now you need to tell Automake how to build newertest. Add a new line below -<code>TEST_PROG</code>:</p> - -<p><code>newtest_SOURCES = source1.c source2.c source3.c</code></p> - -<p>You don't need to mention header files, as these will be automatically detected.<br> -Now run <code>bin/reconfigure</code> to update the Makefile.in.</p> -<br> - -<p><h4>Adding a directory</h4></p> - -<p>To add the directory for a new tool, <code>h5merge</code>, go to the Makefile.am -in the tools directory (the parent directory of the directory you want to add). -Find the line that reads</p> - -<p><code>SUBDIRS=lib h5dump...</code></p> - -<p>Add <code>h5merge</code> to this list of subdirectories.<br> -Now you probably want to create a Makefile.am in the h5merge directory. A good starting -point for this Makefile.am might be the sample Makefile.am in the config directory -(<code>config/Makefile.am.blank</code>). Alternately, you could copy the Makefile.am -from another directory.<br> -Once you have your new Makefile.am in place, edit <code>configure.in</code> in the root -directory. Near the end of the file is a list of files generated by configure. -Add <code>tools/h5merge/Makefile.in</code> to this list.<br> -Now run <code>bin/reconfigure</code>. This will update configure and generate a Makefile.in in the -<code>tools/h5merge</code> directory. Don't forget to add both the Makefile.am and the Makefile.in to -CVS, and to update the manifest!.</p> -<br> - -<p><h4>Adding a program that is only compiled in parallel</h4></p> - -<p>Suppose you only want to compile a program when HDF5 is configured to run in -parallel--for example, a parallel version of h5repack called <code>h5prepack</code>. -Open up the h5repack Makefile.am<br> -The simple solution is:</p> - -<p><code>if BUILD_PARALLEL_CONDITIONAL<br> - H5PREPACK=h5prepack<br> -endif</code></p> - -<p>Now the variable <code>$H5PREPACK</code> will be "h5prepack" if parallel is -enabled and "" if parallel is disabled. Add <code>$H5PREPACK</code> to the list of -programs to be built:</p> - -<p><code>bin_PROGRAMS=h5repack $(H5PREPACK)</code></p> - -<p>Add sources for this program as usual:</p> - -<p><code>h5prepack_SOURCES=...</code></p> - -<p>Don't forget to run <code>bin/reconfigure</code> when you're done!</p> -<br> - -<p><h4>Changing a program's name when it is compiled in parallel</h4></p> - -<p>Automake conditionals can be a very powerful tool. Suppose that instead of building -two versions of h5repack during a parallel build, you want to change the name of -the tool depending on whether or not HDF5 is configured to run in parallel--you -want to create either h5repack or h5prepack, but not both.<br> -Open up the h5repack Makefile.am and use an automake conditional:</p> - -<p><code>if BUILD_PARALLEL_CONDITIONAL<br> - H5REPACK_NAME=h5prepack<br> -else<br> - H5REPACK_NAME=h5repack<br> -endif<br> -bin_PROGRAMS=$(H5REPACK_NAME)</p> - -<p>Now you only build one program, but the name of that program changes. You still need -to define sources for both h5repack and h5prepack, but you needn't type them out twice if -they are the same:</p> - -<p><code>h5repack_SOURCES=...<br> -h5prepack_SOURCES=$(h5repack_SOURCES)</code></p> - -<p>Don't forget to run <code>bin/reconfigure</code> when you're done!</p> -<br> - -<p><h4>Adding a new library</h4></p> - -<p>Suppose you want to add a new library to the HDF5 build tree, libfoo. The procedure for -building libraries is very similar to that for building programs:</p> - -<p><code>lib_LTLIBRARIES=libfoo.la<br> -libfoo_la_SOURCES=sourcefoo.c sourcefootwo.c </code></p> - -<p>This library will be installed in the lib directory when a user types -"<code>make install</code>".<br> -You might instead be building a convenience library for testing purposes (like -<code>libh5test.la</code>) and not want it to be installed. If this is the case, you -would type</p> - -<p><code>check_LTLIBRARIES=libfoo.la</code><br> -instead of<br> -<code>lib_LTLIBRARIES=libfoo.la</code></p> - -<p>To make it easier for other directories to link to your library, -you might want to assign its path to a variable in all HDF5 Makefiles. You can -make changes to all Makefiles by editing <code>config/commence.am</code> and adding a line -like</p> - -<p><code>LIBFOO=$(top_builddir)/foo/src/libfoo.la</code></p> - -<p><code>config/commence.am</code> is textually included in all Makefiles.am when automake -processes them.<br> -As always, if you change a Makefile.am or <code>config/commence.am</code>, don't forget to run -<code>bin/reconfigure</code>.</p> -<br> - -<p><h4>Changing HDF5's API</h4></p> - -<p>If you have added or removed a function from HDF5, or if you have changed a function -signature, you must indicate this by updating the file <code>lt_vers.am</code> located in -the <code>config</code> directory.<br> -If you have changed the API at all, increment <code>LT_VERS_INTERFACE</code> and set -<code>LT_VERS_REVISION</code> to zero.<br> -If you have added functions but not altered or removed existing ones, also increment -<code>LT_VERS_AGE</code>.<br> -If instead you have altered or removed any functions, reset <code>LT_VERS_AGE</code> to -zero.</p> -</body> -</html> diff --git a/doc/html/TechNotes/Basic_perform.html b/doc/html/TechNotes/Basic_perform.html deleted file mode 100644 index 2a622fc..0000000 --- a/doc/html/TechNotes/Basic_perform.html +++ /dev/null @@ -1,75 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Description of the three basic performance tools</title> - </head> - - <body> - <h1>Description of the three basic performance tools</h1> - - <h2>iopipe</h2> - <p>Times reads and writes to an HDF5 2-d dataset and compares that with - reads and writes using POSIX I/O. Reports seven measurements in - terms of CPU user time, CPU system time, elapsed time, and - bandwidth: - - -<DL> -<DD>fill raw: time it takes to memset() a buffer.</DD> -<DD> fill hdf5: time it takes to read from a dataset never written</DD> -<DD>out raw: time it takes to write using POSIX I/O</DD> -<DD>out hdf5: time it takes to write using H5Dwrite()</DD> -<DD>in raw: time it takes to read data just written using POSIX I/O</DD> -<DD>in hdf5: time it takes to H5Dread() data written with H5Dwrite()</DD> -<DD>in hdf5 partial: time it takes to H5Dread() the "center" area.</DD> -</DL> - - - <p>This is a pretty stupid performance test. It accesses the same area - of file and memory over and over and the file size is way too - small. But it is good at showing how much overhead there is in the - library itself. - - - <h2>chunk</h2> - <p>Determines how efficient the raw data cache is for various access - patterns of a chunked dataset, both reading and writing. The access - pattern is either (a) we access the entire dataset by moving a window - across and down a 2-d dataset in row-major order a full window - height and width at a time, or (b) we access part of a dataset by moving - the window diagonally from the (0,0) corner to the opposite corner - by half the window height and width at a time. The window is - measured in terms of the chunk size. - - - <p>The result is: - <br>A table written to stdout that contains the window size as a - fraction of the chunk size and the efficiencey of the cache (i.e., - number of bytes accessed by H5Dread() or H5Dwrite() divided by the - number of bytes of the dataset actually read or written by lower - layers. - - - <p>A gnuplot script and data files which can be displayed by running - gnuplot and typing the command `load "x-gnuplot"'. - - - <h2>overhead</h2> - <p>Measures the overhead used by the B-tree for indexing chunked - datasets. As data is written to a chunked dataset the B-tree - grows and its nodes get split. When a node splits one of three - ratios are used to determine how many items from the original node - go into the new left and right nodes, and these ratios affect the - total size of the B-tree in a way that depends on the order that - data is written to the dataset. - - - <p>Invoke as `overhead usage' for more information. - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Tue Mar 17 11:13:35 EST 1998 --> -<!-- hhmts start --> -Last modified: Jun 4, 2003 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/BigDataSmMach.html b/doc/html/TechNotes/BigDataSmMach.html deleted file mode 100644 index fe00ff8..0000000 --- a/doc/html/TechNotes/BigDataSmMach.html +++ /dev/null @@ -1,122 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Big Datasets on Small Machines</title> - </head> - - <body> - <h1>Big Datasets on Small Machines</h1> - - <h2>1. Introduction</h2> - - <p>The HDF5 library is able to handle files larger than the - maximum file size, and datasets larger than the maximum memory - size. For instance, a machine where <code>sizeof(off_t)</code> - and <code>sizeof(size_t)</code> are both four bytes can handle - datasets and files as large as 18x10^18 bytes. However, most - Unix systems limit the number of concurrently open files, so a - practical file size limit is closer to 512GB or 1TB. - - <p>Two "tricks" must be imployed on these small systems in order - to store large datasets. The first trick circumvents the - <code>off_t</code> file size limit and the second circumvents - the <code>size_t</code> main memory limit. - - <h2>2. File Size Limits</h2> - - <p>Systems that have 64-bit file addresses will be able to access - those files automatically. One should see the following output - from configure: - - <p><code><pre> -checking size of off_t... 8 - </pre></code> - - <p>Also, some 32-bit operating systems have special file systems - that can support large (>2GB) files and HDF5 will detect - these and use them automatically. If this is the case, the - output from configure will show: - - <p><code><pre> -checking for lseek64... yes -checking for fseek64... yes - </pre></code> - - <p>Otherwise one must use an HDF5 file family. Such a family is - created by setting file family properties in a file access - property list and then supplying a file name that includes a - <code>printf</code>-style integer format. For instance: - - <p><code><pre> -hid_t plist, file; -plist = H5Pcreate (H5P_FILE_ACCESS); -H5Pset_family (plist, 1<<30, H5P_DEFAULT); -file = H5Fcreate ("big%03d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, plist); - </code></pre> - - <p>The second argument (<code>1<<30</code>) to - <code>H5Pset_family()</code> indicates that the family members - are to be 2^30 bytes (1GB) each although we could have used any - reasonably large value. In general, family members cannot be - 2GB because writes to byte number 2,147,483,647 will fail, so - the largest safe value for a family member is 2,147,483,647. - HDF5 will create family members on demand as the HDF5 address - space increases, but since most Unix systems limit the number of - concurrently open files the effective maximum size of the HDF5 - address space will be limited (the system on which this was - developed allows 1024 open files, so if each family member is - approx 2GB then the largest HDF5 file is approx 2TB). - - <p>If the effective HDF5 address space is limited then one may be - able to store datasets as external datasets each spanning - multiple files of any length since HDF5 opens external dataset - files one at a time. To arrange storage for a 5TB dataset split - among 1GB files one could say: - - <p><code><pre> -hid_t plist = H5Pcreate (H5P_DATASET_CREATE); -for (i=0; i<5*1024; i++) { - sprintf (name, "velocity-%04d.raw", i); - H5Pset_external (plist, name, 0, (size_t)1<<30); -} - </code></pre> - - <h2>3. Dataset Size Limits</h2> - - <p>The second limit which must be overcome is that of - <code>sizeof(size_t)</code>. HDF5 defines a data type called - <code>hsize_t</code> which is used for sizes of datasets and is, - by default, defined as <code>unsigned long long</code>. - - <p>To create a dataset with 8*2^30 4-byte integers for a total of - 32GB one first creates the dataspace. We give two examples - here: a 4-dimensional dataset whose dimension sizes are smaller - than the maximum value of a <code>size_t</code>, and a - 1-dimensional dataset whose dimension size is too large to fit - in a <code>size_t</code>. - - <p><code><pre> -hsize_t size1[4] = {8, 1024, 1024, 1024}; -hid_t space1 = H5Screate_simple (4, size1, size1); - -hsize_t size2[1] = {8589934592LL}; -hid_t space2 = H5Screate_simple (1, size2, size2}; - </pre></code> - - <p>However, the <code>LL</code> suffix is not portable, so it may - be better to replace the number with - <code>(hsize_t)8*1024*1024*1024</code>. - - <p>For compilers that don't support <code>long long</code> large - datasets will not be possible. The library performs too much - arithmetic on <code>hsize_t</code> types to make the use of a - struct feasible. - - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Fri Apr 10 13:26:04 EDT 1998 --> -<!-- hhmts start --> -Last modified: Sun Jul 19 11:37:25 EDT 1998 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/ChStudy_1000x1000.gif b/doc/html/TechNotes/ChStudy_1000x1000.gif Binary files differdeleted file mode 100644 index b7d5a83..0000000 --- a/doc/html/TechNotes/ChStudy_1000x1000.gif +++ /dev/null diff --git a/doc/html/TechNotes/ChStudy_250x250.gif b/doc/html/TechNotes/ChStudy_250x250.gif Binary files differdeleted file mode 100644 index fe35f39..0000000 --- a/doc/html/TechNotes/ChStudy_250x250.gif +++ /dev/null diff --git a/doc/html/TechNotes/ChStudy_499x499.gif b/doc/html/TechNotes/ChStudy_499x499.gif Binary files differdeleted file mode 100644 index 0d2038b..0000000 --- a/doc/html/TechNotes/ChStudy_499x499.gif +++ /dev/null diff --git a/doc/html/TechNotes/ChStudy_5000x1000.gif b/doc/html/TechNotes/ChStudy_5000x1000.gif Binary files differdeleted file mode 100644 index 0f3c290..0000000 --- a/doc/html/TechNotes/ChStudy_5000x1000.gif +++ /dev/null diff --git a/doc/html/TechNotes/ChStudy_500x500.gif b/doc/html/TechNotes/ChStudy_500x500.gif Binary files differdeleted file mode 100644 index 38dd7d6..0000000 --- a/doc/html/TechNotes/ChStudy_500x500.gif +++ /dev/null diff --git a/doc/html/TechNotes/ChStudy_p1.gif b/doc/html/TechNotes/ChStudy_p1.gif Binary files differdeleted file mode 100644 index 938d133..0000000 --- a/doc/html/TechNotes/ChStudy_p1.gif +++ /dev/null diff --git a/doc/html/TechNotes/ChStudy_p1.obj b/doc/html/TechNotes/ChStudy_p1.obj deleted file mode 100644 index 6fbf583..0000000 --- a/doc/html/TechNotes/ChStudy_p1.obj +++ /dev/null @@ -1,113 +0,0 @@ -%TGIF 3.0-p5 -state(0,33,100,0,0,0,16,1,9,1,1,0,0,3,7,1,1,'Helvetica',0,24,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -box('black',64,64,384,384,0,1,1,22,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 128,64,128,384],0,1,1,23,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 192,64,192,384],0,1,1,24,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 256,64,256,384],0,1,1,25,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 320,64,320,384],0,1,1,26,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 64,128,384,128],0,1,1,27,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 64,192,384,192],0,1,1,28,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 64,256,384,256],0,1,1,29,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 64,320,384,320],0,1,1,30,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',96,80,'Courier',0,17,1,1,0,1,7,14,37,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1"]). -text('black',160,80,'Courier',0,17,1,1,0,1,7,14,39,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "2"]). -text('black',224,80,'Courier',0,17,1,1,0,1,7,14,41,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "3"]). -text('black',288,80,'Courier',0,17,1,1,0,1,7,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "4"]). -text('black',352,80,'Courier',0,17,1,1,0,1,7,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "5"]). -text('black',96,144,'Courier',0,17,1,1,0,1,7,14,51,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "6"]). -text('black',160,144,'Courier',0,17,1,1,0,1,7,14,53,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "7"]). -text('black',224,144,'Courier',0,17,1,1,0,1,7,14,55,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "8"]). -text('black',288,144,'Courier',0,17,1,1,0,1,7,14,57,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "9"]). -text('black',352,144,'Courier',0,17,1,1,0,1,14,14,59,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "10"]). -text('black',96,208,'Courier',0,17,1,1,0,1,14,14,61,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "11"]). -text('black',160,208,'Courier',0,17,1,1,0,1,14,14,63,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "12"]). -text('black',224,208,'Courier',0,17,1,1,0,1,14,14,65,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "13"]). -text('black',288,208,'Courier',0,17,1,1,0,1,14,14,67,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "14"]). -text('black',352,208,'Courier',0,17,1,1,0,1,14,14,71,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "15"]). -text('black',96,272,'Courier',0,17,1,1,0,1,14,14,75,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "16"]). -text('black',160,272,'Courier',0,17,1,1,0,1,14,14,77,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "17"]). -text('black',224,272,'Courier',0,17,1,1,0,1,14,14,79,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "18"]). -text('black',288,272,'Courier',0,17,1,1,0,1,14,14,81,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "19"]). -text('black',352,272,'Courier',0,17,1,1,0,1,14,14,83,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "20"]). -text('black',96,336,'Courier',0,17,1,1,0,1,14,14,87,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "21"]). -text('black',160,336,'Courier',0,17,1,1,0,1,14,14,89,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "22"]). -text('black',224,336,'Courier',0,17,1,1,0,1,14,14,91,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "23"]). -text('black',288,336,'Courier',0,17,1,1,0,1,14,14,93,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "24"]). -text('black',352,336,'Courier',0,17,1,1,0,1,14,14,95,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "25"]). -poly('black',2,[ - 416,64,416,384],3,1,1,100,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 64,416,384,416],3,1,1,101,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',390,228,'Courier',0,17,1,0,0,1,14,35,102,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,1,0,[ - 390,228,390,228,425,242,0,-1000,1000,0,34,18,389,227,426,243],[ - "5,000"]). -text('black',224,432,'Courier',0,17,1,1,0,1,35,14,116,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "5,000"]). -text('black',160,512,'Courier',0,17,1,0,0,1,105,14,131,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "= 1,000 x 1,000"]). -box('black',80,480,144,544,7,1,1,134,0,0,0,0,0,'1',[ -]). -text('black',224,16,'Helvetica',0,24,1,1,0,1,296,29,144,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Order that data was written"]). -box('black',32,0,464,576,0,1,1,149,0,0,0,0,0,'1',[ -]). diff --git a/doc/html/TechNotes/ChunkingStudy.html b/doc/html/TechNotes/ChunkingStudy.html deleted file mode 100644 index 776b8fe..0000000 --- a/doc/html/TechNotes/ChunkingStudy.html +++ /dev/null @@ -1,190 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Testing the chunked layout of HDF5</title> - </head> - - <body> - -<i> -This document is of interest primarily for its discussion of the -HDF team's motivation for implementing raw data caching. -At a more abstract level, the discussion of the principles of -data chunking is also of interest, but a more recent discussion -of that topic can be found in -<a href="../Chunking.html">Dataset Chunking Issues</a> in the -<cite><a href="../H5.user.html">HDF5 User's Guide</a></cite>. - -The performance study described here predates the current chunking -implementation in the HDF5 library, so the particular performance data -is no longer apropos. - -- the Editor -</i> - - <h1>Testing the chunked layout of HDF5</h1> - - <p>This is the results of studying the chunked layout policy in - HDF5. A 1000 by 1000 array of integers was written to a file - dataset extending the dataset with each write to create, in the - end, a 5000 by 5000 array of 4-byte integers for a total data - storage size of 100 million bytes. - - <p> - <center> - <img alt="Order that data was written" src="ChStudy_p1.gif"> - <br><b>Fig 1: Write-order of Output Blocks</b> - </center> - - <p>After the array was written, it was read back in blocks that - were 500 by 500 bytes in row major order (that is, the top-left - quadrant of output block one, then the top-right quadrant of - output block one, then the top-left quadrant of output block 2, - etc.). - - <p>I tried to answer two questions: - <ul> - <li>How does the storage overhead change as the chunk size - changes? - <li>What does the disk seek pattern look like as the chunk size - changes? - </ul> - - <p>I started with chunk sizes that were multiples of the read - block size or k*(500, 500). - - <p> - <center> - <table border> - <caption align=bottom> - <b>Table 1: Total File Overhead</b> - </caption> - <tr> - <th>Chunk Size (elements)</th> - <th>Meta Data Overhead (ppm)</th> - <th>Raw Data Overhead (ppm)</th> - </tr> - - <tr align=center> - <td>500 by 500</td> - <td>85.84</td> - <td>0.00</td> - </tr> - <tr align=center> - <td>1000 by 1000</td> - <td>23.08</td> - <td>0.00</td> - </tr> - <tr align=center> - <td>5000 by 1000</td> - <td>23.08</td> - <td>0.00</td> - </tr> - <tr align=center> - <td>250 by 250</td> - <td>253.30</td> - <td>0.00</td> - </tr> - <tr align=center> - <td>499 by 499</td> - <td>85.84</td> - <td>205164.84</td> - </tr> - </table> - </center> - - <hr> - <p> - <center> - <img alt="500x500" src="ChStudy_500x500.gif"> - <br><b>Fig 2: Chunk size is 500x500</b> - </center> - - <p>The first half of Figure 2 shows output to the file while the - second half shows input. Each dot represents a file-level I/O - request and the lines that connect the dots are for visual - clarity. The size of the request is not indicated in the - graph. The output block size is four times the chunk size which - results in four file-level write requests per block for a total - of 100 requests. Since file space for the chunks was allocated - in output order, and the input block size is 1/4 the output - block size, the input shows a staircase effect. Each input - request results in one file-level read request. The downward - spike at about the 60-millionth byte is probably the result of a - cache miss for the B-tree and the downward spike at the end is - probably a cache flush or file boot block update. - - <hr> - <p> - <center> - <img alt="1000x1000" src="ChStudy_1000x1000.gif"> - <br><b>Fig 2: Chunk size is 1000x1000</b> - </center> - - <p>In this test I increased the chunk size to match the output - chunk size and one can see from the first half of the graph that - 25 file-level write requests were issued, one for each output - block. The read half of the test shows that four times the - amount of data was read as written. This results from the fact - that HDF5 must read the entire chunk for any request that falls - within that chunk, which is done because (1) if the data is - compressed the entire chunk must be decompressed, and (2) the - library assumes that a chunk size was chosen to optimize disk - performance. - - <hr> - <p> - <center> - <img alt="5000x1000" src="ChStudy_5000x1000.gif"> - <br><b>Fig 3: Chunk size is 5000x1000</b> - </center> - - <p>Increasing the chunk size further results in even worse - performance since both the read and write halves of the test are - re-reading and re-writing vast amounts of data. This proves - that one should be careful that chunk sizes are not much larger - than the typical partial I/O request. - - <hr> - <p> - <center> - <img alt="250x250" src="ChStudy_250x250.gif"> - <br><b>Fig 4: Chunk size is 250x250</b> - </center> - - <p>If the chunk size is decreased then the amount of data - transfered between the disk and library is optimal for no - caching, but the amount of meta data required to describe the - chunk locations increases to 250 parts per million. One can - also see that the final downward spike contains more file-level - write requests as the meta data is flushed to disk just before - the file is closed. - - <hr> - <p> - <center> - <img alt="499x499" src="ChStudy_499x499.gif"> - <br><b>Fig 4: Chunk size is 499x499</b> - </center> - - <p>This test shows the result of choosing a chunk size which is - close to the I/O block size. Because the total size of the - array isn't a multiple of the chunk size, the library allocates - an extra zone of chunks around the top and right edges of the - array which are only partially filled. This results in - 20,516,484 extra bytes of storage, a 20% increase in the total - raw data storage size. But the amount of meta data overhead is - the same as for the 500 by 500 test. In addition, the mismatch - causes entire chunks to be read in order to update a few - elements along the edge or the chunk which results in a 3.6-fold - increase in the amount of data transfered. - - <hr> - <address><a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</a></address> -<!-- Created: Fri Jan 30 21:04:49 EST 1998 --> -<!-- hhmts start --> -Last modified: 30 Jan 1998 (technical content) -<br> -Last modified: 9 May 2000 (editor's note) -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/CodeReview.html b/doc/html/TechNotes/CodeReview.html deleted file mode 100644 index 213cbbe..0000000 --- a/doc/html/TechNotes/CodeReview.html +++ /dev/null @@ -1,300 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Code Review</title> - </head> - <body> - <center><h1>Code Review 1</h1></center> - - <h3>Some background...</h3> - <p>This is one of the functions exported from the - <code>H5B.c</code> file that implements a B-link-tree class - without worrying about concurrency yet (thus the `Note:' in the - function prologue). The <code>H5B.c</code> file provides the - basic machinery for operating on generic B-trees, but it isn't - much use by itself. Various subclasses of the B-tree (like - symbol tables or indirect storage) provide their own interface - and back end to this function. For instance, - <code>H5G_stab_find()</code> takes a symbol table OID and a name - and calls <code>H5B_find()</code> with an appropriate - <code>udata</code> argument that eventually gets passed to the - <code>H5G_stab_find()</code> function. - - <p><code><pre> - 1 /*------------------------------------------------------------------------- - 2 * Function: H5B_find - 3 * - 4 * Purpose: Locate the specified information in a B-tree and return - 5 * that information by filling in fields of the caller-supplied - 6 * UDATA pointer depending on the type of leaf node - 7 * requested. The UDATA can point to additional data passed - 8 * to the key comparison function. - 9 * -10 * Note: This function does not follow the left/right sibling -11 * pointers since it assumes that all nodes can be reached -12 * from the parent node. -13 * -14 * Return: Success: SUCCEED if found, values returned through the -15 * UDATA argument. -16 * -17 * Failure: FAIL if not found, UDATA is undefined. -18 * -19 * Programmer: Robb Matzke -20 * matzke@llnl.gov -21 * Jun 23 1997 -22 * -23 * Modifications: -24 * -25 *------------------------------------------------------------------------- -26 */ -27 herr_t -28 H5B_find (H5F_t *f, const H5B_class_t *type, const haddr_t *addr, void *udata) -29 { -30 H5B_t *bt=NULL; -31 intn idx=-1, lt=0, rt, cmp=1; -32 int ret_value = FAIL; - </pre></code> - - <p>All pointer arguments are initialized when defined. I don't - worry much about non-pointers because it's usually obvious when - the value isn't initialized. - - <p><code><pre> -33 -34 FUNC_ENTER (H5B_find, NULL, FAIL); -35 -36 /* -37 * Check arguments. -38 */ -39 assert (f); -40 assert (type); -41 assert (type->decode); -42 assert (type->cmp3); -43 assert (type->found); -44 assert (addr && H5F_addr_defined (addr)); - </pre></code> - - <p>I use <code>assert</code> to check invariant conditions. At - this level of the library, none of these assertions should fail - unless something is majorly wrong. The arguments should have - already been checked by higher layers. It also provides - documentation about what arguments might be optional. - - <p><code><pre> -45 -46 /* -47 * Perform a binary search to locate the child which contains -48 * the thing for which we're searching. -49 */ -50 if (NULL==(bt=H5AC_protect (f, H5AC_BT, addr, type, udata))) { -51 HGOTO_ERROR (H5E_BTREE, H5E_CANTLOAD, FAIL); -52 } - </pre></code> - - <p>You'll see this quite often in the low-level stuff and it's - documented in the <code>H5AC.c</code> file. The - <code>H5AC_protect</code> insures that the B-tree node (which - inherits from the H5AC package) whose OID is <code>addr</code> - is locked into memory for the duration of this function (see the - <code>H5AC_unprotect</code> on line 90). Most likely, if this - node has been accessed in the not-to-distant past, it will still - be in memory and the <code>H5AC_protect</code> is almost a - no-op. If cache debugging is compiled in, then the protect also - prevents other parts of the library from accessing the node - while this function is protecting it, so this function can allow - the node to be in an inconsistent state while calling other - parts of the library. - - <p>The alternative is to call the slighlty cheaper - <code>H5AC_find</code> and assume that the pointer it returns is - valid only until some other library function is called, but - since we're accessing the pointer throughout this function, I - chose to use the simpler protect scheme. All protected objects - <em>must be unprotected</em> before the file is closed, thus the - use of <code>HGOTO_ERROR</code> instead of - <code>HRETURN_ERROR</code>. - - <p><code><pre> -53 rt = bt->nchildren; -54 -55 while (lt<rt && cmp) { -56 idx = (lt + rt) / 2; -57 if (H5B_decode_keys (f, bt, idx)<0) { -58 HGOTO_ERROR (H5E_BTREE, H5E_CANTDECODE, FAIL); -59 } -60 -61 /* compare */ -62 if ((cmp=(type->cmp3)(f, bt->key[idx].nkey, udata, -63 bt->key[idx+1].nkey))<0) { -64 rt = idx; -65 } else { -66 lt = idx+1; -67 } -68 } -69 if (cmp) { -70 HGOTO_ERROR (H5E_BTREE, H5E_NOTFOUND, FAIL); -71 } - </pre></code> - - <p>Code is arranged in paragraphs with a comment starting each - paragraph. The previous paragraph is a standard binary search - algorithm. The <code>(type->cmp3)()</code> is an indirect - function call into the subclass of the B-tree. All indirect - function calls have the function part in parentheses to document - that it's indirect (quite obvious here, but not so obvious when - the function is a variable). - - <p>It's also my standard practice to have side effects in - conditional expressions because I can write code faster and it's - more apparent to me what the condition is testing. But if I - have an assignment in a conditional expr, then I use an extra - set of parens even if they're not required (usually they are, as - in this case) so it's clear that I meant <code>=</code> instead - of <code>==</code>. - - <p><code><pre> -72 -73 /* -74 * Follow the link to the subtree or to the data node. -75 */ -76 assert (idx>=0 && idx<bt->nchildren); -77 if (bt->level > 0) { -78 if ((ret_value = H5B_find (f, type, bt->child+idx, udata))<0) { -79 HGOTO_ERROR (H5E_BTREE, H5E_NOTFOUND, FAIL); -80 } -81 } else { -82 ret_value = (type->found)(f, bt->child+idx, bt->key[idx].nkey, -83 udata, bt->key[idx+1].nkey); -84 if (ret_value<0) { -85 HGOTO_ERROR (H5E_BTREE, H5E_NOTFOUND, FAIL); -86 } -87 } - </pre></code> - - <p>Here I broke the "side effect in conditional" rule, which I - sometimes do if the expression is so long that the - <code><0</code> gets lost at the end. Another thing to note is - that success/failure is always determined by comparing with zero - instead of <code>SUCCEED</code> or <code>FAIL</code>. I do this - because occassionally one might want to return other meaningful - values (always non-negative) or distinguish between various types of - failure (always negative). - - <p><code><pre> -88 -89 done: -90 if (bt && H5AC_unprotect (f, H5AC_BT, addr, bt)<0) { -91 HRETURN_ERROR (H5E_BTREE, H5E_PROTECT, FAIL); -92 } -93 FUNC_LEAVE (ret_value); -94 } - </pre></code> - - <p>For lack of a better way to handle errors during error cleanup, - I just call the <code>HRETURN_ERROR</code> macro even though it - will make the error stack not quite right. I also use short - circuiting boolean operators instead of nested <code>if</code> - statements since that's standard C practice. - - <center><h1>Code Review 2</h1></center> - - - <p>The following code is an API function from the H5F package... - - <p><code><pre> - 1 /*-------------------------------------------------------------------------- - 2 NAME - 3 H5Fflush - 4 - 5 PURPOSE - 6 Flush all cached data to disk and optionally invalidates all cached - 7 data. - 8 - 9 USAGE -10 herr_t H5Fflush(fid, invalidate) -11 hid_t fid; IN: File ID of file to close. -12 hbool_t invalidate; IN: Invalidate all of the cache? -13 -14 ERRORS -15 ARGS BADTYPE Not a file atom. -16 ATOM BADATOM Can't get file struct. -17 CACHE CANTFLUSH Flush failed. -18 -19 RETURNS -20 SUCCEED/FAIL -21 -22 DESCRIPTION -23 This function flushes all cached data to disk and, if INVALIDATE -24 is non-zero, removes cached objects from the cache so they must be -25 re-read from the file on the next access to the object. -26 -27 MODIFICATIONS: -28 --------------------------------------------------------------------------*/ - </pre></code> - - <p>An API prologue is used for each API function instead of my - normal function prologue. I use the prologue from Code Review 1 - for non-API functions because it's more suited to C programmers, - it requires less work to keep it synchronized with the code, and - I have better editing tools for it. - - <p><code><pre> -29 herr_t -30 H5Fflush (hid_t fid, hbool_t invalidate) -31 { -32 H5F_t *file = NULL; -33 -34 FUNC_ENTER (H5Fflush, H5F_init_interface, FAIL); -35 H5ECLEAR; - </pre></code> - - <p>API functions are never called internally, therefore I always - clear the error stack before doing anything. - - <p><code><pre> -36 -37 /* check arguments */ -38 if (H5_FILE!=H5Aatom_group (fid)) { -39 HRETURN_ERROR (H5E_ARGS, H5E_BADTYPE, FAIL); /*not a file atom*/ -40 } -41 if (NULL==(file=H5Aatom_object (fid))) { -42 HRETURN_ERROR (H5E_ATOM, H5E_BADATOM, FAIL); /*can't get file struct*/ -43 } - </pre></code> - - <p>If something is wrong with the arguments then we raise an - error. We never <code>assert</code> arguments at this level. - We also convert atoms to pointers since atoms are really just a - pointer-hiding mechanism. Functions that can be called - internally always have pointer arguments instead of atoms - because (1) then they don't have to always convert atoms to - pointers, and (2) the various pointer data types provide more - documentation and type checking than just an <code>hid_t</code> - type. - - <p><code><pre> -44 -45 /* do work */ -46 if (H5F_flush (file, invalidate)<0) { -47 HRETURN_ERROR (H5E_CACHE, H5E_CANTFLUSH, FAIL); /*flush failed*/ -48 } - </pre></code> - - <p>An internal version of the function does the real work. That - internal version calls <code>assert</code> to check/document - it's arguments and can be called from other library functions. - - <p><code><pre> -49 -50 FUNC_LEAVE (SUCCEED); -51 } - </pre></code> - - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Sat Nov 8 17:09:33 EST 1997 --> -<!-- hhmts start --> -Last modified: Mon Nov 10 15:33:33 EST 1997 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/Daily_Test_Explained.htm b/doc/html/TechNotes/Daily_Test_Explained.htm deleted file mode 100644 index ffa4798..0000000 --- a/doc/html/TechNotes/Daily_Test_Explained.htm +++ /dev/null @@ -1,863 +0,0 @@ -<html xmlns:o="urn:schemas-microsoft-com:office:office" -xmlns:w="urn:schemas-microsoft-com:office:word" -xmlns="http://www.w3.org/TR/REC-html40"> - -<head> -<meta http-equiv=Content-Type content="text/html; charset=windows-1252"> -<meta name=ProgId content=Word.Document> -<meta name=Generator content="Microsoft Word 9"> -<meta name=Originator content="Microsoft Word 9"> -<link rel=File-List href="./Daily_Test_Explained_files/filelist.xml"> -<title>Daily Test Explained</title> -<!--[if gte mso 9]><xml> - <o:DocumentProperties> - <o:Author>Albert Cheng</o:Author> - <o:Template>Normal</o:Template> - <o:LastAuthor>Albert Cheng</o:LastAuthor> - <o:Revision>4</o:Revision> - <o:TotalTime>21</o:TotalTime> - <o:LastPrinted>2002-10-24T16:02:00Z</o:LastPrinted> - <o:Created>2002-10-28T17:28:00Z</o:Created> - <o:LastSaved>2002-10-28T17:49:00Z</o:LastSaved> - <o:Pages>2</o:Pages> - <o:Words>608</o:Words> - <o:Characters>3468</o:Characters> - <o:Company>NCSA</o:Company> - <o:Lines>28</o:Lines> - <o:Paragraphs>6</o:Paragraphs> - <o:CharactersWithSpaces>4258</o:CharactersWithSpaces> - <o:Version>9.3821</o:Version> - </o:DocumentProperties> -</xml><![endif]--> -<style> -<!-- - /* Font Definitions */ -@font-face - {font-family:Tahoma; - panose-1:2 11 6 4 3 5 4 4 2 4; - mso-font-charset:0; - mso-generic-font-family:swiss; - mso-font-pitch:variable; - mso-font-signature:553679495 -2147483648 8 0 66047 0;} - /* Style Definitions */ -p.MsoNormal, li.MsoNormal, div.MsoNormal - {mso-style-parent:""; - margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - font-size:12.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman";} -h1 - {mso-style-next:Normal; - margin-top:12.0pt; - margin-right:0in; - margin-bottom:3.0pt; - margin-left:0in; - mso-pagination:widow-orphan; - page-break-after:avoid; - mso-outline-level:1; - font-size:16.0pt; - font-family:Arial; - mso-font-kerning:16.0pt;} -h2 - {mso-style-next:Normal; - margin-top:12.0pt; - margin-right:0in; - margin-bottom:3.0pt; - margin-left:0in; - mso-pagination:widow-orphan; - page-break-after:avoid; - mso-outline-level:2; - font-size:14.0pt; - font-family:Arial; - font-style:italic;} -p.MsoHeader, li.MsoHeader, div.MsoHeader - {margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - tab-stops:center 3.0in right 6.0in; - font-size:12.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman";} -p.MsoFooter, li.MsoFooter, div.MsoFooter - {margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - tab-stops:center 3.0in right 6.0in; - font-size:12.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman";} -p.MsoDocumentMap, li.MsoDocumentMap, div.MsoDocumentMap - {margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - background:navy; - font-size:12.0pt; - font-family:Tahoma; - mso-fareast-font-family:"Times New Roman";} -@page Section1 - {size:8.5in 11.0in; - margin:1.0in 1.2in .9in 1.2in; - mso-header-margin:.5in; - mso-footer-margin:.3in; - mso-paper-source:0;} -div.Section1 - {page:Section1;} - /* List Definitions */ -@list l0 - {mso-list-id:28529256; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l0:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.75in; - mso-level-number-position:left; - margin-left:.75in; - text-indent:-.25in; - font-family:Symbol;} -@list l0:level2 - {mso-level-number-format:alpha-lower; - mso-level-tab-stop:1.25in; - mso-level-number-position:left; - margin-left:1.25in; - text-indent:-.25in;} -@list l0:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l0:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l0:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l0:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l0:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l0:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l0:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1 - {mso-list-id:535626758; - mso-list-type:hybrid; - mso-list-template-ids:2106240834 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l1:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in; - font-family:Symbol;} -@list l1:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2 - {mso-list-id:610862704; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l2:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.75in; - mso-level-number-position:left; - margin-left:.75in; - text-indent:-.25in; - font-family:Symbol;} -@list l2:level2 - {mso-level-number-format:alpha-lower; - mso-level-tab-stop:1.25in; - mso-level-number-position:left; - margin-left:1.25in; - text-indent:-.25in;} -@list l2:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3 - {mso-list-id:831875463; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l3:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.75in; - mso-level-number-position:left; - margin-left:.75in; - text-indent:-.25in; - font-family:Symbol;} -@list l3:level2 - {mso-level-number-format:alpha-lower; - mso-level-tab-stop:1.25in; - mso-level-number-position:left; - margin-left:1.25in; - text-indent:-.25in;} -@list l3:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4 - {mso-list-id:1024131184; - mso-list-type:hybrid; - mso-list-template-ids:-218050110 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} -@list l4:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in; - font-family:Symbol;} -@list l4:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5 - {mso-list-id:1066879027; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l5:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.75in; - mso-level-number-position:left; - margin-left:.75in; - text-indent:-.25in; - font-family:Symbol;} -@list l5:level2 - {mso-level-number-format:alpha-lower; - mso-level-tab-stop:1.25in; - mso-level-number-position:left; - margin-left:1.25in; - text-indent:-.25in;} -@list l5:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6 - {mso-list-id:1353801443; - mso-list-type:hybrid; - mso-list-template-ids:56765176 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} -@list l6:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in; - font-family:Symbol;} -@list l6:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7 - {mso-list-id:1585335932; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l7:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.75in; - mso-level-number-position:left; - margin-left:.75in; - text-indent:-.25in; - font-family:Symbol;} -@list l7:level2 - {mso-level-number-format:alpha-lower; - mso-level-tab-stop:1.25in; - mso-level-number-position:left; - margin-left:1.25in; - text-indent:-.25in;} -@list l7:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l7:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8 - {mso-list-id:1594970409; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698689 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l8:level1 - {mso-level-number-format:bullet; - mso-level-text:\F0B7; - mso-level-tab-stop:.75in; - mso-level-number-position:left; - margin-left:.75in; - text-indent:-.25in; - font-family:Symbol;} -@list l8:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l8:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9 - {mso-list-id:1945573783; - mso-list-type:hybrid; - mso-list-template-ids:-796201994 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l9:level1 - {mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level2 - {mso-level-number-format:alpha-lower; - mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l9:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -ol - {margin-bottom:0in;} -ul - {margin-bottom:0in;} ---> -</style> -</head> - -<body lang=EN-US style='tab-interval:.5in'> - -<div class=Section1> - -<h1 align=center style='text-align:center'>Daily Test Explained</h1> - -<h2>Requirements for a Daily Test Host</h2> - -<ul style='margin-top:0in' type=disc> - <li class=MsoNormal style='mso-list:l1 level1 lfo3;tab-stops:list .5in'>Kerberos - and AFS support</li> - <li class=MsoNormal style='mso-list:l1 level1 lfo3;tab-stops:list .5in'>Remote - command execution (rsh or ssh) with Kerberos authentication support</li> - <li class=MsoNormal style='mso-list:l1 level1 lfo3;tab-stops:list .5in'>make - that support srcdir compiling (highly desirable)</li> - <li class=MsoNormal style='mso-list:l1 level1 lfo3;tab-stops:list .5in'>diff - that supports –I option (highly desirable for launching host)</li> - <li class=MsoNormal style='mso-list:l1 level1 lfo3;tab-stops:list .5in'>cvs - command support (desirable)</li> -</ul> - -<h2>Directories/Files Used</h2> - -<p class=MsoNormal><b>$HOME/snapshots-XXX</b> is where daily tests occur.</p> - -<ul style='margin-top:0in' type=disc> - <li class=MsoNormal style='mso-list:l4 level1 lfo6;tab-stops:list .5in'><b>$HOME/snapshots-hdf5</b> - for hdf5 main trunk version (currently v1.5).</li> - <li class=MsoNormal style='mso-list:l4 level1 lfo6;tab-stops:list .5in'><b>$HOME/snapshots-hdf5_1_4 - </b>for hdf5 version 1.4.</li> - <li class=MsoNormal style='mso-list:l4 level1 lfo6;tab-stops:list .5in'><b>$HOME/snapshots-hdf4 - </b>for hdf4 main trunk version (currenly post 4.1r5).</li> -</ul> - -<h2>Inside snapshots-XXX Directory</h2> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>current/<span style='mso-tab-count:1'> </span>latest -version</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>previous/<span style='mso-tab-count:1'> </span>last -released version</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>log/<span style='mso-tab-count:1'> </span>log -files of most recent tests</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>log/OLD/<span style='mso-tab-count:1'> </span>previous -log files</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>TestDir/<host>/<span style='mso-tab-count:1'> </span>build -and test area of machine <host> supporting srcdir build</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>allhostfile<span style='mso-tab-count:1'> </span>holds -all test host names</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>snaptest.cfg<span style='mso-tab-count:1'> </span>holds -various test configurations</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>release_always<span style='mso-tab-count:1'> </span>always -make snapshot release tarball if all tests pass (implemented for hdf4 daily -tests only)</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>release_asap<span style='mso-tab-count:1'> </span>make -<i>one</i> snapshot release tarball if all tests pass (file is renamed after -release)</p> - -<p class=MsoNormal style='margin-left:135.0pt;text-indent:-117.0pt;mso-list: -l6 level1 lfo8;tab-stops:list .5in'><![if !supportLists]><span -style='font-family:Symbol'>·<span style='font:7.0pt "Times New Roman"'> -</span></span><![endif]>release_not<span style='mso-tab-count:1'> </span>do -not make snapshot release tarball even if all tests pass</p> - -<h2>Steps</h2> - -<p class=MsoNormal>This shows steps of the daily tests for HDF5 development -version (currenly v1.5).<span style="mso-spacerun: yes"> </span>The HDF5 v1.4 -and HDF4 are similar.<span style="mso-spacerun: yes"> </span>snapshots-XXX -here means $HOME/snapshots-hdf5/.</p> - -<p class=MsoNormal><![if !supportEmptyParas]> <![endif]><o:p></o:p></p> - -<ol style='margin-top:0in' start=1 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>“hdfadmin” - starts a cron job after midnight in eirene.</li> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>Cron - job acquires kerberos credential and AFS tokens.</li> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>Execute - <b>$HOME/.crondir/DailyMaint</b> to start daily maintenance</li> -</ol> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l8 level1 lfo13; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>CVS -updates some documents on websites</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l8 level1 lfo13; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>CVS -updates $HOME/HDF5/v_1_5/hdf5/<span style="mso-spacerun: yes"> </span>(the -bin/runtest in it is ready to be used in<span style="mso-spacerun: yes"> -</span>next step)</p> - -<ol style='margin-top:0in' start=4 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>Execute - <b>$HOME/.bin-sys/DailyHDF5Test</b></li> -</ol> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l7 level1 lfo16; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Clean -up snapshots-XXX/log area</p> - -<p class=MsoNormal style='margin-left:1.25in;text-indent:-.25in;mso-list:l7 level2 lfo16; -tab-stops:list 1.25in'><![if !supportLists]>a.<span style='font:7.0pt "Times New Roman"'> -</span><![endif]>Purge older files from OLD/</p> - -<p class=MsoNormal style='margin-left:1.25in;text-indent:-.25in;mso-list:l7 level2 lfo16; -tab-stops:list 1.25in'><![if !supportLists]>b.<span style='font:7.0pt "Times New Roman"'> -</span><![endif]>Moves log files from yesterday to OLD/</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l7 level1 lfo16; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>cd -$HOME/HDF5/v_1_5/hdf5</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l7 level1 lfo16; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Launch -“bin/runtest –all” from eirene</p> - -<ol style='margin-top:0in' start=5 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'><b>bin/runtest - –all<o:p></o:p></b></li> -</ol> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l5 level1 lfo19; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>CVS -updates $HOME/snapshots-XXX/current (the commands in bin/ are now ready be used -in the following steps).</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l5 level1 lfo19; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Executes -snapshots-XXX/current/bin/chkmanifest for MANIFEST file.</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l5 level1 lfo19; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Diff -current/ and previous/ versions.<span style="mso-spacerun: yes"> </span>If no -significant differences found, no need to run daily test per hosts.<span -style="mso-spacerun: yes"> </span>Will not make snapshot release tarball -either.</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l5 level1 lfo19; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>If -significant differences found, prepare to run the daily tests for all hosts.</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l5 level1 lfo19; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Reads -allhostfile for test hosts.<span style="mso-spacerun: yes"> </span>For each -host:</p> - -<p class=MsoNormal style='margin-left:1.25in;text-indent:-.25in;mso-list:l5 level2 lfo19; -tab-stops:list 1.25in'><![if !supportLists]>a.<span style='font:7.0pt "Times New Roman"'> -</span><![endif]>use ping then rsh/ssh to make sure the host is on line and -responding</p> - -<p class=MsoNormal style='margin-left:1.25in;text-indent:-.25in;mso-list:l5 level2 lfo19; -tab-stops:list 1.25in'><![if !supportLists]>b.<span style='font:7.0pt "Times New Roman"'> -</span><![endif]>if srcdir is support, fork off the following command for all -hosts and wait for them to finish.<span style="mso-spacerun: yes"> -</span>Otherwise, launch one at a time.</p> - -<p class=MsoNormal style='margin-left:1.25in;text-indent:-.25in;mso-list:l5 level2 lfo19; -tab-stops:list 1.25in'><![if !supportLists]>c.<span style='font:7.0pt "Times New Roman"'> -</span><![endif]>rsh host “cd $HOME/snapshots-XXX/hdf5; bin/runtest” >& -#<host></p> - -<ol style='margin-top:0in' start=6 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'><b>bin/runtest - </b>(one each in multiple hosts)</li> -</ol> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo22; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Since -“-all” is not used, it is for launching the test for this host only.</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo22; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Reads -snapshots-XXX/snaptest.cfg and looks for configuration entries that are for -this host.</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo22; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>For -each configuration, runs snapshots-XXX/bin/snapshot with the configuration.</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo22; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Configure, -build and test results are stored in log/<host>_YYMMDD_HHMM (e.g., -arabica_021024_0019)</p> - -<ol style='margin-top:0in' start=7 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>Back - to <b>“bin/runtest –all”</b> in eirene</li> -</ol> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l2 level1 lfo25; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Gather -all those #<host> files and other summary report into one daily report -(e.g., DailyHDF5Tests-eirene_021024)</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l3 level1 lfo27; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Checks -the tail of log/<host>_YYMMDD_HHMM to make sure it does complete -properly.</p> - -<ol style='margin-top:0in' start=8 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>Back - to <b>“.bin-sys/DailyHDF5Test”</b></li> -</ol> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l3 level1 lfo27; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Do -a snapshot release if<br> -<span style="mso-spacerun: yes"> </span>test-succeeded &&<br> -<span style="mso-spacerun: yes"> </span>release-not-is-not-present -&&<br> -<span style="mso-spacerun: yes"> </span>( today-is-saturday || -release-asap-is-requested )</p> - -<p class=MsoNormal style='margin-left:.75in;text-indent:-.25in;mso-list:l3 level1 lfo27; -tab-stops:list .75in'><![if !supportLists]><span style='font-family:Symbol'>·<span -style='font:7.0pt "Times New Roman"'> </span></span><![endif]>HDF4 -does not know how to create a release tarball.<span style="mso-spacerun: yes"> -</span>Its release process only renames current/ as previous/ to reduce future -test time.<span style="mso-spacerun: yes"> </span>It also supports an option -of release-always which tells daily test to make a release whenever all tests -pass.<span style="mso-spacerun: yes"> </span>The release-asap only make the -release once and the file is renamed, blocking any future ASAP release until -someone turns it on again.</p> - -<ol style='margin-top:0in' start=9 type=1> - <li class=MsoNormal style='mso-list:l9 level1 lfo11;tab-stops:list .5in'>Compose - a report and email “hdf5-cvs”</li> -</ol> - -<h2>Acknowledgement</h2> - -<p class=MsoNormal>Robb Matzke first setup the snapshot directory structure and -created pretty complete version of commands snaptest, release and h5ver. The -initial version is for testing in one host with the default configuration.<span -style="mso-spacerun: yes"> </span>I just added more whistles and bells.<span -style="mso-spacerun: yes"> </span>Jim Barlow helped me how to authenticate a -cron task with <i>keytab</i>.</p> - -<p class=MsoHeader style='tab-stops:.5in center 3.0in right 6.0in'>----</p> - -<p class=MsoNormal><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt'>First -created by Albert Cheng, October 24, 2002.<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-size:10.0pt;mso-bidi-font-size:12.0pt'>Revised -October 28, 2002.<o:p></o:p></span></p> - -</div> - -</body> - -</html> diff --git a/doc/html/TechNotes/DataTransformReport.htm b/doc/html/TechNotes/DataTransformReport.htm deleted file mode 100644 index 5a1a158..0000000 --- a/doc/html/TechNotes/DataTransformReport.htm +++ /dev/null @@ -1,877 +0,0 @@ -<html xmlns:v="urn:schemas-microsoft-com:vml" -xmlns:o="urn:schemas-microsoft-com:office:office" -xmlns:w="urn:schemas-microsoft-com:office:word" -xmlns:st1="urn:schemas-microsoft-com:office:smarttags" -xmlns="http://www.w3.org/TR/REC-html40"> - -<head> -<meta http-equiv=Content-Type content="text/html; charset=windows-1252"> -<meta name=ProgId content=Word.Document> -<meta name=Generator content="Microsoft Word 10"> -<meta name=Originator content="Microsoft Word 10"> -<link rel=File-List href="Data_Transforms_Report_files/filelist.xml"> -<title>Arithmetic Data Transforms</title> -<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" - name="date"/> -<!--[if gte mso 9]><xml> - <o:DocumentProperties> - <o:Author>install</o:Author> - <o:Template>Normal</o:Template> - <o:LastAuthor>Albert Cheng</o:LastAuthor> - <o:Revision>4</o:Revision> - <o:TotalTime>212</o:TotalTime> - <o:LastPrinted>2004-12-10T21:28:00Z</o:LastPrinted> - <o:Created>2004-12-10T22:44:00Z</o:Created> - <o:LastSaved>2004-12-16T00:13:00Z</o:LastSaved> - <o:Pages>1</o:Pages> - <o:Words>1721</o:Words> - <o:Characters>9815</o:Characters> - <o:Company>UIUC</o:Company> - <o:Lines>81</o:Lines> - <o:Paragraphs>23</o:Paragraphs> - <o:CharactersWithSpaces>11513</o:CharactersWithSpaces> - <o:Version>10.2625</o:Version> - </o:DocumentProperties> -</xml><![endif]--><!--[if gte mso 9]><xml> - <w:WordDocument> - <w:View>Print</w:View> - <w:Zoom>105</w:Zoom> - <w:PunctuationKerning/> - <w:Compatibility> - <w:BreakWrappedTables/> - <w:SnapToGridInCell/> - <w:ApplyBreakingRules/> - <w:WrapTextWithPunct/> - <w:UseAsianBreakRules/> - <w:UseFELayout/> - </w:Compatibility> - <w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel> - </w:WordDocument> -</xml><![endif]--><!--[if !mso]><object - classid="clsid:38481807-CA0E-42D2-BF39-B33AF135CC4D" id=ieooui></object> -<style> -st1\:*{behavior:url(#ieooui) } -</style> -<![endif]--> -<style> -<!-- - /* Font Definitions */ - @font-face - {font-family:Courier; - panose-1:2 7 4 9 2 2 5 2 4 4; - mso-font-alt:"Courier New"; - mso-font-charset:0; - mso-generic-font-family:modern; - mso-font-format:other; - mso-font-pitch:fixed; - mso-font-signature:3 0 0 0 1 0;} -@font-face - {font-family:SimSun; - panose-1:2 1 6 0 3 1 1 1 1 1; - mso-font-alt:\5B8B\4F53; - mso-font-charset:134; - mso-generic-font-family:auto; - mso-font-pitch:variable; - mso-font-signature:3 135135232 16 0 262145 0;} -@font-face - {font-family:"\@SimSun"; - panose-1:2 1 6 0 3 1 1 1 1 1; - mso-font-charset:134; - mso-generic-font-family:auto; - mso-font-pitch:variable; - mso-font-signature:3 135135232 16 0 262145 0;} - /* Style Definitions */ - p.MsoNormal, li.MsoNormal, div.MsoNormal - {mso-style-parent:""; - margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - font-size:12.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman"; - mso-fareast-language:EN-US;} -h1 - {mso-style-next:Normal; - margin-top:12.0pt; - margin-right:0in; - margin-bottom:3.0pt; - margin-left:0in; - mso-pagination:widow-orphan; - page-break-after:avoid; - mso-outline-level:1; - font-size:16.0pt; - font-family:Arial; - mso-font-kerning:16.0pt; - mso-fareast-language:EN-US;} -h2 - {mso-style-next:Normal; - margin-top:12.0pt; - margin-right:0in; - margin-bottom:3.0pt; - margin-left:0in; - mso-pagination:widow-orphan; - page-break-after:avoid; - mso-outline-level:2; - font-size:14.0pt; - font-family:Arial; - mso-fareast-language:EN-US; - font-style:italic;} -p.MsoFootnoteText, li.MsoFootnoteText, div.MsoFootnoteText - {mso-style-noshow:yes; - margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - font-size:10.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman"; - mso-fareast-language:EN-US;} -p.MsoHeader, li.MsoHeader, div.MsoHeader - {margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - tab-stops:center 3.0in right 6.0in; - font-size:12.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman"; - mso-fareast-language:EN-US;} -p.MsoFooter, li.MsoFooter, div.MsoFooter - {margin:0in; - margin-bottom:.0001pt; - mso-pagination:widow-orphan; - tab-stops:center 3.0in right 6.0in; - font-size:12.0pt; - font-family:"Times New Roman"; - mso-fareast-font-family:"Times New Roman"; - mso-fareast-language:EN-US;} -span.MsoFootnoteReference - {mso-style-noshow:yes; - vertical-align:super;} - /* Page Definitions */ - @page - {mso-footnote-separator:url("Data_Transforms_Report_files/header.htm") fs; - mso-footnote-continuation-separator:url("Data_Transforms_Report_files/header.htm") fcs; - mso-endnote-separator:url("Data_Transforms_Report_files/header.htm") es; - mso-endnote-continuation-separator:url("Data_Transforms_Report_files/header.htm") ecs;} -@page Section1 - {size:8.5in 11.0in; - margin:1.0in 1.25in 1.0in 1.25in; - mso-header-margin:.5in; - mso-footer-margin:.5in; - mso-even-footer:url("Data_Transforms_Report_files/header.htm") ef1; - mso-footer:url("Data_Transforms_Report_files/header.htm") f1; - mso-paper-source:0;} -div.Section1 - {page:Section1;} - /* List Definitions */ - @list l0 - {mso-list-id:270281622; - mso-list-template-ids:-1038041382;} -@list l1 - {mso-list-id:332993863; - mso-list-type:hybrid; - mso-list-template-ids:1458465098 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l1:level1 - {mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l1:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2 - {mso-list-id:550459489; - mso-list-type:hybrid; - mso-list-template-ids:1085822362 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l2:level1 - {mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l2:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l3 - {mso-list-id:756709709; - mso-list-template-ids:-1385235826;} -@list l4 - {mso-list-id:1187134103; - mso-list-type:hybrid; - mso-list-template-ids:1324645902 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l4:level1 - {mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l4:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5 - {mso-list-id:1433163999; - mso-list-type:hybrid; - mso-list-template-ids:-438374070 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} -@list l5:level1 - {mso-level-tab-stop:.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level2 - {mso-level-tab-stop:1.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level3 - {mso-level-tab-stop:1.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level4 - {mso-level-tab-stop:2.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level5 - {mso-level-tab-stop:2.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level6 - {mso-level-tab-stop:3.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level7 - {mso-level-tab-stop:3.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level8 - {mso-level-tab-stop:4.0in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l5:level9 - {mso-level-tab-stop:4.5in; - mso-level-number-position:left; - text-indent:-.25in;} -@list l6 - {mso-list-id:1929534752; - mso-list-template-ids:-760440376;} -@list l7 - {mso-list-id:2059475982; - mso-list-template-ids:658286432;} -ol - {margin-bottom:0in;} -ul - {margin-bottom:0in;} ---> -</style> -<!--[if gte mso 10]> -<style> - /* Style Definitions */ - table.MsoNormalTable - {mso-style-name:"Table Normal"; - mso-tstyle-rowband-size:0; - mso-tstyle-colband-size:0; - mso-style-noshow:yes; - mso-style-parent:""; - mso-padding-alt:0in 5.4pt 0in 5.4pt; - mso-para-margin:0in; - mso-para-margin-bottom:.0001pt; - mso-pagination:widow-orphan; - font-size:10.0pt; - font-family:"Times New Roman";} -</style> -<![endif]--><!--[if gte mso 9]><xml> - <o:shapedefaults v:ext="edit" spidmax="3074"/> -</xml><![endif]--><!--[if gte mso 9]><xml> - <o:shapelayout v:ext="edit"> - <o:idmap v:ext="edit" data="1"/> - </o:shapelayout></xml><![endif]--> -</head> - -<body lang=EN-US style='tab-interval:.5in'> - -<div class=Section1> - -<p class=MsoNormal align=center style='text-align:center'><b><span -style='font-size:16.0pt'>Arithmetic Data Transforms<o:p></o:p></span></b></p> - -<p class=MsoNormal align=center style='text-align:center'>Leon Arber, Albert -Cheng, William Wendling<a style='mso-footnote-id:ftn1' href="#_ftn1" -name="_ftnref1" title=""><span class=MsoFootnoteReference><span -style='mso-special-character:footnote'><![if !supportFootnotes]><span -class=MsoFootnoteReference><span style='font-size:12.0pt;font-family:"Times New Roman"; -mso-fareast-font-family:"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language: -EN-US;mso-bidi-language:AR-SA'>[1]</span></span><![endif]></span></span></a></p> - -<p class=MsoNormal align=center style='text-align:center'><st1:date Year="2004" -Day="10" Month="12">December 10, 2004</st1:date></p> - -<h1>Purpose</h1> - -<p class=MsoNormal>Data can be stored and represented in many different -ways.<span style='mso-spacerun:yes'> </span>In most fields of science, for -example, the metric system is used for storing all data.<span -style='mso-spacerun:yes'> </span>However, many fields of engineering still use -the English system.<span style='mso-spacerun:yes'> </span>In such scenarios, -there needs to be a way to easily perform arbitrary scaling of data.<span -style='mso-spacerun:yes'> </span>The data transforms provide just such -functionality.<span style='mso-spacerun:yes'> </span>They allow arbitrary -arithmetic expressions to be applied to a dataset during read and write -operations.<span style='mso-spacerun:yes'> </span>This means that data can be -stored in Celsius in a data file, but read in and automatically converted to -Fahrenheit.<span style='mso-spacerun:yes'> </span>Alternatively, data that is -obtained in Fahrenheit can be written out to the data file in Celsius.<span -style='mso-spacerun:yes'> </span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Although a user can always manually modify the data they -read and write, having the data transform as a property means that the user -doesn’t have to worry about forgetting to call the conversion function or even -writing it in the first place.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>Usage</h1> - -<p class=MsoNormal>The data transform functionality is implemented as a -property that is set on a dataset transfer property list.<span -style='mso-spacerun:yes'> </span>There are two functions available: one for -setting the transform and another for finding out what transform, if any, is -currently set.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>The function for setting the transform is:</p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>herr_t -H5Pset_data_transform(hid_t plist_id, const char* expression)<o:p></o:p></span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>plist_id -</span>is the identifier of the dataset transfer property list on which the -data transform property should be set.</p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>expression -</span>is a pointer to a string of the form “(5/9.0)*(x-32)” which describes -the transform.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>The function for getting the transform is: </p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>ssize_t -H5Pget_data_transform(hid_t plist_id, char* expression, size_t size)<o:p></o:p></span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>plist_id -</span>is the identifier of the dataset transfer property list which will be -queried for its data transform property.</p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>expression -</span>is either NULL or a pointer to memory where the data transform string, -if present, will be copied.</p> - -<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>size -</span>is the number of bytes to copy from the transform string into -expression. <span style='mso-spacerun:yes'> </span>H5Pget_data_transform will -never copy more than the length of the transform expression.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>Data Transform Expressions</h1> - -<p class=MsoNormal>Data transforms are set by passing a pointer to a string, -which is the data transform expression.<span style='mso-spacerun:yes'> -</span>This string describes what sort of arithmetic transform should be done -during data transfer of read or write.<span style='mso-spacerun:yes'> -</span>The string is a standard mathematical expression, as would be entered -into a something like MATLAB.<span style='mso-spacerun:yes'> </span></p> - -<p class=MsoNormal>Expressions are defined by the following context-free -grammar:</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>expr:=<span style='mso-spacerun:yes'> </span>term | term + -term | term - term</p> - -<p class=MsoNormal>term := factor | factor * factor | factor / factor</p> - -<p class=MsoNormal>factor :=<span style='mso-spacerun:yes'> </span>number | -symbol | - factor | + factor | ( expr )</p> - -<p class=MsoNormal>symbol := [a-zA-Z][a-zA-Z0-9]*</p> - -<p class=MsoNormal>number := INT | FLOAT</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>where INT is interpreted as a C long int and FLOAT is interpreted -as a C double</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>This grammar allows for order of operations (multiplication -and dividision take precedence over addition and subtraction), floating and -integer constants, and grouping of terms by way of parentheses.<span -style='mso-spacerun:yes'> </span>Although the grammar allows symbols to be -arbitrary strings, this documentation will always use ‘x’ for symbols.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Within a transform expression, the symbol represents a -variable which contains the data to be manipulated.<span -style='mso-spacerun:yes'> </span>For this reason, the terms symbol and -variable will be used interchangeably.<span style='mso-spacerun:yes'> -</span>Furthermore, in the current implementation of data transforms, all -symbols appearing in an expression are interpreted as referring to the same -dataset.<span style='mso-spacerun:yes'> </span>So, an expression such as -“alpha + 5” is equivalent to “x+5” and an expression such as “alpha + 3*beta + -5” is equivalent to “alpha + 3*alpha + 5” which is equivalent to “4*x + -5”.<span style='mso-spacerun:yes'> </span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>Data Transform Implementation</h1> - -<p class=MsoNormal>When the data transform property of a dataset transfer -property list is set, a parse tree of the expression is immediately generated -and its root is saved in the property list.<span style='mso-spacerun:yes'> -</span>The generation of the parse involves several steps.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>First, the expression is reduced, so as to simply the final -parse and speed up the transform operations.<span style='mso-spacerun:yes'> -</span>Expressions such as “(5/9.0) * (x-32)” will be reduced to -“.555555*(x-32).”<span style='mso-spacerun:yes'> </span>While further -simplification is algebraically possible, the data transform code will only -reduce simple trivial arithmetic operations.<span style='mso-spacerun:yes'> -</span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Then, this reduced expression is parsed into a set of -tokens, from which the parse tree is generated.<span style='mso-spacerun:yes'> -</span>From the expression “(5/9.0)*(x-32),” for example, the following parse -tree would be created:</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><span style='mso-tab-count:4'> </span><span -style='mso-spacerun:yes'> </span>*</p> - -<p class=MsoNormal><span style='mso-tab-count:4'> </span>/<span -style='mso-tab-count:1'> </span>\<span style='mso-spacerun:yes'> -</span></p> - -<p class=MsoNormal><span style='mso-tab-count:3'> </span><span -style='mso-spacerun:yes'> </span>.555555<span -style='mso-spacerun:yes'> </span>-</p> - -<p class=MsoNormal><span style='mso-tab-count:5'> </span>/<span -style='mso-spacerun:yes'> </span>\<span style='mso-tab-count:5'> </span></p> - -<p class=MsoNormal><span style='mso-tab-count:4'> </span><span -style='mso-spacerun:yes'> </span>x<span style='mso-spacerun:yes'> -</span>32</p> - -<p class=MsoNormal><span style='mso-tab-count:1'> </span></p> - -<h2>HDread with Data Transform Expressions</h2> - -<p class=MsoNormal>When a read is performed with a dataset transfer property -list that has the data transform property set, the following sequence of events -occurs:</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<ol style='margin-top:0in' start=1 type=1> - <li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>A - piece of the file is read into memory</li> - <li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>The - data transform is performed on this piece of memory</li> - <li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>This piece - of memory is then copied to the user</li> - <li class=MsoNormal style='mso-list:l2 level1 lfo3;tab-stops:list .5in'>Steps - 1 – 3 are repeated until the read is complete.</li> -</ol> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Step 2 works like this:</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<ol style='margin-top:0in' start=1 type=1> - <li class=MsoNormal style='mso-list:l1 level1 lfo6;tab-stops:list .5in'>The - function responsible for doing the transform is passed a buffer and is - informed what type of data is inside this buffer and how many elements - there are.</li> - <li class=MsoNormal style='mso-list:l1 level1 lfo6;tab-stops:list .5in'>This - buffer is then treated as the variable in the data transform expression - and the transform expression is applied.</li> - <li class=MsoNormal style='mso-list:l1 level1 lfo6;tab-stops:list .5in'>The - transformed buffer is returned to the library.</li> -</ol> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>If the transform expression is “(5/9.0)*(x-32),” with the -parse tree shown above and the buffer contains [-10 0 10 50 100], then the -intermediate steps involved in the transform are:</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<ol style='margin-top:0in' start=1 type=1> - <li class=MsoNormal style='mso-pagination:widow-orphan lines-together; - page-break-after:avoid;mso-list:l5 level1 lfo9;mso-hyphenate:none; - tab-stops:list .5in'>First, the (x-32) subexpression is evaluated.<span - style='mso-spacerun:yes'> </span>Now the buffer would contain<span - style='mso-spacerun:yes'> </span>[-42 -32 -22 18 68]</li> - <li class=MsoNormal style='mso-list:l5 level1 lfo9;tab-stops:list .5in'>Then, - the .55555 * part of the expression is evaluated.<span - style='mso-spacerun:yes'> </span>Now the buffer would contain: [-23.3333 - -17.7777 -12.2222 9.9999 37.7777]</li> - <li class=MsoNormal style='mso-list:l5 level1 lfo9;tab-stops:list .5in'>Now, - the transform would be completed and the resulting buffer returned.</li> -</ol> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Note that the original data in the file was not modified. </p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h2>HDwrite with Data Transform Expressions</h2> - -<p class=MsoNormal>The process of a write works much the same way, but in the -reverse order.<span style='mso-spacerun:yes'> </span>When a file is written -out with a dataset transfer property list that has the data transform property -set:</p> - -<p class=MsoNormal><span style='mso-spacerun:yes'> </span></p> - -<ol style='margin-top:0in' start=1 type=1> - <li class=MsoNormal style='mso-list:l4 level1 lfo12;tab-stops:list .5in'>The - user passes a buffer to HDwrite, along with the type and number of - elements.</li> - <li class=MsoNormal style='mso-list:l4 level1 lfo12;tab-stops:list .5in'>The - data transform is performed on a copy of this piece of memory.</li> - <li class=MsoNormal style='mso-list:l4 level1 lfo12;tab-stops:list .5in'>This - copy with the transformed data is then written out to the file.</li> -</ol> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Step 2 works exactly as in the read example.<span -style='mso-spacerun:yes'> </span>Note that the user’s data is not modified. <span -style='mso-spacerun:yes'> </span>Also, since the transform property is not -saved with the dataset, in order to recover the original data, a user must know -the inverse of the transform that was applied in order to recover it.<span -style='mso-spacerun:yes'> </span>In the case of “(5/9.0)*(x-32)” this inverse -would be “(9/5.0)*x + 32”.<span style='mso-spacerun:yes'> </span>Reading from -a data file that had previously been written out with a transform string of -“(5/9.0)*(x-32)” with a transform string of “(9/5.0)*x + 32” would effectively -recover the original data the author of the file had been using.<a -style='mso-footnote-id:ftn2' href="#_ftn2" name="_ftnref2" title=""><span -class=MsoFootnoteReference><span style='mso-special-character:footnote'><![if !supportFootnotes]><span -class=MsoFootnoteReference><span style='font-size:12.0pt;font-family:"Times New Roman"; -mso-fareast-font-family:"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language: -EN-US;mso-bidi-language:AR-SA'>[2]</span></span><![endif]></span></span></a></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>Mixed Mode and Truncation</h1> - -<p class=MsoNormal>Because the data transform sits and modifies data between -the file space and the memory space, various effects can occur that are the -result of the typecasting that may be involved in the operations.<span -style='mso-spacerun:yes'> </span>In addition, because constants in the data -transform expression can be either INT or FLOAT, the data transform itself can -be a source of truncation.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>In the example above, the reason that the transform -expression is always written as “(5/9.0)*(x-32)” is because, if it were written -without a floating point constant, it would always evaluate to 0.<span -style='mso-spacerun:yes'> </span>The expression “(5/9)*(x-32)” would, when -set, get reduced to “0*(x-32)” because both 5 and 9 would get read as C long -ints and, when divided, the result would get truncated to 0.<span -style='mso-spacerun:yes'> </span>This resulting expression, “0*(x-32),” would -cause any data read or written to be saved as an array of all 0’s.<span -style='mso-spacerun:yes'> </span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Another source of unpredictability caused by truncation -occurs when intermediate data is of a type that is more precise than the -destination memory type.<span style='mso-spacerun:yes'> </span>For example, if -the transform expression “(1/2.0)*x” is applied to data read from a file that -is being read into an integer memory buffer, the results can be -unpredictable.<span style='mso-spacerun:yes'> </span>If the source array is [1 -2 3 4], then the resulting array could be either [0 1 1 2] or [0 0 1 1], -depending on the floating point unit of the processors.<span -style='mso-spacerun:yes'> </span>Note that this result is independent of the -source data type.<span style='mso-spacerun:yes'> </span>It doesn’t matter if -the source data is integer or floating point because the 2.0 in the data -transform expression will cause everything to be evaluated in a floating-point -context.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>When setting transform expressions, care must be taken to -ensure that the truncation does not adversely affect the data.<span -style='mso-spacerun:yes'> </span>A workaround for the possible effects of a -transform such as “(1/2.0) * x” would be to used the transform expression -“(1/2.0)*x + 0.5” instead of the original.<span style='mso-spacerun:yes'> -</span>This will ensure that all truncation rounds up, with the possible -exception of a boundary condition.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>Data Transform Example</h1> - -<p class=MsoNormal>The following code snippet shows an example using data -transform, where the data transform property is set and a write is -performed.<span style='mso-spacerun:yes'> </span>Then, a read is performed -with no data transform property set.<span style='mso-spacerun:yes'> </span>It -is assumed that <span style='font-family:Courier'>dataset </span>is a dataset -that has been opened and <span style='font-family:Courier'>windchillF </span>and -<span style='font-family:Courier'>windchillC </span>are both arrays that hold -floating point data.<span style='mso-spacerun:yes'> </span>The result of this -snippet is to fill <span style='font-family:Courier'>windchillC </span>with the -data in <span style='font-family:Courier'>windchillF</span>, converted to -Celcius.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal><span style='font-family:Courier'>hid_t dxpl_id_c_to_f;<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>const char* c_to_f = -“(9/5.0)*x + 32”;<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><o:p> </o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>/* Create the dataset -transfer property list */<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><span -style='mso-spacerun:yes'> </span>dxpl_id_c_to_f = -H5Pcreate(H5P_DATASET_XFER);<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><o:p> </o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>/* Set the data transform -to be used on the read*/<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><span -style='mso-spacerun:yes'> </span>H5Pset_data_transform(dxpl_id_c_to_f, -c_to_f);<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><o:p> </o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><span -style='mso-spacerun:yes'> </span><o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>/*<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>* Write the data to the -dataset using the f_to_c transform<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>*/<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><span -style='mso-spacerun:yes'> </span>status = H5Dwrite(dataset, H5T_NATIVE_FLOAT, -H5S_ALL, H5S_ALL, dxpl_id_f_to_c, windchillF);<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><span -style='mso-spacerun:yes'> </span><o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'>/* Read the data with the -c_to_f data transform */<o:p></o:p></span></p> - -<p class=MsoNormal><span style='font-family:Courier'><span -style='mso-spacerun:yes'> </span>H5Dread(dataset, H5T_NATIVE_FLOAT, H5S_ALL, -H5S_ALL, H5P_DEFAULT, windchillC);<o:p></o:p></span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>H5Pget_data_transform Details</h1> - -<p class=MsoNormal>Querying the data transform string of a dataset transfer -property list requires the use of the H5Pget_data_transform function.<span -style='mso-spacerun:yes'> </span>This function provides the ability to both -query the size of the string stored and retrieve part or all of it.<span -style='mso-spacerun:yes'> </span>Note that H5Pget_data_transform will return -the expression that was set by H5Pset_data_transform.<span -style='mso-spacerun:yes'> </span>The reduced transform string, computed when -H5Pset_data_transform is called, is not stored in string form and is not -available to the user.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>In order to ascertain the size of the string, a NULL <span -style='font-family:Courier'>expression</span> should be passed to the -function.<span style='mso-spacerun:yes'> </span>This will make the function -return the length of the transform string (not including the terminated ‘\0’ -character).</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>To actually retrieve the string, a pointer to a valid memory -location should be passed in for <span style='font-family:Courier'>expression </span>and -the number of bytes from the string that should be copied to that memory -location should be passed in as <span style='font-family:Courier'>size</span>.</p> - -<p class=MsoNormal><o:p> </o:p></p> - -<h1>Further Work</h1> - -<p class=MsoNormal>Some additional functionality can still be added to the data -transform.<span style='mso-spacerun:yes'> </span>Currently the most important -feature lacking is the addition of operators, such as exponentiation and the -trigonometric functions.<span style='mso-spacerun:yes'> </span>Although -exponentiation can be explicitly carried with a transform expression such as -“x*x*x” it may be easier to support expression like “x^3.” Also lacking are the -commonly used trigonometric functions, such as sin, cos, and tan.<span -style='mso-spacerun:yes'> </span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>Popular constants could also be added, such as π or -e.<span style='mso-spacerun:yes'> </span></p> - -<p class=MsoNormal><o:p> </o:p></p> - -<p class=MsoNormal>More advanced functionality, such as the ability to perform -a transform on multiple datasets is also a possibility, but is a feature is -more a completely new addition than an extension to data transforms.<span -style='mso-spacerun:yes'> </span></p> - -</div> - -<div style='mso-element:footnote-list'><![if !supportFootnotes]><br clear=all> - -<hr align=left size=1 width="33%"> - -<![endif]> - -<div style='mso-element:footnote' id=ftn1> - -<p class=MsoFootnoteText><a style='mso-footnote-id:ftn1' href="#_ftnref1" -name="_ftn1" title=""><span class=MsoFootnoteReference><span style='mso-special-character: -footnote'><![if !supportFootnotes]><span class=MsoFootnoteReference><span -style='font-size:10.0pt;font-family:"Times New Roman";mso-fareast-font-family: -"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language:EN-US; -mso-bidi-language:AR-SA'>[1]</span></span><![endif]></span></span></a> Mr. -Wendling, who involved in the initial design and implemented the expression -parser, has left NCSA.</p> - -</div> - -<div style='mso-element:footnote' id=ftn2> - -<p class=MsoFootnoteText><a style='mso-footnote-id:ftn2' href="#_ftnref2" -name="_ftn2" title=""><span class=MsoFootnoteReference><span style='mso-special-character: -footnote'><![if !supportFootnotes]><span class=MsoFootnoteReference><span -style='font-size:10.0pt;font-family:"Times New Roman";mso-fareast-font-family: -"Times New Roman";mso-ansi-language:EN-US;mso-fareast-language:EN-US; -mso-bidi-language:AR-SA'>[2]</span></span><![endif]></span></span></a> See the -h5_dtransform.c example in the examples directory of the hdf5 library for just -such an illustration.</p> - -</div> - -</div> - -</body> - -</html> diff --git a/doc/html/TechNotes/ExternalFiles.html b/doc/html/TechNotes/ExternalFiles.html deleted file mode 100644 index c3197af..0000000 --- a/doc/html/TechNotes/ExternalFiles.html +++ /dev/null @@ -1,279 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>External Files in HDF5</title> - </head> - - <body> - <center><h1>External Files in HDF5</h1></center> - - <h3>Overview of Layers</h3> - - <p>This table shows some of the layers of HDF5. Each layer calls - functions at the same or lower layers and never functions at - higher layers. An object identifier (OID) takes various forms - at the various layers: at layer 0 an OID is an absolute physical - file address; at layers 1 and 2 it's an absolute virtual file - address. At layers 3 through 6 it's a relative address, and at - layers 7 and above it's an object handle. - - <p><center> - <table border cellpadding=4 width="60%"> - <tr align=center> - <td>Layer-7</td> - <td>Groups</td> - <td>Datasets</td> - </tr> - <tr align=center> - <td>Layer-6</td> - <td>Indirect Storage</td> - <td>Symbol Tables</td> - </tr> - <tr align=center> - <td>Layer-5</td> - <td>B-trees</td> - <td>Object Hdrs</td> - <td>Heaps</td> - </tr> - <tr align=center> - <td>Layer-4</td> - <td>Caching</td> - </tr> - <tr align=center> - <td>Layer-3</td> - <td>H5F chunk I/O</td> - </tr> - <tr align=center> - <td>Layer-2</td> - <td>H5F low</td> - </tr> - <tr align=center> - <td>Layer-1</td> - <td>File Family</td> - <td>Split Meta/Raw</td> - </tr> - <tr align=center> - <td>Layer-0</td> - <td>Section-2 I/O</td> - <td>Standard I/O</td> - <td>Malloc/Free</td> - </tr> - </table> - </center> - - <h3>Single Address Space</h3> - - <p>The simplest form of hdf5 file is a single file containing only - hdf5 data. The file begins with the boot block, which is - followed until the end of the file by hdf5 data. The next most - complicated file allows non-hdf5 data (user defined data or - internal wrappers) to appear before the boot block and after the - end of the hdf5 data. The hdf5 data is treated as a single - linear address space in both cases. - - <p>The next level of complexity comes when non-hdf5 data is - interspersed with the hdf5 data. We handle that by including - the non-hdf5 interspersed data in the hdf5 address space and - simply not referencing it (eventually we might add those - addresses to a "do-not-disturb" list using the same mechanism as - the hdf5 free list, but it's not absolutely necessary). This is - implemented except for the "do-not-disturb" list. - - <p>The most complicated single address space hdf5 file is when we - allow the address space to be split among multiple physical - files. For instance, a >2GB file can be split into smaller - chunks and transfered to a 32 bit machine, then accessed as a - single logical hdf5 file. The library already supports >32 bit - addresses, so at layer 1 we split a 64-bit address into a 32-bit - file number and a 32-bit offset (the 64 and 32 are - arbitrary). The rest of the library still operates with a linear - address space. - - <p>Another variation might be a family of two files where all the - meta data is stored in one file and all the raw data is stored - in another file to allow the HDF5 wrapper to be easily replaced - with some other wrapper. - - <p>The <code>H5Fcreate</code> and <code>H5Fopen</code> functions - would need to be modified to pass file-type info down to layer 2 - so the correct drivers can be called and parameters passed to - the drivers to initialize them. - - <h4>Implementation</h4> - - <p>I've implemented fixed-size family members. The entire hdf5 - file is partitioned into members where each member is the same - size. The family scheme is used if one passes a name to - <code>H5F_open</code> (which is called by <code>H5Fopen()</code> - and <code>H5Fcreate</code>) that contains a - <code>printf(3c)</code>-style integer format specifier. - Currently, the default low-level file driver is used for all - family members (H5F_LOW_DFLT, usually set to be Section 2 I/O or - Section 3 stdio), but we'll probably eventually want to pass - that as a parameter of the file access property list, which - hasn't been implemented yet. When creating a family, a default - family member size is used (defined at the top H5Ffamily.c, - currently 64MB) but that also should be settable in the file - access property list. When opening an existing family, the size - of the first member is used to determine the member size - (flushing/closing a family ensures that the first member is the - correct size) but the other family members don't have to be that - large (the local address space, however, is logically the same - size for all members). - - <p>I haven't implemented a split meta/raw family yet but am rather - curious to see how it would perform. I was planning to use the - `.h5' extension for the meta data file and `.raw' for the raw - data file. The high-order bit in the address would determine - whether the address refers to meta data or raw data. If the user - passes a name that ends with `.raw' to <code>H5F_open</code> - then we'll chose the split family and use the default low level - driver for each of the two family members. Eventually we'll - want to pass these kinds of things through the file access - property list instead of relying on naming convention. - - <h3>External Raw Data</h3> - - <p>We also need the ability to point to raw data that isn't in the - HDF5 linear address space. For instance, a dataset might be - striped across several raw data files. - - <p>Fortunately, the only two packages that need to be aware of - this are the packages for reading/writing contiguous raw data - and discontiguous raw data. Since contiguous raw data is a - special case, I'll discuss how to implement external raw data in - the discontiguous case. - - <p>Discontiguous data is stored as a B-tree whose keys are the - chunk indices and whose leaf nodes point to the raw data by - storing a file address. So what we need is some way to name the - external files, and a way to efficiently store the external file - name for each chunk. - - <p>I propose adding to the object header an <em>External File - List</em> message that is a 1-origin array of file names. - Then, in the B-tree, each key has an index into the External - File List (or zero for the HDF5 file) for the file where the - chunk can be found. The external file index is only used at - the leaf nodes to get to the raw data (the entire B-tree is in - the HDF5 file) but because of the way keys are copied among - the B-tree nodes, it's much easier to store the index with - every key. - - <h3>Multiple HDF5 Files</h3> - - <p>One might also want to combine two or more HDF5 files in a - manner similar to mounting file systems in Unix. That is, the - group structure and meta data from one file appear as though - they exist in the first file. One opens File-A, and then - <em>mounts</em> File-B at some point in File-A, the <em>mount - point</em>, so that traversing into the mount point actually - causes one to enter the root object of File-B. File-A and - File-B are each complete HDF5 files and can be accessed - individually without mounting them. - - <p>We need a couple additional pieces of machinery to make this - work. First, an haddr_t type (a file address) doesn't contain - any info about which HDF5 file's address space the address - belongs to. But since haddr_t is an opaque type except at - layers 2 and below, it should be quite easy to add a pointer to - the HDF5 file. This would also remove the H5F_t argument from - most of the low-level functions since it would be part of the - OID. - - <p>The other thing we need is a table of mount points and some - functions that understand them. We would add the following - table to each H5F_t struct: - - <p><code><pre> -struct H5F_mount_t { - H5F_t *parent; /* Parent HDF5 file if any */ - struct { - H5F_t *f; /* File which is mounted */ - haddr_t where; /* Address of mount point */ - } *mount; /* Array sorted by mount point */ - intn nmounts; /* Number of mounted files */ - intn alloc; /* Size of mount table */ -} - </pre></code> - - <p>The <code>H5Fmount</code> function takes the ID of an open - file or group, the name of a to-be-mounted file, the name of the mount - point, and a file access property list (like <code>H5Fopen</code>). - It opens the new file and adds a record to the parent's mount - table. The <code>H5Funmount</code> function takes the parent - file or group ID and the name of the mount point and disassociates - the mounted file from the mount point. It does not close the - mounted file. The <code>H5Fclose</code> - function closes/unmounts files recursively. - - <p>The <code>H5G_iname</code> function which translates a name to - a file address (<code>haddr_t</code>) looks at the mount table - at each step in the translation and switches files where - appropriate. All name-to-address translations occur through - this function. - - <h3>How Long?</h3> - - <p>I'm expecting to be able to implement the two new flavors of - single linear address space in about two days. It took two hours - to implement the malloc/free file driver at level zero and I - don't expect this to be much more work. - - <p>I'm expecting three days to implement the external raw data for - discontiguous arrays. Adding the file index to the B-tree is - quite trivial; adding the external file list message shouldn't - be too hard since the object header message class from wich this - message derives is fully implemented; and changing - <code>H5F_istore_read</code> should be trivial. Most of the - time will be spent designing a way to cache Unix file - descriptors efficiently since the total number open files - allowed per process could be much smaller than the total number - of HDF5 files and external raw data files. - - <p>I'm expecting four days to implement being able to mount one - HDF5 file on another. I was originally planning a lot more, but - making <code>haddr_t</code> opaque turned out to be much easier - than I planned (I did it last Fri). Most of the work will - probably be removing the redundant H5F_t arguments for lots of - functions. - - <h3>Conclusion</h3> - - <p>The external raw data could be implemented as a single linear - address space, but doing so would require one to allocate large - enough file addresses throughout the file (>32bits) before the - file was created. It would make mixing an HDF5 file family with - external raw data, or external HDF5 wrapper around an HDF4 file - a more difficult process. So I consider the implementation of - external raw data files as a single HDF5 linear address space a - kludge. - - <p>The ability to mount one HDF5 file on another might not be a - very important feature especially since each HDF5 file must be a - complete file by itself. It's not possible to stripe an array - over multiple HDF5 files because the B-tree wouldn't be complete - in any one file, so the only choice is to stripe the array - across multiple raw data files and store the B-tree in the HDF5 - file. On the other hand, it might be useful if one file - contains some public data which can be mounted by other files - (e.g., a mesh topology shared among collaborators and mounted by - files that contain other fields defined on the mesh). Of course - the applications can open the two files separately, but it might - be more portable if we support it in the library. - - <p>So we're looking at about two weeks to implement all three - versions. I didn't get a chance to do any of them in AIO - although we had long-term plans for the first two with a - possibility of the third. They'll be much easier to implement in - HDF5 than AIO since I've been keeping these in mind from the - start. - - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Sat Nov 8 18:08:52 EST 1997 --> -<!-- hhmts start --> -Last modified: Tue Sep 8 14:43:32 EDT 1998 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/FreeLists.html b/doc/html/TechNotes/FreeLists.html deleted file mode 100644 index 1a4b8e8..0000000 --- a/doc/html/TechNotes/FreeLists.html +++ /dev/null @@ -1,205 +0,0 @@ -<html> -<head> -<title>Memory Management and Free Lists</title> -</head> - -<body> - -<h1>Memory Management and Free Lists</h1> - -<pre> - -At Release 1.2.2, free list management code was introduced to the HDF5 -library. This included one user-level function, H5garbage_collect, which -garbage collects on all the free-lists. H5garbage_collect is the only user- -accessible (i.e., application developer-accessible) element of this -functionality. - -The free-lists generally reduce the amount of dynamic memory used to around -75% of the pre-optimized amount as well as speed up the time in the library -code by ~5% The free-lists also help linearize the amount of memory used with -increasing numbers of datasets or re-writes on the data, so the amount of -memory used for the 1500/45 free-list case is only 66% of the memory used for -the unoptimized case. - -Overall, the introduction of free list management is a win: the library is -slightly faster and uses much less system resources than before. Most of the -emphasis has been focused on the main "thouroughfares" through the code; -less attention was paid to the "back streets" which are used much less -frequently and offer less potential for abuse. - -Adding a free-list for a data structure in the HDF5 library code is easy: - -Old code: ---------- - int foo(void) - { - H5W_t *w; - - for(i=0; i<x; i++) { - w=H5MM_malloc(sizeof(H5W_t)); - <use w> - H5MM_xfree(w); - } - } - -New code: ---------- -H5FL_DEFINE(H5W_t); - - int foo(void) - { - H5W_t *w; - - for(i=0; i<x; i++) { - w=H5FL_ALLOC(H5W_t,0); - <use w> - H5FL_FREE(H5W_t,w); - } - } - - -There are three kinds of free-lists: - -- for "regular" objects, - -- arrays of fixed size object (both fixed length and unknown length), and - -- "blocks" of bytes. - - "Regular" free-lists use the H5FL_<*> macros in H5FLprivate.h and are - designed for single, fixed-size data structures like typedef'ed structs, - etc. - - Arrays of objects use the H5FL_ARR_<*> macros and are designed for arrays - (both fixed in length and varying lengths) of fixed length data structures - (like typedef'ed types). - - "Block" free-lists use the H5FL_BLK_<*> macros and are designed to hold - varying sized buffers of bytes, with no structure. - - H5S.c contains examples for "regular" and fixed-sized arrays; - H5B.c contains examples for variable-sized arrays and "blocks". - -A free-list doesn't have to be used for every data structure allocated and -freed, just for those which are prone to abuse when multiple operations are -being performed. It is important to use the macros for declaring and -manipulating the free-lists however; they allow the free'd objects on the -lists to be garbage collected by the library at the library's termination -or at the user's request. - -One public API function has been added: H5garbage_collect, which garbage -collects on all the free-lists of all the different types. It's not required -to be called and is only necessary in situations when the application -performs actions which cause the library to allocate many objects and then -the application eventually releases those objects and wants to reduce the -memory used by the library from the peak usage required. The library -automatically garbage collects all the free lists when the application ends. - -Questions should be sent to the HDF Help Desk at hdfhelp@ncsa.uiuc.edu. - - -=========================================== -BENCHMARK INFORMATION -=========================================== - -New version with free lists: - -Datasets=500, Data Rewrites=15: - Peak Heap Usage: 18210820 bytes - Time in library code: 2.260 seconds - # of malloc calls: 22864 - -Datasets=1000, Data Rewrites=15: - Peak Heap Usage: 31932420 bytes - Time in library code: 5.090 seconds - # of malloc calls: 43045 - -Datasets=1500, Data Rewrites=15: - Peak Heap Usage: 41566212 bytes - Time in library code: 8.623 seconds - # of malloc calls: 60623 - -Datasets=500, Data Rewrites=30: - Peak Heap Usage: 19456004 bytes - Time in library code: 4.274 seconds - # of malloc calls: 23353 - -Datasets=1000, Data Rewrites=30: - Peak Heap Usage: 33988612 bytes - Time in library code: 9.955 seconds - # of malloc calls: 43855 - -Datasets=1500, Data Rewrites=30: - Peak Heap Usage: 43950084 bytes - Time in library code: 17.413 seconds - # of malloc calls: 61554 - -Datasets=500, Data Rewrites=45: - Peak Heap Usage: 20717572 bytes - Time in library code: 6.326 seconds - # of malloc calls: 23848 - -Datasets=1000, Data Rewrites=45: - Peak Heap Usage: 35807236 bytes - Time in library code: 15.146 seconds - # of malloc calls: 44572 - -Datasets=1500, Data Rewrites=45: - Peak Heap Usage: 46022660 bytes - Time in library code: 27.140 seconds - # of malloc calls: 62370 - - -Older version with no free lists: - -Datasets=500, Data Rewrites=15: - Peak Heap Usage: 25370628 bytes - Time in library code: 2.329 seconds - # of malloc calls: 194991 - -Datasets=1000, Data Rewrites=15: - Peak Heap Usage: 39550980 bytes - Time in library code: 5.251 seconds - # of malloc calls: 417971 - -Datasets=1500, Data Rewrites=15: - Peak Heap Usage: 68870148 bytes - Time in library code: 8.913 seconds - # of malloc calls: 676564 - -Datasets=500, Data Rewrites=30: - Peak Heap Usage: 31670276 bytes - Time in library code: 4.435 seconds - # of malloc calls: 370320 - -Datasets=1000, Data Rewrites=30: - Peak Heap Usage: 44646404 bytes - Time in library code: 10.325 seconds - # of malloc calls: 797125 - -Datasets=1500, Data Rewrites=30: - Peak Heap Usage: 68870148 bytes - Time in library code: 18.057 seconds - # of malloc calls: 1295336 - -Datasets=500, Data Rewrites=45: - Peak Heap Usage: 33906692 bytes - Time in library code: 6.577 seconds - # of malloc calls: 545656 - -Datasets=1000, Data Rewrites=45: - Peak Heap Usage: 56778756 bytes - Time in library code: 15.720 seconds - # of malloc calls: 1176285 - -Datasets=1500, Data Rewrites=45: - Peak Heap Usage: 68870148 bytes - Time in library code: 28.138 seconds - # of malloc calls: 1914097 - - -=========================================== -Last Modified: 3 May 2000 -HDF Help Desk: hdfhelp@ncsa.uiuc.edu - -</pre> -</body> -</html> diff --git a/doc/html/TechNotes/H4-H5Compat.html b/doc/html/TechNotes/H4-H5Compat.html deleted file mode 100644 index 2992476..0000000 --- a/doc/html/TechNotes/H4-H5Compat.html +++ /dev/null @@ -1,271 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Backward/Forward Compatability</title> - </head> - - <body> - <h1>Backward/Forward Compatability</h1> - - <p>The HDF5 development must proceed in such a manner as to - satisfy the following conditions: - - <ol type=A> - <li>HDF5 applications can produce data that HDF5 - applications can read and write and HDF4 applications can produce - data that HDF4 applications can read and write. The situation - that demands this condition is obvious.</li> - - <li>HDF5 applications are able to produce data that HDF4 applications - can read and HDF4 applications can subsequently modify the - file subject to certain constraints depending on the - implementation. This condition is for the temporary - situation where a consumer has neither been relinked with a new - HDF4 API built on top of the HDF5 API nor recompiled with the - HDF5 API.</li> - - <li>HDF5 applications can read existing HDF4 files and subsequently - modify the file subject to certain constraints depending on - the implementation. This is condition is for the temporary - situation in which the producer has neither been relinked with a - new HDF4 API built on top of the HDF5 API nor recompiled with - the HDF5 API, or the permanent situation of HDF5 consumers - reading archived HDF4 files.</li> - </ul> - - <p>There's at least one invarient: new object features introduced - in the HDF5 file format (like 2-d arrays of structs) might be - impossible to "translate" to a format that an old HDF4 - application can understand either because the HDF4 file format - or the HDF4 API has no mechanism to describe the object. - - <p>What follows is one possible implementation based on how - Condition B was solved in the AIO/PDB world. It also attempts - to satisfy these goals: - - <ol type=1> - <li>The main HDF5 library contains as little extra baggage as - possible by either relying on external programs to take care - of compatability issues or by incorporating the logic of such - programs as optional modules in the HDF5 library. Conditions B - and C are separate programs/modules.</li> - - <li>No extra baggage not only means the library proper is small, - but also means it can be implemented (rather than migrated - from HDF4 source) from the ground up with minimal regard for - HDF4 thus keeping the logic straight forward.</li> - - <li>Compatability issues are handled behind the scenes when - necessary (and possible) but can be carried out explicitly - during things like data migration.</li> - </ol> - - <hr> - <h2>Wrappers</h2> - - <p>The proposed implementation uses <i>wrappers</i> to handle - compatability issues. A Format-X file is <i>wrapped</i> in a - Format-Y file by creating a Format-Y skeleton that replicates - the Format-X meta data. The Format-Y skeleton points to the raw - data stored in Format-X without moving the raw data. The - restriction is that raw data storage methods in Format-Y is a - superset of raw data storage methods in Format-X (otherwise the - raw data must be copied to Format-Y). We're assuming that meta - data is small wrt the entire file. - - <p>The wrapper can be a separate file that has pointers into the - first file or it can be contained within the first file. If - contained in a single file, the file can appear as a Format-Y - file or simultaneously a Format-Y and Format-X file. - - <p>The Format-X meta-data can be thought of as the original - wrapper around raw data and Format-Y is a second wrapper around - the same data. The wrappers are independend of one another; - modifying the meta-data in one wrapper causes the other to - become out of date. Modification of raw data doesn't invalidate - either view as long as the meta data that describes its storage - isn't modifed. For instance, an array element can change values - if storage is already allocated for the element, but if storage - isn't allocated then the meta data describing the storage must - change, invalidating all wrappers but one. - - <p>It's perfectly legal to modify the meta data of one wrapper - without modifying the meta data in the other wrapper(s). The - illegal part is accessing the raw data through a wrapper which - is out of date. - - <p>If raw data is wrapped by more than one internal wrapper - (<i>internal</i> means that the wrapper is in the same file as - the raw data) then access to that file must assume that - unreferenced parts of that file contain meta data for another - wrapper and cannot be reclaimed as free memory. - - <hr> - <h2>Implementation of Condition B</h2> - - <p>Since this is a temporary situation which can't be - automatically detected by the HDF5 library, we must rely - on the application to notify the HDF5 library whether or not it - must satisfy Condition B. (Even if we don't rely on the - application, at some point someone is going to remove the - Condition B constraint from the library.) So the module that - handles Condition B is conditionally compiled and then enabled - on a per-file basis. - - <p>If the application desires to produce an HDF4 file (determined - by arguments to <code>H5Fopen</code>), and the Condition B - module is compiled into the library, then <code>H5Fclose</code> - calls the module to traverse the HDF5 wrapper and generate an - additional internal or external HDF4 wrapper (wrapper specifics - are described below). If Condition B is implemented as a module - then it can benefit from the metadata already cached by the main - library. - - <p>An internal HDF4 wrapper would be used if the HDF5 file is - writable and the user doesn't mind that the HDF5 file is - modified. An external wrapper would be used if the file isn't - writable or if the user wants the data file to be primarily HDF5 - but a few applications need an HDF4 view of the data. - - <p>Modifying through the HDF5 library an HDF5 file that has - internal HDF4 wrapper should invalidate the HDF4 wrapper (and - optionally regenerate it when <code>H5Fclose</code> is - called). The HDF5 library must understand how wrappers work, but - not necessarily anything about the HDF4 file format. - - <p>Modifying through the HDF5 library an HDF5 file that has an - external HDF4 wrapper will cause the HDF4 wrapper to become out - of date (but possibly regenerated during <code>H5Fclose</code>). - <b>Note: Perhaps the next release of the HDF4 library should - insure that the HDF4 wrapper file has a more recent modification - time than the raw data file (the HDF5 file) to which it - points(?)</b> - - <p>Modifying through the HDF4 library an HDF5 file that has an - internal or external HDF4 wrapper will cause the HDF5 wrapper to - become out of date. However, there is now way for the old HDF4 - library to notify the HDF5 wrapper that it's out of date. - Therefore the HDF5 library must be able to detect when the HDF5 - wrapper is out of date and be able to fix it. If the HDF4 - wrapper is complete then the easy way is to ignore the original - HDF5 wrapper and generate a new one from the HDF4 wrapper. The - other approach is to compare the HDF4 and HDF5 wrappers and - assume that if they differ HDF4 is the right one, if HDF4 omits - data then it was because HDF4 is a partial wrapper (rather than - assume HDF4 deleted the data), and if HDF4 has new data then - copy the new meta data to the HDF5 wrapper. On the other hand, - perhaps we don't need to allow these situations (modifying an - HDF5 file with the old HDF4 library and then accessing it with - the HDF5 library is either disallowed or causes HDF5 objects - that can't be described by HDF4 to be lost). - - <p>To convert an HDF5 file to an HDF4 file on demand, one simply - opens the file with the HDF4 flag and closes it. This is also - how AIO implemented backward compatability with PDB in its file - format. - - <hr> - <h2>Implementation of Condition C</h2> - - <p>This condition must be satisfied for all time because there - will always be archived HDF4 files. If a pure HDF4 file (that - is, one without HDF5 meta data) is opened with an HDF5 library, - the <code>H5Fopen</code> builds an internal or external HDF5 - wrapper and then accesses the raw data through that wrapper. If - the HDF5 library modifies the file then the HDF4 wrapper becomes - out of date. However, since the HDF5 library hasn't been - released, we can at least implement it to disable and/or reclaim - the HDF4 wrapper. - - <p>If an external and temporary HDF5 wrapper is desired, the - wrapper is created through the cache like all other HDF5 files. - The data appears on disk only if a particular cached datum is - preempted. Instead of calling <code>H5Fclose</code> on the HDF5 - wrapper file we call <code>H5Fabort</code> which immediately - releases all file resources without updating the file, and then - we unlink the file from Unix. - - <hr> - <h2>What do wrappers look like?</h2> - - <p>External wrappers are quite obvious: they contain only things - from the format specs for the wrapper and nothing from the - format specs of the format which they wrap. - - <p>An internal HDF4 wrapper is added to an HDF5 file in such a way - that the file appears to be both an HDF4 file and an HDF5 - file. HDF4 requires an HDF4 file header at file offset zero. If - a user block is present then we just move the user block down a - bit (and truncate it) and insert the minimum HDF4 signature. - The HDF4 <code>dd</code> list and any other data it needs are - appended to the end of the file and the HDF5 signature uses the - logical file length field to determine the beginning of the - trailing part of the wrapper. - - <p> - <center> - <table border width="60%"> - <tr> - <td>HDF4 minimal file header. Its main job is to point to - the <code>dd</code> list at the end of the file.</td> - </tr> - <tr> - <td>User-defined block which is truncated by the size of the - HDF4 file header so that the HDF5 boot block file address - doesn't change.</td> - </tr> - <tr> - <td>The HDF5 boot block and data, unmodified by adding the - HDF4 wrapper.</td> - </tr> - <tr> - <td>The main part of the HDF4 wrapper. The <code>dd</code> - list will have entries for all parts of the file so - hdpack(?) doesn't (re)move anything.</td> - </tr> - </table> - </center> - - <p>When such a file is opened by the HDF5 library for - modification it shifts the user block back down to address zero - and fills with zeros, then truncates the file at the end of the - HDF5 data or adds the trailing HDF4 wrapper to the free - list. This prevents HDF4 applications from reading the file with - an out of date wrapper. - - <p>If there is no user block then we have a problem. The HDF5 - boot block must be moved to make room for the HDF4 file header. - But moving just the boot block causes problems because all file - addresses stored in the file are relative to the boot block - address. The only option is to shift the entire file contents - by 512 bytes to open up a user block (too bad we don't have - hooks into the Unix i-node stuff so we could shift the entire - file contents by the size of a file system page without ever - performing I/O on the file :-) - - <p>Is it possible to place an HDF5 wrapper in an HDF4 file? I - don't know enough about the HDF4 format, but I would suspect it - might be possible to open a hole at file address 512 (and - possibly before) by moving some things to the end of the file - to make room for the HDF5 signature. The remainder of the HDF5 - wrapper goes at the end of the file and entries are added to the - HDF4 <code>dd</code> list to mark the location(s) of the HDF5 - wrapper. - - <hr> - <h2>Other Thoughts</h2> - - <p>Conversion programs that copy an entire HDF4 file to a separate, - self-contained HDF5 file and vice versa might be useful. - - - - - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Fri Oct 3 11:52:31 EST 1997 --> -<!-- hhmts start --> -Last modified: Wed Oct 8 12:34:42 EST 1997 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/HeapMgmt.html b/doc/html/TechNotes/HeapMgmt.html deleted file mode 100644 index e9e8db4..0000000 --- a/doc/html/TechNotes/HeapMgmt.html +++ /dev/null @@ -1,84 +0,0 @@ -<html> -<body> - -<h1>Heap Management in HDF5</h1> - -<pre> - -Heap functions are in the H5H package. - - -off_t -H5H_new (hdf5_file_t *f, size_t size_hint, size_t realloc_hint); - - Creates a new heap in the specified file which can efficiently - store at least SIZE_HINT bytes. The heap can store more than - that, but doing so may cause the heap to become less efficient - (for instance, a heap implemented as a B-tree might become - discontigous). The REALLOC_HINT is the minimum number of bytes - by which the heap will grow when it must be resized. The hints - may be zero in which case reasonable (but probably not - optimal) values will be chosen. - - The return value is the address of the new heap relative to - the beginning of the file boot block. - -off_t -H5H_insert (hdf5_file_t *f, off_t addr, size_t size, const void *buf); - - Copies SIZE bytes of data from BUF into the heap whose address - is ADDR in file F. BUF must be the _entire_ heap object. The - return value is the byte offset of the new data in the heap. - -void * -H5H_read (hdf5_file_t *f, off_t addr, off_t offset, size_t size, void *buf); - - Copies SIZE bytes of data from the heap whose address is ADDR - in file F into BUF and then returns the address of BUF. If - BUF is the null pointer then a new buffer will be malloc'd by - this function and its address is returned. - - Returns buffer address or null. - -const void * -H5H_peek (hdf5_file_t *f, off_t addr, off_t offset) - - A more efficient version of H5H_read that returns a pointer - directly into the cache; the data is not copied from the cache - to a buffer. The pointer is valid until the next call to an - H5AC function directly or indirectly. - - Returns a pointer or null. Do not free the pointer. - -void * -H5H_write (hdf5_file_t *f, off_t addr, off_t offset, size_t size, - const void *buf); - - Modifies (part of) an object in the heap at address ADDR of - file F by copying SIZE bytes from the beginning of BUF to the - file. OFFSET is the address withing the heap where the output - is to occur. - - This function can fail if the combination of OFFSET and SIZE - would write over a boundary between two heap objects. - -herr_t -H5H_remove (hdf5_file_t *f, off_t addr, off_t offset, size_t size); - - Removes an object or part of an object which begins at byte - OFFSET within a heap whose address is ADDR in file F. SIZE - bytes are returned to the free list. Removing the middle of - an object has the side effect that one object is now split - into two objects. - - Returns success or failure. - -=========================================== -Last Modified: 8 July 1998 (technical content) -Last Modified: 28 April 2000 (included in HDF5 Technical Notes) -HDF Help Desk: hdfhelp@ncsa.uiuc.edu - -</pre> - -</body> -</html> diff --git a/doc/html/TechNotes/IOPipe.html b/doc/html/TechNotes/IOPipe.html deleted file mode 100644 index 7c24e2c..0000000 --- a/doc/html/TechNotes/IOPipe.html +++ /dev/null @@ -1,114 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>The Raw Data I/O Pipeline</title> - </head> - - <body> - <h1>The Raw Data I/O Pipeline</h1> - - <p>The HDF5 raw data pipeline is a complicated beast that handles - all aspects of raw data storage and transfer of that data - between the file and the application. Data can be stored - contiguously (internal or external), in variable size external - segments, or regularly chunked; it can be sparse, extendible, - and/or compressible. Data transfers must be able to convert from - one data space to another, convert from one number type to - another, and perform partial I/O operations. Furthermore, - applications will expect their common usage of the pipeline to - perform well. - - <p>To accomplish these goals, the pipeline has been designed in a - modular way so no single subroutine is overly complicated and so - functionality can be inserted easily at the appropriate - locations in the pipeline. A general pipeline was developed and - then certain paths through the pipeline were optimized for - performance. - - <p>We describe only the file-to-memory side of the pipeline since - the memory-to-file side is a mirror image. We also assume that a - proper hyperslab of a simple data space is being read from the - file into a proper hyperslab of a simple data space in memory, - and that the data type is a compound type which may require - various number conversions on its members. - - <img alt="Figure 1" src="pipe1.gif"> - - <p>The diagrams should be read from the top down. The Line A - in the figure above shows that <code>H5Dread()</code> copies - data from a hyperslab of a file dataset to a hyperslab of an - application buffer by calling <code>H5D_read()</code>. And - <code>H5D_read()</code> calls, in a loop, - <code>H5S_simp_fgath()</code>, <code>H5T_conv_struct()</code>, - and <code>H5S_simp_mscat()</code>. A temporary buffer, TCONV, is - loaded with data points from the file, then data type conversion - is performed on the temporary buffer, and finally data points - are scattered out to application memory. Thus, data type - conversion is an in-place operation and data space conversion - consists of two steps. An additional temporary buffer, BKG, is - large enough to hold <em>N</em> instances of the destination - data type where <em>N</em> is the same number of data points - that can be held by the TCONV buffer (which is large enough to - hold either source or destination data points). - - <p>The application sets an upper limit for the size of the TCONV - buffer and optionally supplies a buffer. If no buffer is - supplied then one will be created by calling - <code>malloc()</code> when the pipeline is executed (when - necessary) and freed when the pipeline exits. The size of the - BKG buffer depends on the size of the TCONV buffer and if the - application supplies a BKG buffer it should be at least as large - as the TCONV buffer. The default size for these buffers is one - megabyte but the buffer might not be used to full capacity if - the buffer size is not an integer multiple of the source or - destination data point size (whichever is larger, but only - destination for the BKG buffer). - - - - <p>Occassionally the destination data points will be partially - initialized and the <code>H5Dread()</code> operation should not - clobber those values. For instance, the destination type might - be a struct with members <code>a</code> and <code>b</code> where - <code>a</code> is already initialized and we're reading - <code>b</code> from the file. An extra line, G, is added to the - pipeline to provide the type conversion functions with the - existing data. - - <img alt="Figure 2" src="pipe2.gif"> - - <p>It will most likely be quite common that no data type - conversion is necessary. In such cases a temporary buffer for - data type conversion is not needed and data space conversion - can happen in a single step. In fact, when the source and - destination data are both contiguous (they aren't in the - picture) the loop degenerates to a single iteration. - - - <img alt="Figure 3" src="pipe3.gif"> - - <p>So far we've looked only at internal contiguous storage, but by - replacing Line B in Figures 1 and 2 and Line A in Figure 3 with - Figure 4 the pipeline is able to handle regularly chunked - objects. Line B of Figure 4 is executed once for each chunk - which contains data to be read and the chunk address is found by - looking at a multi-dimensional key in a chunk B-tree which has - one entry per chunk. - - <img alt="Figure 4" src="pipe4.gif"> - - <p>If a single chunk is requested and the destination buffer is - the same size/shape as the chunk, then the CHUNK buffer is - bypassed and the destination buffer is used instead as shown in - Figure 5. - - <img alt="Figure 5" src="pipe5.gif"> - - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Tue Mar 17 11:13:35 EST 1998 --> -<!-- hhmts start --> -Last modified: Wed Mar 18 10:38:30 EST 1998 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/LibMaint.html b/doc/html/TechNotes/LibMaint.html deleted file mode 100644 index 718f085..0000000 --- a/doc/html/TechNotes/LibMaint.html +++ /dev/null @@ -1,128 +0,0 @@ -<html> -<body> - - -<h1>Information for HDF5 Maintainers</h1> - -<pre> - -* You can run make from any directory. However, running in a - subdirectory only knows how to build things in that directory and - below. However, all makefiles know when their target depends on - something outside the local directory tree: - - $ cd test - $ make - make: *** No rule to make target ../src/libhdf5.a - -* All Makefiles understand the following targets: - - all -- build locally. - install -- install libs, headers, progs. - uninstall -- remove installed files. - mostlyclean -- remove temp files (eg, *.o but not *.a). - clean -- mostlyclean plus libs and progs. - distclean -- all non-distributed files. - maintainer-clean -- all derived files but H5config.h.in and configure. - -* Most Makefiles also understand: - - TAGS -- build a tags table - dep, depend -- recalculate source dependencies - lib -- build just the libraries w/o programs - -* If you have personal preferences for which make, compiler, compiler - flags, preprocessor flags, etc., that you use and you don't want to - set environment variables, then use a site configuration file. - - When configure starts, it looks in the config directory for files - whose name is some combination of the CPU name, vendor, and - operating system in this order: - - CPU-VENDOR-OS - VENDOR-OS - CPU-VENDOR - OS - VENDOR - CPU - - The first file which is found is sourced and can therefore affect - the behavior of the rest of configure. See config/BlankForm for the - template. - -* If you use GNU make along with gcc the Makefile will contain targets - that automatically maintain a list of source interdependencies; you - seldom have to say `make clean'. I say `seldom' because if you - change how one `*.h' file includes other `*.h' files you'll have - to force an update. - - To force an update of all dependency information remove the - `.depend' file from each directory and type `make'. For - instance: - - $ cd $HDF5_HOME - $ find . -name .depend -exec rm {} \; - $ make - - If you're not using GNU make and gcc then dependencies come from - ".distdep" files in each directory. Those files are generated on - GNU systems and inserted into the Makefile's by running - config.status (which happens near the end of configure). - -* If you use GNU make along with gcc then the Perl script `trace' is - run just before dependencies are calculated to update any H5TRACE() - calls that might appear in the file. Otherwise, after changing the - type of a function (return type or argument types) one should run - `trace' manually on those source files (e.g., ../bin/trace *.c). - -* Object files stay in the directory and are added to the library as a - final step instead of placing the file in the library immediately - and removing it from the directory. The reason is three-fold: - - 1. Most versions of make don't allow `$(LIB)($(SRC:.c=.o))' - which makes it necessary to have two lists of files, one - that ends with `.c' and the other that has the library - name wrapped around each `.o' file. - - 2. Some versions of make/ar have problems with modification - times of archive members. - - 3. Adding object files immediately causes problems on SMP - machines where make is doing more than one thing at a - time. - -* When using GNU make on an SMP you can cause it to compile more than - one thing at a time. At the top of the source tree invoke make as - - $ make -j -l6 - - which causes make to fork as many children as possible as long as - the load average doesn't go above 6. In subdirectories one can say - - $ make -j2 - - which limits the number of children to two (this doesn't work at the - top level because the `-j2' is not passed to recursive makes). - -* To create a release tarball go to the top-level directory and run - ./bin/release. You can optionally supply one or more of the words - `tar', `gzip', `bzip2' or `compress' on the command line. The - result will be a (compressed) tar file(s) in the `releases' - directory. The README file is updated to contain the release date - and version number. - -* To create a tarball of all the files which are part of HDF5 go to - the top-level directory and type: - - tar cvf foo.tar `grep '^\.' MANIFEST |unexpand |cut -f1` - - -=========================================== -Last Modified: 15 October 1999 (technical content) -Last Modified: 28 April 2000 (included in HDF5 Technical Notes) -HDF Help Desk: hdfhelp@ncsa.uiuc.edu - -</pre> - -</body> -</html> diff --git a/doc/html/TechNotes/Makefile.am b/doc/html/TechNotes/Makefile.am deleted file mode 100644 index a0aca2d..0000000 --- a/doc/html/TechNotes/Makefile.am +++ /dev/null @@ -1,25 +0,0 @@ -# HDF5 Library Doc Makefile(.in) -# -# Copyright (C) 1997, 2002 -# National Center for Supercomputing Applications. -# All rights reserved. -# -## -## Makefile.am -## Run automake to generate a Makefile.in from this file. -# - -include $(top_srcdir)/config/commence-doc.am - -localdocdir = $(docdir)/hdf5/TechNotes - -# Public doc files (to be installed)... -localdoc_DATA=BigDataSmMach.html ChStudy_1000x1000.gif ChStudy_250x250.gif \ - ChStudy_499x499.gif ChStudy_5000x1000.gif ChStudy_500x500.gif \ - ChStudy_p1.gif ChunkingStudy.html CodeReview.html \ - ExternalFiles.html FreeLists.html H4-H5Compat.html HeapMgmt.html \ - IOPipe.html LibMaint.html MemoryMgmt.html MoveDStruct.html \ - NamingScheme.html ObjectHeader.html RawDStorage.html \ - SWControls.html SymbolTables.html ThreadSafeLibrary.html VFL.html \ - VFLfunc.html Version.html openmp-hdf5.c openmp-hdf5.html \ - pipe1.gif pipe2.gif pipe3.gif pipe4.gif pipe5.gif version.gif diff --git a/doc/html/TechNotes/Makefile.in b/doc/html/TechNotes/Makefile.in deleted file mode 100644 index 2dc4278..0000000 --- a/doc/html/TechNotes/Makefile.in +++ /dev/null @@ -1,494 +0,0 @@ -# Makefile.in generated by automake 1.9.5 from Makefile.am. -# @configure_input@ - -# Copyright (C) 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, -# 2003, 2004, 2005 Free Software Foundation, Inc. -# This Makefile.in is free software; the Free Software Foundation -# gives unlimited permission to copy and/or distribute it, -# with or without modifications, as long as this notice is preserved. - -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY, to the extent permitted by law; without -# even the implied warranty of MERCHANTABILITY or FITNESS FOR A -# PARTICULAR PURPOSE. - -@SET_MAKE@ - -# HDF5 Library Doc Makefile(.in) -# -# Copyright (C) 1997, 2002 -# National Center for Supercomputing Applications. -# All rights reserved. -# -# - -srcdir = @srcdir@ -top_srcdir = @top_srcdir@ -VPATH = @srcdir@ -pkgdatadir = $(datadir)/@PACKAGE@ -pkglibdir = $(libdir)/@PACKAGE@ -pkgincludedir = $(includedir)/@PACKAGE@ -top_builddir = ../../.. -am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd -INSTALL = @INSTALL@ -install_sh_DATA = $(install_sh) -c -m 644 -install_sh_PROGRAM = $(install_sh) -c -install_sh_SCRIPT = $(install_sh) -c -INSTALL_HEADER = $(INSTALL_DATA) -transform = $(program_transform_name) -NORMAL_INSTALL = : -PRE_INSTALL = : -POST_INSTALL = : -NORMAL_UNINSTALL = : -PRE_UNINSTALL = : -POST_UNINSTALL = : -build_triplet = @build@ -host_triplet = @host@ -DIST_COMMON = $(srcdir)/Makefile.am $(srcdir)/Makefile.in \ - $(top_srcdir)/config/commence-doc.am \ - $(top_srcdir)/config/commence.am -subdir = doc/html/TechNotes -ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 -am__aclocal_m4_deps = $(top_srcdir)/configure.in -am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \ - $(ACLOCAL_M4) -mkinstalldirs = $(SHELL) $(top_srcdir)/bin/mkinstalldirs -CONFIG_HEADER = $(top_builddir)/src/H5config.h -CONFIG_CLEAN_FILES = -SOURCES = -DIST_SOURCES = -am__vpath_adj_setup = srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; -am__vpath_adj = case $$p in \ - $(srcdir)/*) f=`echo "$$p" | sed "s|^$$srcdirstrip/||"`;; \ - *) f=$$p;; \ - esac; -am__strip_dir = `echo $$p | sed -e 's|^.*/||'`; -am__installdirs = "$(DESTDIR)$(localdocdir)" -localdocDATA_INSTALL = $(INSTALL_DATA) -DATA = $(localdoc_DATA) -DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) - -# Set the paths for AFS installs of autotools for Linux machines -# Ideally, these tools should never be needed during the build. -ACLOCAL = /afs/ncsa/projects/hdf/packages/automake_1.9.5/Linux_2.4/bin/aclocal -I /afs/ncsa/projects/hdf/packages/libtool_1.5.14/Linux_2.4/share/aclocal -ADD_PARALLEL_FILES = @ADD_PARALLEL_FILES@ -AMDEP_FALSE = @AMDEP_FALSE@ -AMDEP_TRUE = @AMDEP_TRUE@ -AMTAR = @AMTAR@ -AM_MAKEFLAGS = @AM_MAKEFLAGS@ -AR = @AR@ -AUTOCONF = /afs/ncsa/projects/hdf/packages/autoconf_2.59/Linux_2.4/bin/autoconf -AUTOHEADER = /afs/ncsa/projects/hdf/packages/autoconf_2.59/Linux_2.4/bin/autoheader -AUTOMAKE = /afs/ncsa/projects/hdf/packages/automake_1.9.5/Linux_2.4/bin/automake -AWK = @AWK@ -BUILD_CXX_CONDITIONAL_FALSE = @BUILD_CXX_CONDITIONAL_FALSE@ -BUILD_CXX_CONDITIONAL_TRUE = @BUILD_CXX_CONDITIONAL_TRUE@ -BUILD_FORTRAN_CONDITIONAL_FALSE = @BUILD_FORTRAN_CONDITIONAL_FALSE@ -BUILD_FORTRAN_CONDITIONAL_TRUE = @BUILD_FORTRAN_CONDITIONAL_TRUE@ -BUILD_HDF5_HL_CONDITIONAL_FALSE = @BUILD_HDF5_HL_CONDITIONAL_FALSE@ -BUILD_HDF5_HL_CONDITIONAL_TRUE = @BUILD_HDF5_HL_CONDITIONAL_TRUE@ -BUILD_PABLO_CONDITIONAL_FALSE = @BUILD_PABLO_CONDITIONAL_FALSE@ -BUILD_PABLO_CONDITIONAL_TRUE = @BUILD_PABLO_CONDITIONAL_TRUE@ -BUILD_PARALLEL_CONDITIONAL_FALSE = @BUILD_PARALLEL_CONDITIONAL_FALSE@ -BUILD_PARALLEL_CONDITIONAL_TRUE = @BUILD_PARALLEL_CONDITIONAL_TRUE@ -BUILD_PDB2HDF = @BUILD_PDB2HDF@ -BUILD_PDB2HDF_CONDITIONAL_FALSE = @BUILD_PDB2HDF_CONDITIONAL_FALSE@ -BUILD_PDB2HDF_CONDITIONAL_TRUE = @BUILD_PDB2HDF_CONDITIONAL_TRUE@ -BYTESEX = @BYTESEX@ -CC = @CC@ -CCDEPMODE = @CCDEPMODE@ -CC_VERSION = @CC_VERSION@ -CFLAGS = @CFLAGS@ -CONFIG_DATE = @CONFIG_DATE@ -CONFIG_MODE = @CONFIG_MODE@ -CONFIG_USER = @CONFIG_USER@ -CPP = @CPP@ -CPPFLAGS = @CPPFLAGS@ -CXX = @CXX@ -CXXCPP = @CXXCPP@ -CXXDEPMODE = @CXXDEPMODE@ -CXXFLAGS = @CXXFLAGS@ -CYGPATH_W = @CYGPATH_W@ -DEBUG_PKG = @DEBUG_PKG@ -DEFS = @DEFS@ -DEPDIR = @DEPDIR@ -DYNAMIC_DIRS = @DYNAMIC_DIRS@ -ECHO = @ECHO@ -ECHO_C = @ECHO_C@ -ECHO_N = @ECHO_N@ -ECHO_T = @ECHO_T@ -EGREP = @EGREP@ -EXEEXT = @EXEEXT@ -F77 = @F77@ - -# Make sure that these variables are exported to the Makefiles -F9XMODEXT = @F9XMODEXT@ -F9XMODFLAG = @F9XMODFLAG@ -F9XSUFFIXFLAG = @F9XSUFFIXFLAG@ -FC = @FC@ -FCFLAGS = @FCFLAGS@ -FCLIBS = @FCLIBS@ -FFLAGS = @FFLAGS@ -FILTERS = @FILTERS@ -FSEARCH_DIRS = @FSEARCH_DIRS@ -H5_VERSION = @H5_VERSION@ -HADDR_T = @HADDR_T@ -HDF5_INTERFACES = @HDF5_INTERFACES@ -HID_T = @HID_T@ -HL = @HL@ -HL_FOR = @HL_FOR@ -HSIZET = @HSIZET@ -HSIZE_T = @HSIZE_T@ -HSSIZE_T = @HSSIZE_T@ -INSTALL_DATA = @INSTALL_DATA@ -INSTALL_PROGRAM = @INSTALL_PROGRAM@ -INSTALL_SCRIPT = @INSTALL_SCRIPT@ -INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@ -INSTRUMENT_LIBRARY = @INSTRUMENT_LIBRARY@ -LDFLAGS = @LDFLAGS@ -LIBOBJS = @LIBOBJS@ -LIBS = @LIBS@ -LIBTOOL = @LIBTOOL@ -LN_S = @LN_S@ -LTLIBOBJS = @LTLIBOBJS@ -LT_STATIC_EXEC = @LT_STATIC_EXEC@ -MAINT = @MAINT@ -MAINTAINER_MODE_FALSE = @MAINTAINER_MODE_FALSE@ -MAINTAINER_MODE_TRUE = @MAINTAINER_MODE_TRUE@ -MAKEINFO = @MAKEINFO@ -MPE = @MPE@ -OBJECT_NAMELEN_DEFAULT_F = @OBJECT_NAMELEN_DEFAULT_F@ -OBJEXT = @OBJEXT@ -PACKAGE = @PACKAGE@ -PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@ -PACKAGE_NAME = @PACKAGE_NAME@ -PACKAGE_STRING = @PACKAGE_STRING@ -PACKAGE_TARNAME = @PACKAGE_TARNAME@ -PACKAGE_VERSION = @PACKAGE_VERSION@ -PARALLEL = @PARALLEL@ -PATH_SEPARATOR = @PATH_SEPARATOR@ -PERL = @PERL@ -PTHREAD = @PTHREAD@ -RANLIB = @RANLIB@ -ROOT = @ROOT@ -RUNPARALLEL = @RUNPARALLEL@ -RUNSERIAL = @RUNSERIAL@ -R_INTEGER = @R_INTEGER@ -R_LARGE = @R_LARGE@ -SEARCH = @SEARCH@ -SETX = @SETX@ -SET_MAKE = @SET_MAKE@ - -# Hardcode SHELL to be /bin/sh. Most machines have this shell, and -# on at least one machine configure fails to detect its existence (janus). -# Also, when HDF5 is configured on one machine but run on another, -# configure's automatic SHELL detection may not work on the build machine. -SHELL = /bin/sh -SIZE_T = @SIZE_T@ -STATIC_SHARED = @STATIC_SHARED@ -STRIP = @STRIP@ -TESTPARALLEL = @TESTPARALLEL@ -TRACE_API = @TRACE_API@ -USE_FILTER_DEFLATE = @USE_FILTER_DEFLATE@ -USE_FILTER_FLETCHER32 = @USE_FILTER_FLETCHER32@ -USE_FILTER_NBIT = @USE_FILTER_NBIT@ -USE_FILTER_SCALEOFFSET = @USE_FILTER_SCALEOFFSET@ -USE_FILTER_SHUFFLE = @USE_FILTER_SHUFFLE@ -USE_FILTER_SZIP = @USE_FILTER_SZIP@ -VERSION = @VERSION@ -ac_ct_AR = @ac_ct_AR@ -ac_ct_CC = @ac_ct_CC@ -ac_ct_CXX = @ac_ct_CXX@ -ac_ct_F77 = @ac_ct_F77@ -ac_ct_FC = @ac_ct_FC@ -ac_ct_RANLIB = @ac_ct_RANLIB@ -ac_ct_STRIP = @ac_ct_STRIP@ -am__fastdepCC_FALSE = @am__fastdepCC_FALSE@ -am__fastdepCC_TRUE = @am__fastdepCC_TRUE@ -am__fastdepCXX_FALSE = @am__fastdepCXX_FALSE@ -am__fastdepCXX_TRUE = @am__fastdepCXX_TRUE@ -am__include = @am__include@ -am__leading_dot = @am__leading_dot@ -am__quote = @am__quote@ -am__tar = @am__tar@ -am__untar = @am__untar@ -bindir = @bindir@ -build = @build@ -build_alias = @build_alias@ -build_cpu = @build_cpu@ -build_os = @build_os@ -build_vendor = @build_vendor@ -datadir = @datadir@ -exec_prefix = @exec_prefix@ -host = @host@ -host_alias = @host_alias@ -host_cpu = @host_cpu@ -host_os = @host_os@ -host_vendor = @host_vendor@ - -# Install directories that automake doesn't know about -includedir = $(exec_prefix)/include -infodir = @infodir@ -install_sh = @install_sh@ -libdir = @libdir@ -libexecdir = @libexecdir@ -localstatedir = @localstatedir@ -mandir = @mandir@ -mkdir_p = @mkdir_p@ -oldincludedir = @oldincludedir@ -prefix = @prefix@ -program_transform_name = @program_transform_name@ -sbindir = @sbindir@ -sharedstatedir = @sharedstatedir@ -sysconfdir = @sysconfdir@ -target_alias = @target_alias@ - -# Shell commands used in Makefiles -RM = rm -f -CP = cp - -# Some machines need a command to run executables; this is that command -# so that our tests will run. -# We use RUNTESTS instead of RUNSERIAL directly because it may be that -# some tests need to be run with a different command. Older versions -# of the makefiles used the command -# $(LIBTOOL) --mode=execute -# in some directories, for instance. -RUNTESTS = $(RUNSERIAL) - -# Libraries to link to while building -LIBHDF5 = $(top_builddir)/src/libhdf5.la -LIBH5TEST = $(top_builddir)/test/libh5test.la -LIBH5F = $(top_builddir)/fortran/src/libhdf5_fortran.la -LIBH5FTEST = $(top_builddir)/fortran/test/libh5test_fortran.la -LIBH5CPP = $(top_builddir)/c++/src/libhdf5_cpp.la -LIBH5TOOLS = $(top_builddir)/tools/lib/libh5tools.la -LIBH5_HL = $(top_builddir)/hl/src/libhdf5_hl.la -LIBH5F_HL = $(top_builddir)/hl/fortran/src/libhdf5hl_fortran.la -LIBH5CPP_HL = $(top_builddir)/hl/c++/src/libhdf5_hl_cpp.la -docdir = $(exec_prefix)/doc - -# Scripts used to build examples -H5CC = $(bindir)/h5cc -H5CC_PP = $(bindir)/h5pcc -H5FC = $(bindir)/h5fc -H5FC_PP = $(bindir)/h5pfc - -# .chkexe and .chksh files are used to mark tests that have run successfully. -MOSTLYCLEANFILES = *.chkexe *.chksh -localdocdir = $(docdir)/hdf5/TechNotes - -# Public doc files (to be installed)... -localdoc_DATA = BigDataSmMach.html ChStudy_1000x1000.gif ChStudy_250x250.gif \ - ChStudy_499x499.gif ChStudy_5000x1000.gif ChStudy_500x500.gif \ - ChStudy_p1.gif ChunkingStudy.html CodeReview.html \ - ExternalFiles.html FreeLists.html H4-H5Compat.html HeapMgmt.html \ - IOPipe.html LibMaint.html MemoryMgmt.html MoveDStruct.html \ - NamingScheme.html ObjectHeader.html RawDStorage.html \ - SWControls.html SymbolTables.html ThreadSafeLibrary.html VFL.html \ - VFLfunc.html Version.html openmp-hdf5.c openmp-hdf5.html \ - pipe1.gif pipe2.gif pipe3.gif pipe4.gif pipe5.gif version.gif - -all: all-am - -.SUFFIXES: -$(srcdir)/Makefile.in: @MAINTAINER_MODE_TRUE@ $(srcdir)/Makefile.am $(top_srcdir)/config/commence-doc.am $(top_srcdir)/config/commence.am $(am__configure_deps) - @for dep in $?; do \ - case '$(am__configure_deps)' in \ - *$$dep*) \ - cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh \ - && exit 0; \ - exit 1;; \ - esac; \ - done; \ - echo ' cd $(top_srcdir) && $(AUTOMAKE) --foreign doc/html/TechNotes/Makefile'; \ - cd $(top_srcdir) && \ - $(AUTOMAKE) --foreign doc/html/TechNotes/Makefile -.PRECIOUS: Makefile -Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status - @case '$?' in \ - *config.status*) \ - cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh;; \ - *) \ - echo ' cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__depfiles_maybe)'; \ - cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__depfiles_maybe);; \ - esac; - -$(top_builddir)/config.status: $(top_srcdir)/configure $(CONFIG_STATUS_DEPENDENCIES) - cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh - -$(top_srcdir)/configure: @MAINTAINER_MODE_TRUE@ $(am__configure_deps) - cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh -$(ACLOCAL_M4): @MAINTAINER_MODE_TRUE@ $(am__aclocal_m4_deps) - cd $(top_builddir) && $(MAKE) $(AM_MAKEFLAGS) am--refresh - -mostlyclean-libtool: - -rm -f *.lo - -clean-libtool: - -rm -rf .libs _libs - -distclean-libtool: - -rm -f libtool -uninstall-info-am: -install-localdocDATA: $(localdoc_DATA) - @$(NORMAL_INSTALL) - test -z "$(localdocdir)" || $(mkdir_p) "$(DESTDIR)$(localdocdir)" - @list='$(localdoc_DATA)'; for p in $$list; do \ - if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \ - f=$(am__strip_dir) \ - echo " $(localdocDATA_INSTALL) '$$d$$p' '$(DESTDIR)$(localdocdir)/$$f'"; \ - $(localdocDATA_INSTALL) "$$d$$p" "$(DESTDIR)$(localdocdir)/$$f"; \ - done - -uninstall-localdocDATA: - @$(NORMAL_UNINSTALL) - @list='$(localdoc_DATA)'; for p in $$list; do \ - f=$(am__strip_dir) \ - echo " rm -f '$(DESTDIR)$(localdocdir)/$$f'"; \ - rm -f "$(DESTDIR)$(localdocdir)/$$f"; \ - done -tags: TAGS -TAGS: - -ctags: CTAGS -CTAGS: - - -distdir: $(DISTFILES) - $(mkdir_p) $(distdir)/../../../config - @srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; \ - topsrcdirstrip=`echo "$(top_srcdir)" | sed 's|.|.|g'`; \ - list='$(DISTFILES)'; for file in $$list; do \ - case $$file in \ - $(srcdir)/*) file=`echo "$$file" | sed "s|^$$srcdirstrip/||"`;; \ - $(top_srcdir)/*) file=`echo "$$file" | sed "s|^$$topsrcdirstrip/|$(top_builddir)/|"`;; \ - esac; \ - if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \ - dir=`echo "$$file" | sed -e 's,/[^/]*$$,,'`; \ - if test "$$dir" != "$$file" && test "$$dir" != "."; then \ - dir="/$$dir"; \ - $(mkdir_p) "$(distdir)$$dir"; \ - else \ - dir=''; \ - fi; \ - if test -d $$d/$$file; then \ - if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \ - cp -pR $(srcdir)/$$file $(distdir)$$dir || exit 1; \ - fi; \ - cp -pR $$d/$$file $(distdir)$$dir || exit 1; \ - else \ - test -f $(distdir)/$$file \ - || cp -p $$d/$$file $(distdir)/$$file \ - || exit 1; \ - fi; \ - done -check-am: all-am -check: check-am -all-am: Makefile $(DATA) -installdirs: - for dir in "$(DESTDIR)$(localdocdir)"; do \ - test -z "$$dir" || $(mkdir_p) "$$dir"; \ - done -install: install-am -install-exec: install-exec-am -install-data: install-data-am -uninstall: uninstall-am - -install-am: all-am - @$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am - -installcheck: installcheck-am -install-strip: - $(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \ - install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \ - `test -z '$(STRIP)' || \ - echo "INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'"` install -mostlyclean-generic: - -test -z "$(MOSTLYCLEANFILES)" || rm -f $(MOSTLYCLEANFILES) - -clean-generic: - -distclean-generic: - -test -z "$(CONFIG_CLEAN_FILES)" || rm -f $(CONFIG_CLEAN_FILES) - -maintainer-clean-generic: - @echo "This command is intended for maintainers to use" - @echo "it deletes files that may require special tools to rebuild." -clean: clean-am - -clean-am: clean-generic clean-libtool mostlyclean-am - -distclean: distclean-am - -rm -f Makefile -distclean-am: clean-am distclean-generic distclean-libtool - -dvi: dvi-am - -dvi-am: - -html: html-am - -info: info-am - -info-am: - -install-data-am: install-localdocDATA - -install-exec-am: - -install-info: install-info-am - -install-man: - -installcheck-am: - -maintainer-clean: maintainer-clean-am - -rm -f Makefile -maintainer-clean-am: distclean-am maintainer-clean-generic - -mostlyclean: mostlyclean-am - -mostlyclean-am: mostlyclean-generic mostlyclean-libtool - -pdf: pdf-am - -pdf-am: - -ps: ps-am - -ps-am: - -uninstall-am: uninstall-info-am uninstall-localdocDATA - -.PHONY: all all-am check check-am clean clean-generic clean-libtool \ - distclean distclean-generic distclean-libtool distdir dvi \ - dvi-am html html-am info info-am install install-am \ - install-data install-data-am install-exec install-exec-am \ - install-info install-info-am install-localdocDATA install-man \ - install-strip installcheck installcheck-am installdirs \ - maintainer-clean maintainer-clean-generic mostlyclean \ - mostlyclean-generic mostlyclean-libtool pdf pdf-am ps ps-am \ - uninstall uninstall-am uninstall-info-am \ - uninstall-localdocDATA - - -# Ignore most rules -lib progs check test _test check-p check-s: - @echo "Nothing to be done" - -tests dep depend: - @@SETX@; for d in X $(SUBDIRS); do \ - if test $$d != X; then \ - (cd $$d && $(MAKE) $(AM_MAKEFLAGS) $@) || exit 1; \ - fi; - done - -# In docs directory, install-doc is the same as install -install-doc install-all: - $(MAKE) $(AM_MAKEFLAGS) install -uninstall-doc uninstall-all: - $(MAKE) $(AM_MAKEFLAGS) uninstall -# Tell versions [3.59,3.63) of GNU make to not export all variables. -# Otherwise a system limit (for SysV at least) may be exceeded. -.NOEXPORT: diff --git a/doc/html/TechNotes/MemoryMgmt.html b/doc/html/TechNotes/MemoryMgmt.html deleted file mode 100644 index 93782b5..0000000 --- a/doc/html/TechNotes/MemoryMgmt.html +++ /dev/null @@ -1,510 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Memory Management in HDF5</title> - </head> - - <body> - <h1>Memory Management in HDF5</h1> - - <!-- ---------------------------------------------------------------- --> - <h2>Is a Memory Manager Necessary?</h2> - - <p>Some form of memory management may be necessary in HDF5 when - the various deletion operators are implemented so that the - file memory is not permanently orphaned. However, since an - HDF5 file was designed with persistent data in mind, the - importance of a memory manager is questionable. - - <p>On the other hand, when certain meta data containers (file glue) - grow, they may need to be relocated in order to keep the - container contiguous. - - <blockquote> - <b>Example:</b> An object header consists of up to two - chunks of contiguous memory. The first chunk is a fixed - size at a fixed location when the header link count is - greater than one. Thus, inserting additional items into an - object header may require the second chunk to expand. When - this occurs, the second chunk may need to move to another - location in the file, freeing the file memory which that - chunk originally occupied. - </blockquote> - - <p>The relocation of meta data containers could potentially - orphan a significant amount of file memory if the application - has made poor estimates for preallocation sizes. - - <!-- ---------------------------------------------------------------- --> - <h2>Levels of Memory Management</h2> - - <p>Memory management by the library can be independent of memory - management support by the file format. The file format can - support no memory management, some memory management, or full - memory management. Similarly with the library. - - <h3>Support in the Library</h3> - - <dl> - <dt><b>No Support: I</b> - <dd>When memory is deallocated it simply becomes unreferenced - (orphaned) in the file. Memory allocation requests are - satisfied by extending the file. - - <dd>A separate off-line utility can be used to detect the - unreferenced bytes of a file and "bubble" them up to the end - of the file and then truncate the file. - - <dt><b>Some Support: II</b> - <dd>The library could support partial memory management all - the time, or full memory management some of the time. - Orphaning free blocks instead of adding them to a free list - should not affect the file integrity, nor should fulfilling - new requests by extending the file instead of using the free - list. - - <dt><b>Full Support: III</b> - <dd>The library supports space-efficient memory management by - always fulfilling allocation requests from the free list when - possible, and by coalescing adjacent free blocks into a - single larger free block. - </dl> - - <h3>Support in the File Format</h3> - - <dl> - <dt><b>No Support: A</b> - <dd>The file format does not support memory management; any - unreferenced block in the file is assumed to be free. If - the library supports full memory management then it will - have to traverse the entire file to determine which blocks - are unreferenced. - - <dt><b>Some Support: B</b> - <dd>Assuming that unreferenced blocks are free can be - dangerous in a situation where the file is not consistent. - For instance, if a directory tree becomes detached from the - main directory hierarchy, then the detached directory and - everything that is referenced only through the detached - directory become unreferenced. File repair utilities will - be unable to determine which unreferenced blocks need to be - linked back into the file hierarchy. - - <dd>Therefore, it might be useful to keep an unsorted, - doubly-linked list of free blocks in the file. The library - can add and remove blocks from the list in constant time, - and can generate its own internal free-block data structure - in time proportional to the number of free blocks instead of - the size of the file. Additionally, a library can use a - subset of the free blocks, an alternative which is not - feasible if the file format doesn't support any form of - memory management. - - <dt><b>Full Support: C</b> - <dd>The file format can mirror library data structures for - space-efficient memory management. The free blocks are - linked in unsorted, doubly-linked lists with one list per - free block size. The heads of the lists are pointed to by a - B-tree whose nodes are sorted by free block size. At the - same time, all free blocks are the leaf nodes of another - B-tree sorted by starting and ending address. When the - trees are used in combination we can deallocate and allocate - memory in O(log <em>N</em>) time where <em>N</em> is the - number of free blocks. - </dl> - - <h3>Combinations of Library and File Format Support</h3> - - <p>We now evaluate each combination of library support with file - support: - - <dl> - <dt><b>I-A</b> - <dd>If neither the library nor the file support memory - management, then each allocation request will come from the - end of the file and each deallocation request is a no-op - that simply leaves the free block unreferenced. - - <ul> - <li>Advantages - <ul> - <li>No file overhead for allocation or deallocation. - <li>No library overhead for allocation or - deallocation. - <li>No file traversal required at time of open. - <li>No data needs to be written back to the file when - it's closed. - <li>Trivial to implement (already implemented). - </ul> - - <li>Disadvantages - <ul> - <li>Inefficient use of file space. - <li>A file repair utility must reclaim lost file space. - <li>Difficulties for file repair utilities. (Is an - unreferenced block a free block or orphaned data?) - </ul> - </ul> - - <dt><b>II-A</b> - <dd>In order for the library to support memory management, it - will be required to build the internal free block - representation by traversing the entire file looking for - unreferenced blocks. - - <ul> - <li>Advantages - <ul> - <li>No file overhead for allocation or deallocation. - <li>Variable amount of library overhead for allocation - and deallocation depending on how much work the - library wants to do. - <li>No data needs to be written back to the file when - it's closed. - <li>Might use file space efficiently. - </ul> - <li>Disadvantages - <ul> - <li>Might use file space inefficiently. - <li>File traversal required at time of open. - <li>A file repair utility must reclaim lost file space. - <li>Difficulties for file repair utilities. - <li>Sharing of the free list between processes falls - outside the HDF5 file format documentation. - </ul> - </ul> - - <dt><b>III-A</b> - <dd>In order for the library to support full memory - management, it will be required to build the internal free - block representation by traversing the entire file looking - for unreferenced blocks. - - <ul> - <li>Advantages - <ul> - <li>No file overhead for allocation or deallocation. - <li>Efficient use of file space. - <li>No data needs to be written back to the file when - it's closed. - </ul> - <li>Disadvantages - <ul> - <li>Moderate amount of library overhead for allocation - and deallocation. - <li>File traversal required at time of open. - <li>A file repair utility must reclaim lost file space. - <li>Difficulties for file repair utilities. - <li>Sharing of the free list between processes falls - outside the HDF5 file format documentation. - </ul> - </ul> - - <dt><b>I-B</b> - <dd>If the library doesn't support memory management but the - file format supports some level of management, then a file - repair utility will have to be run occasionally to reclaim - unreferenced blocks. - - <ul> - <li>Advantages - <ul> - <li>No file overhead for allocation or deallocation. - <li>No library overhead for allocation or - deallocation. - <li>No file traversal required at time of open. - <li>No data needs to be written back to the file when - it's closed. - </ul> - <li>Disadvantages - <ul> - <li>A file repair utility must reclaim lost file space. - <li>Difficulties for file repair utilities. - </ul> - </ul> - - <dt><b>II-B</b> - <dd>Both the library and the file format support some level - of memory management. - - <ul> - <li>Advantages - <ul> - <li>Constant file overhead per allocation or - deallocation. - <li>Variable library overhead per allocation or - deallocation depending on how much work the library - wants to do. - <li>Traversal at file open time is on the order of the - free list size instead of the file size. - <li>The library has the option of reading only part of - the free list. - <li>No data needs to be written at file close time if - it has been amortized into the cost of allocation - and deallocation. - <li>File repair utilties don't have to be run to - reclaim memory. - <li>File repair utilities can detect whether an - unreferenced block is a free block or orphaned data. - <li>Sharing of the free list between processes might - be easier. - <li>Possible efficient use of file space. - </ul> - <li>Disadvantages - <ul> - <li>Possible inefficient use of file space. - </ul> - </ul> - - <dt><b>III-B</b> - <dd>The library provides space-efficient memory management but - the file format only supports an unsorted list of free - blocks. - - <ul> - <li>Advantages - <ul> - <li>Constant time file overhead per allocation or - deallocation. - <li>No data needs to be written at file close time if - it has been amortized into the cost of allocation - and deallocation. - <li>File repair utilities don't have to be run to - reclaim memory. - <li>File repair utilities can detect whether an - unreferenced block is a free block or orphaned data. - <li>Sharing of the free list between processes might - be easier. - <li>Efficient use of file space. - </ul> - <li>Disadvantages - <ul> - <li>O(log <em>N</em>) library overhead per allocation or - deallocation where <em>N</em> is the total number of - free blocks. - <li>O(<em>N</em>) time to open a file since the entire - free list must be read to construct the in-core - trees used by the library. - <li>Library is more complicated. - </ul> - </ul> - - <dt><b>I-C</b> - <dd>This has the same advantages and disadvantages as I-C with - the added disadvantage that the file format is much more - complicated. - - <dt><b>II-C</b> - <dd>If the library only provides partial memory management but - the file requires full memory management, then this method - degenerates to the same as II-A with the added disadvantage - that the file format is much more complicated. - - <dt><b>III-C</b> - <dd>The library and file format both provide complete data - structures for space-efficient memory management. - - <ul> - <li>Advantages - <ul> - <li>Files can be opened in constant time since the - free list is read on demand and amortised into the - allocation and deallocation requests. - <li>No data needs to be written back to the file when - it's closed. - <li>File repair utilities don't have to be run to - reclaim memory. - <li>File repair utilities can detect whether an - unreferenced block is a free block or orphaned data. - <li>Sharing the free list between processes is easy. - <li>Efficient use of file space. - </ul> - <li>Disadvantages - <ul> - <li>O(log <em>N</em>) file allocation and deallocation - cost where <em>N</em> is the total number of free - blocks. - <li>O(log <em>N</em>) library allocation and - deallocation cost. - <li>Much more complicated file format. - <li>More complicated library. - </ul> - </ul> - - </dl> - - <!-- ---------------------------------------------------------------- --> - <h2>The Algorithm for II-B</h2> - - <p>The file contains an unsorted, doubly-linked list of free - blocks. The address of the head of the list appears in the - boot block. Each free block contains the following fields: - - <center> - <table border cellpadding=4 width="60%"> - <tr align=center> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - - <tr align=center> - <th colspan=4>Free Block Signature</th> - - <tr align=center> - <th colspan=4>Total Free Block Size</th> - - <tr align=center> - <th colspan=4>Address of Left Sibling</th> - - <tr align=center> - <th colspan=4>Address of Right Sibling</th> - - <tr align=center> - <th colspan=4><br><br>Remainder of Free Block<br><br><br></th> - </table> - </center> - - <p>The library reads as much of the free list as convenient when - convenient and pushes those entries onto stacks. This can - occur when a file is opened or any time during the life of the - file. There is one stack for each free block size and the - stacks are sorted by size in a balanced tree in memory. - - <p>Deallocation involves finding the correct stack or creating - a new one (an O(log <em>K</em>) operation where <em>K</em> is - the number of stacks), pushing the free block info onto the - stack (a constant-time operation), and inserting the free - block into the file free block list (a constant-time operation - which doesn't necessarily involve any I/O since the free blocks - can be cached like other objects). No attempt is made to - coalesce adjacent free blocks into larger blocks. - - <p>Allocation involves finding the correct stack (an O(log - <em>K</em>) operation), removing the top item from the stack - (a constant-time operation), and removing the block from the - file free block list (a constant-time operation). If there is - no free block of the requested size or larger, then the file - is extended. - - <p>To provide sharability of the free list between processes, - the last step of an allocation will check for the free block - signature and if it doesn't find one will repeat the process. - Alternatively, a process can temporarily remove free blocks - from the file and hold them in it's own private pool. - - <p>To summarize... - <dl> - <dt>File opening - <dd>O(<em>N</em>) amortized over the time the file is open, - where <em>N</em> is the number of free blocks. The library - can still function without reading any of the file free - block list. - - <dt>Deallocation - <dd>O(log <em>K</em>) where <em>K</em> is the number of unique - sizes of free blocks. File access is constant. - - <dt>Allocation - <dd>O(log <em>K</em>). File access is constant. - - <dt>File closing - <dd>O(1) even if the library temporarily removes free - blocks from the file to hold them in a private pool since - the pool can still be a linked list on disk. - </dl> - - <!-- ---------------------------------------------------------------- --> - <h2>The Algorithm for III-C</h2> - - <p>The HDF5 file format supports a general B-tree mechanism - for storing data with keys. If we use a B-tree to represent - all parts of the file that are free and the B-tree is indexed - so that a free file chunk can be found if we know the starting - or ending address, then we can efficiently determine whether a - free chunk begins or ends at the specified address. Call this - the <em>Address B-Tree</em>. - - <p>If a second B-tree points to a set of stacks where the - members of a particular stack are all free chunks of the same - size, and the tree is indexed by chunk size, then we can - efficiently find the best-fit chunk size for a memory request. - Call this the <em>Size B-Tree</em>. - - <p>All free blocks of a particular size can be linked together - with an unsorted, doubly-linked, circular list and the left - and right sibling addresses can be stored within the free - chunk, allowing us to remove or insert items from the list in - constant time. - - <p>Deallocation of a block fo file memory consists of: - - <ol type="I"> - <li>Add the new free block whose address is <em>ADDR</em> to the - address B-tree. - - <ol type="A"> - <li>If the address B-tree contains an entry for a free - block that ends at <em>ADDR</em>-1 then remove that - block from the B-tree and from the linked list (if the - block was the first on the list then the size B-tree - must be updated). Adjust the size and address of the - block being freed to include the block just removed from - the free list. The time required to search for and - possibly remove the left block is O(log <em>N</em>) - where <em>N</em> is the number of free blocks. - - <li>If the address B-tree contains an entry for the free - block that begins at <em>ADDR</em>+<em>LENGTH</em> then - remove that block from the B-tree and from the linked - list (if the block was the first on the list then the - size B-tree must be updated). Adjust the size of the - block being freed to include the block just removed from - the free list. The time required to search for and - possibly remove the right block is O(log <em>N</em>). - - <li>Add the new (adjusted) block to the address B-tree. - The time for this operation is O(log <em>N</em>). - </ol> - - <li>Add the new block to the size B-tree and linked list. - - <ol type="A"> - <li>If the size B-tree has an entry for this particular - size, then add the chunk to the tail of the list. This - is an O(log <em>K</em>) operation where <em>K</em> is - the number of unique free block sizes. - - <li>Otherwise make a new entry in the B-tree for chunks of - this size. This is also O(log <em>K</em>). - </ol> - </ol> - - <p>Allocation is similar to deallocation. - - <p>To summarize... - - <dl> - <dt>File opening - <dd>O(1) - - <dt>Deallocation - <dd>O(log <em>N</em>) where <em>N</em> is the total number of - free blocks. File access time is O(log <em>N</em>). - - <dt>Allocation - <dd>O(log <em>N</em>). File access time is O(log <em>N</em>). - - <dt>File closing - <dd>O(1). - </dl> - - - <hr> - <address><a href="mailto:matzke@llnl.gov">Robb Matzke</a></address> -<!-- Created: Thu Jul 24 15:16:40 PDT 1997 --> -<!-- hhmts start --> -Last modified: Thu Jul 31 14:41:01 EST -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/MoveDStruct.html b/doc/html/TechNotes/MoveDStruct.html deleted file mode 100644 index 4576bd2..0000000 --- a/doc/html/TechNotes/MoveDStruct.html +++ /dev/null @@ -1,66 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Relocating a File Data Structure</title> - </head> - - <body> - <h1>Relocating a File Data Structure</h1> - - <p>Since file data structures can be cached in memory by the H5AC - package it becomes problematic to move such a data structure in - the file. One cannot just copy a portion of the file from one - location to another because: - - <ol> - <li>the file might not contain the latest information, and</li> - <li>the H5AC package might not realize that the object's - address has changed and attempt to write the object to disk - at the old address.</li> - </ol> - - <p>Here's a correct method to move data from one location to - another. The example code assumes that one is moving a B-link - tree node from <code>old_addr</code> to <code>new_addr</code>. - - <ol> - <li>Make sure the disk is up-to-date with respect to the - cache. There is no need to remove the item from the cache, - hence the final argument to <code>H5AC_flush</code> is - <code>FALSE</code>. - <br><br> - <code> - H5AC_flush (f, H5AC_BT, old_addr, FALSE);<br> - </code> - <br> - </li> - - <li>Read the data from the old address and write it to the new - address. - <br><br> - <code> - H5F_block_read (f, old_addr, size, buf);<br> - H5F_block_write (f, new_addr, size, buf);<br> - </code> - <br> - </li> - - <li>Notify the cache that the address of the object changed. - <br><br> - <code> - H5AC_rename (f, H5AC_BT, old_addr, new_addr);<br> - </code> - <br> - </li> - </ol> - - - - <hr> - <address><a href="mailto:robb@maya.nuance.com">Robb Matzke</a></address> -<!-- Created: Mon Jul 14 15:09:06 EST 1997 --> -<!-- hhmts start --> -Last modified: Mon Jul 14 15:38:29 EST -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/NamingScheme.html b/doc/html/TechNotes/NamingScheme.html deleted file mode 100644 index dbf55bf..0000000 --- a/doc/html/TechNotes/NamingScheme.html +++ /dev/null @@ -1,300 +0,0 @@ -<HTML> -<HEAD><TITLE> - HDF5 Naming Scheme - </TITLE> </HEAD> - -<BODY bgcolor="#ffffff"> - - -<H1> -<FONT color="#c80028" - <I> <B> <CENTER> HDF5 Naming Scheme for </CENTER> </B> </I> </H1> -</FONT> -<P> -<UL> - -<LI> <A HREF = "#01"><I> FILES </I> </A> -<LI> <A HREF = "#02"><I> PACKAGES </I> </A> -<LI> <A HREF = "#03"><I> PUBLIC vs PRIVATE </I> </A> -<LI> <A HREF = "#04"><I> INTEGRAL TYPES </I> </A> -<LI> <A HREF = "#05"><I> OTHER TYPES </I> </A> -<LI> <A HREF = "#06"><I> GLOBAL VARIABLES </I> </A> -<LI> <A HREF = "#07"><I> MACROS, PREPROCESSOR CONSTANTS, AND ENUM MEMEBERs </I> </A> - -</UL> -<P> -<center> - Authors: <A HREF = "mailto:koziol@ncsa.uiuc.edu"> - <I>Quincey Koziol</I> </A> and - <A HREF = "mailto:matzke@llnl.gov"> - <I> Robb Matzke </I> </A> - -</center> -<UL> - -<FONT color="#c80028" -<LI> <A NAME="01"> <B> <I> FILES </I> </B> </A> -</FONT> - -<UL> - - <LI> Source files are named according to the package they contain (see - below). All files will begin with `H5' so we can stuff our - object files into someone else's library and not worry about file - name conflicts. - <P>For Example: -<i><b> -<dd> H5.c -- "Generic" library functions - <br> -<dd> H5B.c -- B-link tree functions -</i></b> - <p> - <LI> If a package is in more than one file, then another name is tacked - on. It's all lower case with no underscores or hyphens. - <P>For Example: -<i><b> -<dd> H5F.c -- the file for this package - <br> -<dd> H5Fstdio.c -- stdio functions (just an example) - <br> -<dd> H5Ffcntl.c -- fcntl functions (just an example) -</i></b> - <p> - <LI> Each package file has a header file of API stuff (unless there is - no API component to the package) - <P>For Example: -<i><b> -<dd> H5F.h -- things an application would see. </i> </b> - <P> - and a header file of private stuff -<i><b> - <p> -<dd> H5Fprivate.h -- things an application wouldn't see. The - private header includes the public header. -</i></b> - <p> - and a header for private prototypes -<i><b> - <p> -<dd> H5Fproto.h -- prototypes for internal functions. -</i></b> - <P> - By splitting the prototypes into separate include files we don't - have to recompile everything when just one function prototype - changes. - - <LI> The main API header file is `hdf5.h' and it includes each of the - public header files but none of the private header files. Or the - application can include just the public header files it needs. - - <LI> There is no main private or prototype header file because it - prevents make from being efficient. Instead, each source file - includes only the private header and prototype files it needs - (first all the private headers, then all the private prototypes). - - <LI> Header files should include everything they need and nothing more. - -</UL> -<P> - -<FONT color="#c80028" -<LI> <A NAME="02"> <B> <I> PACKAGES </I> </B> </A> -</FONT> - -<P> -Names exported beyond function scope begin with `H5' followed by zero, -one, or two upper-case letters that describe the class of object. -This prefix is the package name. The implementation of packages -doesn't necessarily have to map 1:1 to the source files. -<P> -<i><b> -<dd> H5 -- library functions -<br> -<dd> H5A -- atoms -<br> -<dd> H5AC -- cache -<br> -<dd> H5B -- B-link trees -<br> -<dd> H5D -- datasets -<br> -<dd> H5E -- error handling -<br> -<dd> H5F -- files -<br> -<dd> H5G -- groups -<br> -<dd> H5M -- meta data -<br> -<dd> H5MM -- core memory management -<br> -<dd> H5MF -- file memory management -<br> -<dd> H5O -- object headers -<br> -<dd> H5P -- Property Lists -<br> -<dd> H5S -- dataspaces -<br> -<dd> H5R -- relationships -<br> -<dd> H5T -- datatype -</i></b> -<p> -Each package implements a single main class of object (e.g., the H5B -package implements B-link trees). The main data type of a package is -the package name followed by `_t'. -<p> -<i><b> -<dd> H5F_t -- HDF5 file type -<br> -<dd> H5B_t -- B-link tree data type -</i></b> -<p> - -Not all packages implement a data type (H5, H5MF) and some -packages provide access to a preexisting data type (H5MM, H5S). -<p> - - -<FONT color="#c80028" -<LI> <A NAME="03"> <B> <I> PUBLIC vs PRIVATE </I> </B> </A> -</FONT> -<p> -If the symbol is for internal use only, then the package name is -followed by an underscore and the rest of the name. Otherwise, the -symbol is part of the API and there is no underscore between the -package name and the rest of the name. -<p> -<i><b> -<dd> H5Fopen -- an API function. -<br> -<dd> H5B_find -- an internal function. -</i></b> -<p> -For functions, this is important because the API functions never pass -pointers around (they use atoms instead for hiding the implementation) -and they perform stringent checks on their arguments. Internal -unctions, on the other hand, check arguments with assert(). -<p> -Data types like H5B_t carry no information about whether the type is -public or private since it doesn't matter. - -<p> - - -<FONT color="#c80028" -<LI> <A NAME="04"> <B> <I> INTEGRAL TYPES </I> </B> </A> -</FONT> -<p> -Integral fixed-point type names are an optional `u' followed by `int' -followed by the size in bits (8, 16, -32, or 64). There is no trailing `_t' because these are common -enough and follow their own naming convention. -<p> -<pre><H4> -<dd> hbool_t -- boolean values (BTRUE, BFALSE, BFAIL) -<br> -<dd> int8 -- signed 8-bit integers -<br> -<dd> uint8 -- unsigned 8-bit integers -<br> -<dd> int16 -- signed 16-bit integers -<br> -<dd> uint16 -- unsigned 16-bit integers -<br> -<dd> int32 -- signed 32-bit integers -<br> -<dd> uint32 -- unsigned 32-bit integers -<br> -<dd> int64 -- signed 64-bit integers -<br> -<dd> uint64 -- unsigned 64-bit integers -<br> -<dd> intn -- "native" integers -<br> -<dd> uintn -- "native" unsigned integers - -</pre></H4> -<p> - -<FONT color="#c80028" -<LI> <A NAME="05"> <B> <I> OTHER TYPES </I> </B> </A> -</FONT> - -<p> - -Other data types are always followed by `_t'. -<p> -<pre><H4> -<dd> H5B_key_t-- additional data type used by H5B package. -</pre></H4> -<p> - -However, if the name is so common that it's used almost everywhere, -then we make an alias for it by removing the package name and leading -underscore and replacing it with an `h' (the main datatype for a -package already has a short enough name, so we don't have aliases for -them). -<P> -<pre><H4> -<dd> typedef H5E_err_t herr_t; -</pre> </H4> -<p> - -<FONT color="#c80028" -<LI> <A NAME="06"> <B> <I> GLOBAL VARIABLES </I> </B> </A> -</FONT> -<p> -Global variables include the package name and end with `_g'. -<p> -<pre><H4> -<dd> H5AC_methods_g -- global variable in the H5AC package. -</pre> </H4> -<p> - - -<FONT color="#c80028" -<LI> <A NAME="07"> -<I> <B> -MACROS, PREPROCESSOR CONSTANTS, AND ENUM MEMBERS - </I> </B> </A> -</FONT> -<p> -Same rules as other symbols except the name is all upper case. There -are a few exceptions: <br> -<ul> -<li> Constants and macros defined on a system that is deficient: - <p><pre><H4> -<dd> MIN(x,y), MAX(x,y) and their relatives - </pre></H4> - -<li> Platform constants : - <P> - No naming scheme; determined by OS and compiler.<br> - These appear only in one header file anyway. - <p> -<li> Feature test constants (?)<br> - Always start with `HDF5_HAVE_' like HDF5_HAVE_STDARG_H for a - header file, or HDF5_HAVE_DEV_T for a data type, or - HDF5_HAVE_DIV for a function. -</UL> -<p> - -</UL> -<p> -<H6> -<center> - This file /hdf3/web/hdf/internal/HDF_standard/HDF5.coding_standard.html is - maintained by Elena Pourmal <A HREF = "mailto:epourmal@ncsa.uiuc.edu"> - <I>epourmal@ncsa.uiuc.edu</I> </A>. -</center> -<p> -<center> - Last modified August 5, 1997 -</center> - -</H6> -</BODY> -<HTML> - diff --git a/doc/html/TechNotes/ObjectHeader.html b/doc/html/TechNotes/ObjectHeader.html deleted file mode 100644 index 1335d23..0000000 --- a/doc/html/TechNotes/ObjectHeader.html +++ /dev/null @@ -1,72 +0,0 @@ -<html> -<body> - -<h1>Object Headers</h1> - -<pre> - -haddr_t -H5O_new (hdf5_file_t *f, intn nrefs, size_t size_hint) - - Creates a new empty object header and returns its address. - The SIZE_HINT is the initial size of the data portion of the - object header and NREFS is the number of symbol table entries - that reference this object header (normally one). - - If SIZE_HINT is too small, then at least some default amount - of space is allocated for the object header. - -intn /*num remaining links */ -H5O_link (hdf5_file_t *f, /*file containing header */ - haddr_t addr, /*header file address */ - intn adjust) /*link adjustment amount */ - - -size_t -H5O_sizeof (hdf5_file_t *f, /*file containing header */ - haddr_t addr, /*header file address */ - H5O_class_t *type, /*message type or H5O_ANY */ - intn sequence) /*sequence number, usually zero */ - - Returns the size of a particular instance of a message in an - object header. When an object header has more than one - instance of a particular message type, then SEQUENCE indicates - which instance to return. - -void * -H5O_read (hdf5_file_t *f, /*file containing header */ - haddr_t addr, /*header file address */ - H5G_entry_t *ent, /*optional symbol table entry */ - H5O_class_t *type, /*message type or H5O_ANY */ - intn sequence, /*sequence number, usually zero */ - size_t size, /*size of output message */ - void *mesg) /*output buffer */ - - Reads a message from the object header into memory. - -const void * -H5O_peek (hdf5_file_t *f, /*file containing header */ - haddr_t addr, /*header file address */ - H5G_entry_t *ent, /*optional symbol table entry */ - H5O_class_t *type, /*type of message or H5O_ANY */ - intn sequence) /*sequence number, usually zero */ - -haddr_t /*new heap address */ -H5O_modify (hdf5_file_t *f, /*file containing header */ - haddr_t addr, /*header file address */ - H5G_entry_t *ent, /*optional symbol table entry */ - hbool_t *ent_modified, /*entry modification flag */ - H5O_class_t *type, /*message type */ - intn overwrite, /*sequence number or -1 */ - void *mesg) /*the message */ - - -=========================================== -Last Modified: 8 July 1998 (technical content) -Last Modified: 28 April 2000 (included in HDF5 Technical Notes) -HDF Help Desk: hdfhelp@ncsa.uiuc.edu - -</pre> - -</body> -</html> diff --git a/doc/html/TechNotes/RawDStorage.html b/doc/html/TechNotes/RawDStorage.html deleted file mode 100644 index 87ea54d..0000000 --- a/doc/html/TechNotes/RawDStorage.html +++ /dev/null @@ -1,274 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Raw Data Storage in HDF5</title> - </head> - - <body> - <h1>Raw Data Storage in HDF5</h1> - - <p>This document describes the various ways that raw data is - stored in an HDF5 file and the object header messages which - contain the parameters for the storage. - - <p>Raw data storage has three components: the mapping from some - logical multi-dimensional element space to the linear address - space of a file, compression of the raw data on disk, and - striping of raw data across multiple files. These components - are orthogonal. - - <p>Some goals of the storage mechanism are to be able to - efficently store data which is: - - <dl> - <dt>Small - <dd>Small pieces of raw data can be treated as meta data and - stored in the object header. This will be achieved by storing - the raw data in the object header with message 0x0006. - Compression and striping are not supported in this case. - - <dt>Complete Large - <dd>The library should be able to store large arrays - contiguously in the file provided the user knows the final - array size a priori. The array can then be read/written in a - single I/O request. This is accomplished by describing the - storage with object header message 0x0005. Compression and - striping are not supported in this case. - - <dt>Sparse Large - <dd>A large sparse raw data array should be stored in a manner - that is space-efficient but one in which any element can still - be accessed in a reasonable amount of time. Implementation - details are below. - - <dt>Dynamic Size - <dd>One often doesn't have prior knowledge of the size of an - array. It would be nice to allow arrays to grow dynamically in - any dimension. It might also be nice to allow the array to - grow in the negative dimension directions if convenient to - implement. Implementation details are below. - - <dt>Subslab Access - <dd>Some multi-dimensional arrays are almost always accessed by - subslabs. For instance, a 2-d array of pixels might always be - accessed as smaller 1k-by-1k 2-d arrays always aligned on 1k - index values. We should be able to store the array in such a - way that striding though the entire array is not necessary. - Subslab access might also be useful with compression - algorithms where each storage slab can be compressed - independently of the others. Implementation details are below. - - <dt>Compressed - <dd>Various compression algorithms can be applied to the entire - array. We're not planning to support separate algorithms (or a - single algorithm with separate parameters) for each chunk - although it would be possible to implement that in a manner - similar to the way striping across files is - implemented. - - <dt>Striped Across Files - <dd>The array access functions should support arrays stored - discontiguously across a set of files. - </dl> - - <h1>Implementation of Indexed Storage</h1> - - <p>The Sparse Large, Dynamic Size, and Subslab Access methods - share so much code that they can be described with a single - message. The new Indexed Storage Message (<code>0x0008</code>) - will replace the old Chunked Object (<code>0x0009</code>) and - Sparse Object (<code>0x000A</code>) Messages. - - <p> - <center> - <table border cellpadding=4 width="60%"> - <caption align=bottom> - <b>The Format of the Indexed Storage Message</b> - </caption> - <tr align=center> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - </tr> - - <tr align=center> - <td colspan=4><br>Address of B-tree<br><br></td> - </tr> - <tr align=center> - <td>Number of Dimensions</td> - <td>Reserved</td> - <td>Reserved</td> - <td>Reserved</td> - </tr> - <tr align=center> - <td colspan=4>Reserved (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Alignment for Dimension 0 (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Alignment for Dimension 1 (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>...</td> - </tr> - <tr align=center> - <td colspan=4>Alignment for Dimension N (4 bytes)</td> - </tr> - </table> - </center> - - <p>The alignment fields indicate the alignment in logical space to - use when allocating new storage areas on disk. For instance, - writing every other element of a 100-element one-dimensional - array (using one HDF5 I/O partial write operation per element) - that has unit storage alignment would result in 50 - single-element, discontiguous storage segments. However, using - an alignment of 25 would result in only four discontiguous - segments. The size of the message varies with the number of - dimensions. - - <p>A B-tree is used to point to the discontiguous portions of - storage which has been allocated for the object. All keys of a - particular B-tree are the same size and are a function of the - number of dimensions. It is therefore not possible to change the - dimensionality of an indexed storage array after its B-tree is - created. - - <p> - <center> - <table border cellpadding=4 width="60%"> - <caption align=bottom> - <b>The Format of a B-Tree Key</b> - </caption> - <tr align=center> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - </tr> - - <tr align=center> - <td colspan=4>External File Number or Zero (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Chunk Offset in Dimension 0 (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Chunk Offset in Dimension 1 (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>...</td> - </tr> - <tr align=center> - <td colspan=4>Chunk Offset in Dimension N (4 bytes)</td> - </tr> - </table> - </center> - - <p>The keys within a B-tree obey an ordering based on the chunk - offsets. If the offsets in dimension-0 are equal, then - dimension-1 is used, etc. The External File Number field - contains a 1-origin offset into the External File List message - which contains the name of the external file in which that chunk - is stored. - - <h1>Implementation of Striping</h1> - - <p>The indexed storage will support arbitrary striping at the - chunk level; each chunk can be stored in any file. This is - accomplished by using the External File Number field of an - indexed storage B-tree key as a 1-origin offset into an External - File List Message (0x0009) which takes the form: - - <p> - <center> - <table border cellpadding=4 width="60%"> - <caption align=bottom> - <b>The Format of the External File List Message</b> - </caption> - <tr align=center> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - <th width="25%">byte</th> - </tr> - - <tr align=center> - <td colspan=4><br>Name Heap Address<br><br></td> - </tr> - <tr align=center> - <td colspan=4>Number of Slots Allocated (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Number of File Names (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Byte Offset of Name 1 in Heap (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>Byte Offset of Name 2 in Heap (4 bytes)</td> - </tr> - <tr align=center> - <td colspan=4>...</td> - </tr> - <tr align=center> - <td colspan=4><br>Unused Slot(s)<br><br></td> - </tr> - </table> - </center> - - <p>Each indexed storage array that has all or part of its data - stored in external files will contain a single external file - list message. The size of the messages is determined when the - message is created, but it may be possible to enlarge the - message on demand by moving it. At this time, it's not possible - for multiple arrays to share a single external file list - message. - - <dl> - <dt><code> - H5O_efl_t *H5O_efl_new (H5G_entry_t *object, intn - nslots_hint, intn heap_size_hint) - </code> - <dd>Adds a new, empty external file list message to an object - header and returns a pointer to that message. The message - acts as a cache for file descriptors of external files that - are open. - - <p><dt><code> - intn H5O_efl_index (H5O_efl_t *efl, const char *filename) - </code> - <dd>Gets the external file index number for a particular file name. - If the name isn't in the external file list then it's added to - the H5O_efl_t struct and immediately written to the object - header to which the external file list message belongs. Name - comparison is textual. Each name should be relative to the - directory which contains the HDF5 file. - - <p><dt><code> - H5F_low_t *H5O_efl_open (H5O_efl_t *efl, intn index, uintn mode) - </code> - <dd>Gets a low-level file descriptor for an external file. The - external file list caches file descriptors because we might - have many more external files than there are file descriptors - available to this process. The caller should not close this file. - - <p><dt><code> - herr_t H5O_efl_release (H5O_efl_t *efl) - </code> - <dd>Releases an external file list, closes all files - associated with that list, and if the list has been modified - since the call to <code>H5O_efl_new</code> flushes the message - to disk. - </dl> - - <hr> - <address><a href="mailto:robb@arborea.spizella.com">Robb Matzke</a></address> -<!-- Created: Fri Oct 3 09:52:32 EST 1997 --> -<!-- hhmts start --> -Last modified: Tue Nov 25 12:36:50 EST 1997 -<!-- hhmts end --> - </body> -</html> diff --git a/doc/html/TechNotes/ReservedFileSpace.html b/doc/html/TechNotes/ReservedFileSpace.html deleted file mode 100644 index 22e3614..0000000 --- a/doc/html/TechNotes/ReservedFileSpace.html +++ /dev/null @@ -1,29 +0,0 @@ -<html> - <head> - <title>Reserved File Address Space</title> - </head> - - <body> - <hl>Reserved File Address Space</hl> - - <p>HDF5 files use 8-byte addresses by default, but users can change this to 2, 4, or even 16 bytes. This means that it is possible to have files that only address 64 KB of space, and thus that HDF must handle the case of files that have enough space on disk but not enough internal address space to be written.</p> - - <p>Thus, every time space is allocated in a file, HDF needs to check that this allocation is within the file’s address space. If not, HDF should output an error and ensure that all the data currently in the file (everything that is still addressable) is successfully written to disk.</p> - - <p>Unfortunately, some structures are stored in memory and do not allocate space for themselves until the file is actually flushed to disk (object headers and the local heap). This is good for efficiency, since these structures can grow without creating the fragmentation that would result from frequent allocation and deallocation, but means that if the library runs out of addressable space while allocating memory, these structures will not be present in the file. Without them, HDF5 does not know how to parse the data in the file, rendering it unreadable.</p> - - <p>Thus, HDF keeps track of the space “reserved for allocation” in the file (H5FD_t struct). When a function tries to allocate space in the file, it first checks that the allocation would not overflow the address space, taking the reserved space into account. When object headers or the heap finally allocate the space they have reserved, they free the reserved space before allocating file space.</p> - - <p>A given object header is only flushed to disk once, but the heap can be flushed to disk multiple times over the life of the file and will require contiguous space every time. To handle this, the heap keeps track of how much space it has reserved. This allows it to reserve space only when it grows (when it is dirty and needs to be re-written to disk).</p> - - <p>For instance, if the heap is flushed to disk, it frees its reserved space. If new data is inserted into the heap in memory, the heap may need to flush to disk again in a new, larger section of memory. Thus, not only does it reserve space in the file for this new data, but also for all of the previously-existing data in the heap to be re-written. The next insert, however, will only need to reserve space for its new data, since the rest of the heap already has space reserved for it.</p> - - <p>Potential issues: - <ol> - <li>This system does not take into accout deleted space. Deleted space can still be allocated as usual, but "reserved" space is always taken off the end of a file. This means that a file that was filled to capacity but then had a significant number of objects deleted will still throw errors if more data is written. This occurs because the file's free space is in the middle of the file, not at the end. A more complete space-reservation system would test if the reserved data can fit into the file's free list, but this would be significantly more complicated to implement.</li> - - <li>HDF5 currently claims to support 16-byte addresses, but a number of platforms do not support 16-byte integers, so addresses of this size cannot be represented in memory. This solution does not attempt to address this issue.</li> - </ol> - </p> - </body> -</html> diff --git a/doc/html/TechNotes/SWControls.html b/doc/html/TechNotes/SWControls.html deleted file mode 100755 index 8dac1ce..0000000 --- a/doc/html/TechNotes/SWControls.html +++ /dev/null @@ -1,96 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<HTML> -<HEAD> -<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=windows-1252"> -<META NAME="Generator" CONTENT="Microsoft Word 97"> -<TITLE>HDF5 Software Controls</TITLE> -<META NAME="Template" CONTENT="E:\Program Files\Microsoft Office\Office\html.dot"> -</HEAD> - -<BODY LINK="#0000ff" VLINK="#800080"> - -<H1>HDF5 Software Controls</H1> - -<P> -<I>(Work in progress draft)</i> -</P> - -<P> -A descriptions knobs and turns such as environment variables and settings -that controls the functionality of HDF5 libraries and tools. This is -intended for HDF5 libraries and tools developers. HDF5 application users -may consult the document <A HREF="../Debugging.html"><I>A guide to -debugging HDF5 API calls</I></A>. -</P> - -<P> -<H2><FONT FACE="Arial">Library Building Controls</FONT></H2> -</P> - -<P> -<H3><FONT FACE="Arial">Environment variables</FONT></H3> -</P> - -<DL> - <DT><B>CC</B></DT> - <DD><I>Used by configure.</I> Override the default C compiler.</DD> - <DT><B>LIBS</B></DT> - <DD><I>Used by configure.</I> Add more libraries to be used.</DD> - <DT><B>NP</B></DT> - <DD>Number of MPI-processes to invoke for testing. Default to 2.</DD> - <DT><B>HDF5_NOCLEANUP</B></DT> - <DD><I>Used by most test programs.</I> When set, temporary files - created during tests are NOT removed. Default is to remove them - by the end of each test. Note that the variable value does not - matter. E.g., the values of "yes", "no" and "" all have the same - effect, that is, NO cleanup.</DD> - <DT><B>H5FD_mpio_Debug</B></DT> - <DD><I>Used by the MPIO file driver for debugging.</I> Need to have - H5FDmpio_DEBUG macro defined during compiling. Should be set to a - string to turn on various tracing. Valid values (cases matter) - are: - <DL> - <DT>t</DT> - <DD>Trace all routine</DD> - <DT>r</DT> - <DD>Trace read routines</DD> - <DT>w</DT> - <DD>Trace write routines</DD> - <DT>c</DT> - <DD>Show result of MPI_Get_count</DD> - </DL> - </DD> - <DT><B>HDF5_MPI_OPT_TYPES</B></DT> - <DD><I>Used by the MPIO file driver to control the use of the optimized - mpi input/output routine.</I> 0 turns it off, 1 turns it on (uses - optimized code if it can).</DD> -</DL> - -<P> -<H3><FONT FACE="Arial">Compile Macros</FONT></H3> -</P> - -<DL> - <DT><B>H5FDmpio_DEBUG</B></DT> - <DD><I>Compile macro.</I> Compile in the MPIO file driver related debugging - statements. Defined if macro H5F_DEBUG is defined.</DD> - <DT><B>H5FD_mpio_Debug</B></DT> - <DD>Compile in debugging used by the MPIO file driver. Need to have - H5FDmpio_DEBUG macro defined during compiling. Should be set to a - string to turn on the tracing. See environment variable - H5FD_mpio_Debug for valid values.</DD> -</DL> - -<P> -<HR> -</P> - -<ADDRESS> -<A HREF="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</A> -</ADDRESS> - -<P><!-- Created: Fri Oct 3 11:52:31 EST 1997 --><!-- hhmts start -->Last -modified: December 11, 2000 <!-- hhmts end --></P> - -</BODY> -</HTML> diff --git a/doc/html/TechNotes/SymbolTables.html b/doc/html/TechNotes/SymbolTables.html deleted file mode 100644 index 05ee538..0000000 --- a/doc/html/TechNotes/SymbolTables.html +++ /dev/null @@ -1,329 +0,0 @@ -<html> -<body> - -<h1>Symbol Table Caching Issues</h1> - -<pre> - -A number of issues involving caching of object header messages in -symbol table entries must be resolved. - -What is the motivation for these changes? - - If we make objects completely independent of object name it allows - us to refer to one object by multiple names (a concept called hard - links in Unix file systems), which in turn provides an easy way to - share data between datasets. - - Every object in an HDF5 file has a unique, constant object header - address which serves as a handle (or OID) for the object. The - object header contains messages which describe the object. - - HDF5 allows some of the object header messages to be cached in - symbol table entries so that the object header doesn't have to be - read from disk. For instance, an entry for a directory caches the - directory disk addresses required to access that directory, so the - object header for that directory is seldom read. - - If an object has multiple names (that is, a link count greater than - one), then it has multiple symbol table entries which point to it. - All symbol table entries must agree on header messages. The - current mechanism is to turn off the caching of header messages in - symbol table entries when the header link count is more than one, - and to allow caching once the link count returns to one. - - However, in the current implementation, a package is allowed to - copy a symbol table entry and use it as a private cache for the - object header. This doesn't work for a number of reasons (all but - one require a `delete symbol entry' operation). - - 1. If two packages hold copies of the same symbol table entry, - they don't notify each other of changes to the symbol table - entry. Eventually, one package reads a cached message and - gets the wrong value because the other package changed the - message in the object header. - - 2. If one package holds a copy of the symbol table entry and - some other part of HDF5 removes the object and replaces it - with some other object, then the original package will - continue to access the non-existent object using the new - object header. - - 3. If one package holds a copy of the symbol table entry and - some other part of HDF5 (re)moves the directory which - contains the object, then the package will be unable to - update the symbol table entry with the new cached - data. Packages that refer to the object by the new name will - use old cached data. - - -The basic problem is that there may be multiple copies of the object -symbol table entry floating around in the code when there should -really be at most one per hard link. - - Level 0: A copy may exist on disk as part of a symbol table node, which - is a small 1d array of symbol table entries. - - Level 1: A copy may be cached in memory as part of a symbol table node - in the H5Gnode.c file by the H5AC layer. - - Level 2a: Another package may be holding a copy so it can perform - fast lookup of any header messages that might be cached in - the symbol table entry. It can't point directly to the - cached symbol table node because that node can dissappear - at any time. - - Level 2b: Packages may hold more than one copy of a symbol table - entry. For instance, if H5D_open() is called twice for - the same name, then two copies of the symbol table entry - for the dataset exist in the H5D package. - -How can level 2a and 2b be combined? - - If package data structures contained pointers to symbol table - entries instead of copies of symbol table entries and if H5G - allocated one symbol table entry per hard link, then it's trivial - for Level 2a and 2b to benefit from one another's actions since - they share the same cache. - -How does this work conceptually? - - Level 2a and 2b must notify Level 1 of their intent to use (or stop - using) a symbol table entry to access an object header. The - notification of the intent to access an object header is called - `opening' the object and releasing the access is `closing' the - object. - - Opening an object requires an object name which is used to locate - the symbol table entry to use for caching of object header - messages. The return value is a handle for the object. Figure 1 - shows the state after Dataset1 opens Object with a name that maps - through Entry1. The open request created a copy of Entry1 called - Shadow1 which exists even if SymNode1 is preempted from the H5AC - layer. - - ______ - Object / \ - SymNode1 +--------+ | - +--------+ _____\ | Header | | - | | / / +--------+ | - +--------+ +---------+ \______/ - | Entry1 | | Shadow1 | /____ - +--------+ +---------+ \ \ - : : \ - +--------+ +----------+ - | Dataset1 | - +----------+ - FIGURE 1 - - - - The SymNode1 can appear and disappear from the H5AC layer at any - time without affecting the Object Header data cached in the Shadow. - The rules are: - - * If the SymNode1 is present and is about to disappear and the - Shadow1 dirty bit is set, then Shadow1 is copied over Entry1, the - Entry1 dirty bit is set, and the Shadow1 dirty bit is cleared. - - * If something requests a copy of Entry1 (for a read-only peek - request), and Shadow1 exists, then a copy (not pointer) of Shadow1 - is returned instead. - - * Entry1 cannot be deleted while Shadow1 exists. - - * Entry1 cannot change directly if Shadow1 exists since this means - that some other package has opened the object and may be modifying - it. I haven't decided if it's useful to ever change Entry1 - directly (except of course within the H5G layer itself). - - * Shadow1 is created when Dataset1 `opens' the object through - Entry1. Dataset1 is given a pointer to Shadow1 and Shadow1's - reference count is incremented. - - * When Dataset1 `closes' the Object the Shadow1 reference count is - decremented. When the reference count reaches zero, if the - Shadow1 dirty bit is set, then Shadow1's contents are copied to - Entry1, and the Entry1 dirty bit is set. Shadow1 is then deleted - if its reference count is zero. This may require reading SymNode1 - back into the H5AC layer. - -What happens when another Dataset opens the Object through Entry1? - - If the current state is represented by the top part of Figure 2, - then Dataset2 will be given a pointer to Shadow1 and the Shadow1 - reference count will be incremented to two. The Object header link - count remains at one so Object Header messages continue to be cached - by Shadow1. Dataset1 and Dataset2 benefit from one another - actions. The resulting state is represented by Figure 2. - - _____ - SymNode1 Object / \ - +--------+ _____\ +--------+ | - | | / / | Header | | - +--------+ +---------+ +--------+ | - | Entry1 | | Shadow1 | /____ \_____/ - +--------+ +---------+ \ \ - : : _ \ - +--------+ |\ +----------+ - \ | Dataset1 | - \________ +----------+ - \ \ - +----------+ | - | Dataset2 | |- New Dataset - +----------+ | - / - FIGURE 2 - - -What happens when the link count for Object increases while Dataset -has the Object open? - - SymNode2 - +--------+ - SymNode1 Object | | - +--------+ ____\ +--------+ /______ +--------+ - | | / / | header | \ `| Entry2 | - +--------+ +---------+ +--------+ +--------+ - | Entry1 | | Shadow1 | /____ : : - +--------+ +---------+ \ \ +--------+ - : : \ - +--------+ +----------+ \________________/ - | Dataset1 | | - +----------+ New Link - - FIGURE 3 - - The current state is represented by the left part of Figure 3. To - create a new link the Object Header had to be located by traversing - through Entry1/Shadow1. On the way through, the Entry1/Shadow1 - cache is invalidated and the Object Header link count is - incremented. Entry2 is then added to SymNode2. - - Since the Object Header link count is greater than one, Object - header data will not be cached in Entry1/Shadow1. - - If the initial state had been all of Figure 3 and a third link is - being added and Object is open by Entry1 and Entry2, then creation - of the third link will invalidate the cache in Entry1 or Entry2. It - doesn't matter which since both caches are already invalidated - anyway. - -What happens if another Dataset opens the same object by another name? - - If the current state is represented by Figure 3, then a Shadow2 is - created and associated with Entry2. However, since the Object - Header link count is more than one, nothing gets cached in Shadow2 - (or Shadow1). - -What happens if the link count decreases? - - If the current state is represented by all of Figure 3 then it isn't - possible to delete Entry1 because the object is currently open - through that entry. Therefore, the link count must have - decreased because Entry2 was removed. - - As Dataset1 reads/writes messages in the Object header they will - begin to be cached in Shadow1 again because the Object header link - count is one. - -What happens if the object is removed while it's open? - - That operation is not allowed. - -What happens if the directory containing the object is deleted? - - That operation is not allowed since deleting the directory requires - that the directory be empty. The directory cannot be emptied - because the open object cannot be removed from the directory. - -What happens if the object is moved? - - Moving an object is a process consisting of creating a new - hard-link with the new name and then deleting the old name. - This will fail if the object is open. - -What happens if the directory containing the entry is moved? - - The entry and the shadow still exist and are associated with one - another. - -What if a file is flushed or closed when objects are open? - - Flushing a symbol table with open objects writes correct information - to the file since Shadow is copied to Entry before the table is - flushed. - - Closing a file with open objects will create a valid file but will - return failure. - -How is the Shadow associated with the Entry? - - A symbol table is composed of one or more symbol nodes. A node is a - small 1-d array of symbol table entries. The entries can move - around within a node and from node-to-node as entries are added or - removed from the symbol table and nodes can move around within a - symbol table, being created and destroyed as necessary. - - Since a symbol table has an object header with a unique and constant - file offset, and since H5G contains code to efficiently locate a - symbol table entry given it's name, we use these two values as a key - within a shadow to associate the shadow with the symbol table - entry. - - struct H5G_shadow_t { - haddr_t stab_addr; /*symbol table header address*/ - char *name; /*entry name wrt symbol table*/ - hbool_t dirty; /*out-of-date wrt stab entry?*/ - H5G_entry_t ent; /*my copy of stab entry */ - H5G_entry_t *main; /*the level 1 entry or null */ - H5G_shadow_t *next, *prev; /*other shadows for this stab*/ - }; - - The set of shadows will be organized in a hash table of linked - lists. Each linked list will contain the shadows associated with a - particular symbol table header address and the list will be sorted - lexicographically. - - Also, each Entry will have a pointer to the corresponding Shadow or - null if there is no shadow. - - When a symbol table node is loaded into the main cache, we look up - the linked list of shadows in the shadow hash table based on the - address of the symbol table object header. We then traverse that - list matching shadows with symbol table entries. - - We assume that opening/closing objects will be a relatively - infrequent event compared with loading/flushing symbol table - nodes. Therefore, if we keep the linked list of shadows sorted it - costs O(N) to open and close objects where N is the number of open - objects in that symbol table (instead of O(1)) but it costs only - O(N) to load a symbol table node (instead of O(N^2)). - -What about the root symbol entry? - - Level 1 storage for the root symbol entry is always available since - it's stored in the hdf5_file_t struct instead of a symbol table - node. However, the contents of that entry can move from the file - handle to a symbol table node by H5G_mkroot(). Therefore, if the - root object is opened, we keep a shadow entry for it whose - `stab_addr' field is zero and whose `name' is null. - - For this reason, the root object should always be read through the - H5G interface. - -One more key invariant: The H5O_STAB message in a symbol table header -never changes. This allows symbol table entries to cache the H5O_STAB -message for the symbol table to which it points without worrying about -whether the cache will ever be invalidated. - - -=========================================== -Last Modified: 8 July 1998 (technical content) -Last Modified: 28 April 2000 (included in HDF5 Technical Notes) -HDF Help Desk: hdfhelp@ncsa.uiuc.edu - -</pre> - -</body> -</html> diff --git a/doc/html/TechNotes/TestReview.html b/doc/html/TechNotes/TestReview.html deleted file mode 100644 index 410f662..0000000 --- a/doc/html/TechNotes/TestReview.html +++ /dev/null @@ -1,57 +0,0 @@ -<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<HTML> -<HEAD> - <TITLE>State of API Test Review for HDF5</TITLE> - <META http-equiv="content-type" content="text/html; charset=ISO-8859-1"> - <META name="author" content="Quincey Koziol"> -</HEAD> -<body text="#000000" bgcolor="#FFFFFF"> -<STYLE type="text/css"> -OL.loweralpha { list-style-type: lower-alpha } -OL.upperroman { list-style-type: upper-roman } -</STYLE> - -<CENTER><H1>State of API Testing Review for HDF5</H1></CENTER> - -<OL class="upperroman"> - -<LI><H3><U>Purpose:</U></H3> -<P>This document describes the current state of the API test review. Currently, -the tests for each API function are being reviewed on an individual basis and -each API's tests are being described and improvements made. -</P> - -<LI><H3><U>APIs Reviewed:</U></H3> - -<TABLE border=1 cellpadding=5> -<TR> -<TH>API Function</TH> -<TH>Date Last Reviewed</TH> -<TH>Status</TH> -</TR> - -<TR> -<TD><A href="TestReview/H5Dget_offset.html">H5Dget_offset</A> -</TD> -<TD>Tuesday, November 11th, 2002 -</TD> -<TD><FONT color="orange">Tests need to be updated</FONT> -</TD> -</TR> - -<TR> -<TD><A href="TestReview/H5Tget_native_type.html">H5Tget_native_type</A> -</TD> -<TD>Tuesday, November 11th, 2002 -</TD> -<TD><FONT color="orange">Tests need to be updated</FONT> -</TD> -</TR> - -</TABLE> - - -</OL> - -</BODY> -</HTML> diff --git a/doc/html/TechNotes/TestReview/H5Dget_offset.html b/doc/html/TechNotes/TestReview/H5Dget_offset.html deleted file mode 100644 index 0056f00..0000000 --- a/doc/html/TechNotes/TestReview/H5Dget_offset.html +++ /dev/null @@ -1,199 +0,0 @@ -<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<HTML> -<HEAD> - <TITLE>H5Dget_offset Test Review</TITLE> - <META http-equiv="content-type" content="text/html; charset=ISO-8859-1"> - <META name="author" content="Quincey Koziol"> -</HEAD> -<body text="#000000" bgcolor="#FFFFFF"> -<STYLE type="text/css"> -OL.loweralpha { list-style-type: lower-alpha } -OL.upperroman { list-style-type: upper-roman } -</STYLE> - -<CENTER><H1>H5Dget_offset Test Review</H1></CENTER> - -<OL class="upperroman"> - -<LI><H3><U>Purpose:</U></H3> -<P>This document describes the API test review results for <a href="../../RM_H5D.html#Dataset-GetOffset">H5Dget_offset</a>(). -</P> - -<LI><H3><U>Serial Review:</U></H3> -<TABLE border="1"> -<TR> -<TH>Test case -</TH> - -<TH>Test source file -</TH> - -<TH>Test method -</TH> - -<TH>Expected test results -</TH> - -<TH>Notes -</TH> - -<TR> -<TD>Chunked dataset -</TD> - -<TD>dsets.c -</TD> - -<TD> -<OL> -<LI>Create chunked dataset -<LI>Query dataset offset -</OL> -</TD> - -<TD>FAIL -</TD> - -<TD> -<P>Because dataset is stored in chunks that are indexed by a B-tree, there is -no single piece of data to query the offset of. -</P> -<P>It may be possible in the future to -enhance this function by querying the offset of a particular chunk (or chunks), -but that has limited use because chunks could be compressed, etc. with an I/O -filter. -</P> -</TD> - -</TR> - -<TR> -<TD>Compact dataset -</TD> - -<TD>dsets.c -</TD> - -<TD> -<OL> -<LI>Create chunked dataset -<LI>Query dataset offset -</OL> -</TD> - -<TD>FAIL -</TD> - -<TD> -<P>Because dataset is stored in the object header of the dataset, there is -no separate piece of data to query the offset of. -</P> -<P>It may be possible in the future to get the offset of the data in the object -header, but this is problematic due to the fact that the messages in the object -header can get relocated in the file when changes (like adding attributes, etc.) -are made to the dataset, invalidating the address given to the user. -filter. -</P> -</TD> - -</TR> - -<TR> -<TD>Contiguous dataset, [user block size] == 0, not external -</TD> - -<TD>dsets.c -</TD> - -<TD> -<OL> -<LI>Create file with 0 sized user-block (the default) -<LI>Create contigous dataset -<LI>Query dataset offset -</OL> -</TD> - -<TD> -<P>Succeed in getting the proper address and be able to verify -that the data at that address in the file is what was written out. -</P> -<P>When data storage allocation is "late" (the default), querying the offset -should fail if performed before data is written to the dataset. -</P> -</TD> - -<TD>Needs additional test to verify that the data written out is located at the -correct offset in the file. -</TD> - -</TR> - -<TR> -<TD>Contiguous dataset, [user block size] != 0, not external -</TD> - -<TD>dsets.c -</TD> - -<TD> -<OL> -<LI>Create file with non-0 sized user-block -<LI>Create contigous dataset -<LI>Query dataset offset -</OL> -</TD> - -<TD> -<P>Succeed in getting the proper address and be able to verify -that the data at that address in the file is what was written out. -</P> -<P>When data storage allocation is "late" (the default), querying the offset -should fail if performed before data is written to the dataset. -</P> -</TD> - -<TD>Needs test for this case. -</TD> - -</TR> - -<TR> -<TD>Contiguous dataset, [user block size] == 0, external data storage -</TD> - -<TD>external.c -</TD> - -<TD> -<OL> -<LI>Create contigous dataset with external storage -<LI>Query dataset offset -</OL> -</TD> - -<TD>FAIL -</TD> - -<TD> -<P>In theory, it's easy to return the offset of the data in the external file, -but this wasn't done because it would be too easy for users to assume that the -offset returned was in the HDF5 file instead of the external file. -</P> -</TD> - -</TR> - -</TABLE> -<BR> - -<LI><H3><U>Parallel Review:</U></H3> -<P>The H5Dget_offset() function is not tested in parallel. Currently, there -does not appear to be a need for this. -</P> - - -</OL> - -</BODY> -</HTML> - diff --git a/doc/html/TechNotes/TestReview/H5Tget_native_type.html b/doc/html/TechNotes/TestReview/H5Tget_native_type.html deleted file mode 100644 index 1c6409d..0000000 --- a/doc/html/TechNotes/TestReview/H5Tget_native_type.html +++ /dev/null @@ -1,522 +0,0 @@ -<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> -<HTML> -<HEAD> - <TITLE>H5Dget_native_type Test Review</TITLE> - <META http-equiv="content-type" content="text/html; charset=ISO-8859-1"> - <META name="author" content="Quincey Koziol"> -</HEAD> -<body text="#000000" bgcolor="#FFFFFF"> -<STYLE type="text/css"> -OL.loweralpha { list-style-type: lower-alpha } -OL.upperroman { list-style-type: upper-roman } -</STYLE> - -<CENTER><H1>H5Dget_native_type Test Review</H1></CENTER> - -<OL class="upperroman"> - -<LI><H3><U>Purpose:</U></H3> -<P>This document describes the API test review results for <a href="../../RM_H5T.html#Datatype-GetNativeType">H5Dget_native_type</a>(). -</P> - -<LI><H3><U>Serial Review:</U></H3> -<TABLE border="1"> -<TR> -<TH>Test case -</TH> - -<TH>Test source file -</TH> - -<TH>Test method -</TH> - -<TH>Expected test results -</TH> - -<TH>Notes -</TH> - -<TR> -<TD>Native int datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create dataset with I32BE datatype -<LI>Query dataset's datatype -<LI>Create native datatype from dataset datatype -<LI>Compare order, class & size of native datatype to known results -</OL> -</TD> - -<TD>Check that type's size, order and class are correct. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P>It would be convenient to have a function in the test module for choosing -the correct atomic datatype based on the particular platform settings. This -should use the H5_SIZEOF_<foo> macros. -</P> -</TD> - -</TR> - -<TR> -<TD>Native long long datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create dataset with I64LE datatype -<LI>Query dataset's datatype -<LI>Create native datatype from dataset datatype -<LI>Compare order, class & size of native datatype to known results -</OL> -</TD> - -<TD>Check that type's size, order and class are correct. -</TD> - -<TD> -<P>Data is NOT written & read back in for this test. -</P> -</TD> - -</TR> - -<TR> -<TD>Native char datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create dataset with I8LE datatype -<LI>Query dataset's datatype -<LI>Create native datatype from dataset datatype -<LI>Compare order, class & size of native datatype to known results -</OL> -</TD> - -<TD>Check that type's size, order and class are correct. -</TD> - -<TD> -<P>Data is NOT written & read back in for this test. -</P> -</TD> - -</TR> - -<TR> -<TD>Native float datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create dataset with F32BE datatype -<LI>Query dataset's datatype -<LI>Create native datatype from dataset datatype -<LI>Compare order, class & size of native datatype to known results -</OL> -</TD> - -<TD>Check that type's size, order and class are correct. -</TD> - -<TD> -<P>Data is NOT written & read back in for this test. -</P> -<P>Need test for native double datatype (stored as 32-bit floating-point -datatype in file). This will probably require using an "epsilon" if the data -is compared for this test. -</P> -</TD> - -</TR> - -<TR> -<TD>Compound datatype with atomic fields -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create datatype describing native (unpacked) struct in memory -<LI>Create datatype describing packed struct for disk -<LI>Create dataset with "packed" compound datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the native, -unpacked datatype. -</OL> -</TD> - -<TD>Check that native and unpacked datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -</TD> - -</TR> - -<TR> -<TD>Compound datatype with one compound field -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create datatype describing nested native (unpacked) structs in memory -<LI>Create datatype describing nested packed structs for disk -<LI>Create dataset with "packed" compound datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the native, -unpacked datatype. -</OL> -</TD> - -<TD>Check that native and unpacked datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P>Could use test for compound datatype with multiple compound fields. -</P> -<P>Could use test for 3 or more nested deep compound datatype. -</P> -</TD> - -</TR> - -<TR> -<TD>Enum datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create enum datatype -<LI>Create dataset with enum datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -enum datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -</TD> - -</TR> - -<TR> -<TD>Array datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create array datatype -<LI>Create dataset with array datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -array datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P><EM>This is not tested currently.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>Array of compound datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create array of compound datatype -<LI>Create dataset with array of compound datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -array of compound datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -</TD> - -</TR> - -<TR> -<TD>Compound datatype with array field -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create compound datatype with array field -<LI>Create dataset with compound datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -compound datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P><EM>This is not tested currently.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>VL datatype with atomic base type -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create VL datatype -<LI>Create dataset with VL datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -VL datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P><EM>Combinations with VL datatypes in other composite types and with other -datatypes for the base type of the VL datatype are not tested.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>VL string datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create VL string datatype -<LI>Create dataset with VL string datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -VL string datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P><EM>Combinations with VL string datatypes in composite types -are not tested.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>Reference datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create reference datatype -<LI>Create dataset with reference datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -reference datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P><EM>Combinations with reference datatypes in composite types -are not tested.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>Opaque datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create opaque datatype -<LI>Create dataset with opaque datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -opaque datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P><EM>Combinations with opaque datatypes in composite types -are not tested.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>Bitfield datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create bitfield datatype -<LI>Create dataset with bitfield datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -bitfield datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P>Data is written & read back in for this test. -</P> -<P><EM>Combinations with bitfield datatypes in composite types -are not tested.</EM> -</P> -</TD> - -</TR> - -<TR> -<TD>Time datatype -</TD> - -<TD>native.c -</TD> - -<TD> -<OL> -<LI>Create time datatype -<LI>Create dataset with time datatype -<LI>Query dataset's datatype -<LI>Get native datatype from dataset's datatype -<LI>Use H5Tequal to verify that the native datatype is the same as the original -time datatype. -</OL> -</TD> - -<TD>Check that native and original datatypes are equal. -</TD> - -<TD> -<P><EM>This is not tested currently.</EM> -</P> -</TD> - -</TR> - -</TABLE> -<BR> - -<LI><H3><U>Parallel Review:</U></H3> -<P>The H5Dget_native_type() function is not tested in parallel. Currently, -there does not appear to be a need for this. -</P> - - -</OL> - -</BODY> -</HTML> - diff --git a/doc/html/TechNotes/ThreadSafeLibrary.html b/doc/html/TechNotes/ThreadSafeLibrary.html deleted file mode 100644 index e7fdf11..0000000 --- a/doc/html/TechNotes/ThreadSafeLibrary.html +++ /dev/null @@ -1,794 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" - "http://www.w3.org/TR/REC-html40/loose.dtd"> -<html lang="en-US"> -<head> - <title>Thread Safe Library</title> -</head> - -<body bgcolor=#ffffff> - -<center><h1>HDF5 Thread Safe library</h1></center> - -<p> - -<h1>1. Library header files and conditional compilation</h1> - -<p> -The following code is placed at the beginning of H5private.h: -</p> - -<blockquote> - <pre> - #ifdef H5_HAVE_THREADSAFE - #include <pthread.h> - #endif - </pre> -</blockquote> - -<p> -<code>H5_HAVE_THREADSAFE</code> is defined when the HDF-5 library is -compiled with the --enable-threadsafe configuration option. In general, -code for the non-threadsafe version of HDF-5 library are placed within -the <code>#else</code> part of the conditional compilation. The exception -to this rule are the changes to the <code>FUNC_ENTER</code> (in -H5private.h), <code>HRETURN</code> and <code>HRETURN_ERROR</code> (in -H5Eprivate.h) macros (see section 3.2). -</p> - - -<h1>2. Global variables/structures</h1> - -<h2>2.1 Global library initialization variable</h2> - -<p> -In the threadsafe implementation, the global library initialization -variable <code>H5_libinit_g</code> is changed to a global structure -consisting of the variable with its associated lock (locks are explained -in section 4.1): -</p> - -<blockquote> - <pre> - hbool_t H5_libinit_g = FALSE; - </pre> -</blockquote> - -<p> -becomes -</p> - -<blockquote> - <pre> - H5_api_t H5_g; - </pre> -</blockquote> - -<p> -where <code>H5_api_t</code> is -</p> - -<blockquote> - <pre> - typedef struct H5_api_struct { - H5_mutex_t init_lock; /* API entrance mutex */ - hbool_t H5_libinit_g; - } H5_api_t; - </pre> -</blockquote> - -<p> -All former references to <code>H5_libinit_g</code> in the library are now -made using the macro <code>H5_INIT_GLOBAL</code>. If the threadsafe -library is to be used, the macro is set to <code>H5_g.H5_libinit_g</code> -instead. -</p> - -<h2>2.2 Global serialization variable</h2> - -<p> -A new global boolean variable <code>H5_allow_concurrent_g</code> is used -to determine if multiple threads are allowed to an API call -simultaneously. This is set to <code>FALSE</code>. -</p> - -<p> -All APIs that are allowed to do so have their own local variable that -shadows the global variable and is set to <code>TRUE</code>. In phase 1, -no such APIs exist. -</p> - -<p> -It is defined in <code>H5.c</code> as follows: -</p> - -<blockquote> - <pre> - hbool_t H5_allow_concurrent_g = FALSE; - </pre> -</blockquote> - -<h2>2.3 Global thread initialization variable</h2> - -<p> -The global variable <code>H5_first_init_g</code> of type -<code>pthread_once_t</code> is used to allow only the first thread in the -application process to call an initialization function using -<code>pthread_once</code>. All subsequent calls to -<code>pthread_once</code> by any thread are disregarded. -</p> - -<p> -The call sets up the mutex in the global structure <code>H5_g</code> (see -section 3.1) via an initialization function -<code>H5_first_thread_init</code>. The first thread initialization -function is described in section 4.2. -</p> - -<p> -<code>H5_first_init_g</code> is defined in <code>H5.c</code> as follows: -</p> - -<blockquote> - <pre> - pthread_once_t H5_first_init_g = PTHREAD_ONCE_INIT; - </pre> -</blockquote> - -<h2>2.4 Global key for per-thread error stacks</h2> - -<p> -A global pthread-managed key <code>H5_errstk_key_g</code> is used to -allow pthreads to maintain a separate error stack (of type -<code>H5E_t</code>) for each thread. This is defined in <code>H5.c</code> -as: -</p> - -<blockquote> - <pre> - pthread_key_t H5_errstk_key_g; - </pre> -</blockquote> - -<p> -Error stack management is described in section 4.3. -</p> - -<h2>2.5 Global structure and key for thread cancellation prevention</h2> - -<p> -We need to preserve the thread cancellation status of each thread -individually by using a key <code>H5_cancel_key_g</code>. The status is -preserved using a structure (of type <code>H5_cancel_t</code>) which -maintains the cancellability state of the thread before it entered the -library and a count (which works very much like the recursive lock -counter) which keeps track of the number of API calls the thread makes -within the library. -</p> - -<p> -The structure is defined in <code>H5private.h</code> as: -</p> - -<blockquote> - <pre> - /* cancelability structure */ - typedef struct H5_cancel_struct { - int previous_state; - unsigned int cancel_count; - } H5_cancel_t; - </pre> -</blockquote> - -<p> -Thread cancellation is described in section 4.4. -</p> - - -<h1>3. Changes to Macro expansions</h1> - -<h2>3.1 Changes to FUNC_ENTER</h2> - -<p> -The <code>FUNC_ENTER</code> macro is now extended to include macro calls -to initialize first threads, disable cancellability and wraps a lock -operation around the checking of the global initialization flag. It -should be noted that the cancellability should be disabled before -acquiring the lock on the library. Doing so otherwise would allow the -possibility that the thread be cancelled just after it has acquired the -lock on the library and in that scenario, if the cleanup routines are not -properly set, the library would be permanently locked out. -</p> - -<p> -The additional macro code and new macro definitions can be found in -Appendix E.1 to E.5. The changes are made in <code>H5private.h</code>. -</p> - -<h2>3.2 Changes to HRETURN and HRETURN_ERROR</h2> - -<p> -The <code>HRETURN</code> and <code>HRETURN_ERROR</code> macros are the -counterparts to the <code>FUNC_ENTER</code> macro described in section -3.1. <code>FUNC_LEAVE</code> makes a macro call to <code>HRETURN</code>, -so it is also covered here. -</p> - -<p> -The basic changes to these two macros involve adding macro calls to call -an unlock operation and re-enable cancellability if necessary. It should -be noted that the cancellability should be re-enabled only after the -thread has released the lock to the library. The consequence of doing -otherwise would be similar to that described in section 3.1. -</p> - -<p> -The additional macro code and new macro definitions can be found in -Appendix E.9 to E.9. The changes are made in <code>H5Eprivate.h</code>. -</p> - -<h1>4. Implementation of threadsafe functionality</h1> - -<h2>4.1 Recursive Locks</h2> - -<p> -A recursive mutex lock m allows a thread t1 to successfully lock m more -than once without blocking t1. Another thread t2 will block if t2 tries -to lock m while t1 holds the lock to m. If t1 makes k lock calls on m, -then it also needs to make k unlock calls on m before it releases the -lock. -</p> - -<p> -Our implementation of recursive locks is built on top of a pthread mutex -lock (which is not recursive). It makes use of a pthread condition -variable to have unsuccessful threads wait on the mutex. Waiting threads -are awaken by a signal from the final unlock call made by the thread -holding the lock. -</p> - -<p> -Recursive locks are defined to be the following type -(<code>H5private.h</code>): -</p> - -<blockquote> - <pre> - typedef struct H5_mutex_struct { - pthread_t owner_thread; /* current lock owner */ - pthread_mutex_t atomic_lock; /* lock for atomicity of new mechanism */ - pthread_cond_t cond_var; /* condition variable */ - unsigned int lock_count; - } H5_mutex_t; - </pre> -</blockquote> - -<p> -Detailed implementation code can be found in Appendix A. The -implementation changes are made in <code>H5TS.c</code>. -</p> - -<h2>4.2 First thread initialization</h2> - -<p> -Because the mutex lock associated with a recursive lock cannot be -statically initialized, a mechanism is required to initialize the -recursive lock associated with <code>H5_g</code> so that it can be used -for the first time. -</p> - -<p> -The pthreads library allows this through the pthread_once call which as -described in section 3.3 allows only the first thread accessing the -library in an application to initialize <code>H5_g</code>. -</p> - -<p> -In addition to initializing <code>H5_g</code>, it also initializes the -key (see section 3.4) for use with per-thread error stacks (see section -4.3). -</p> - -<p> -The first thread initialization mechanism is implemented as the function -call <code>H5_first_thread_init()</code> in <code>H5TS.c</code>. This is -described in appendix B. -</p> - -<h2>4.3 Per-thread error stack management</h2> - -<p> -Pthreads allows individual threads to access dynamic and persistent -per-thread data through the use of keys. Each key is associated with -a table that maps threads to data items. Keys can be initialized by -<code>pthread_key_create()</code> in pthreads (see sections 3.4 and 4.2). -Per-thread data items are accessed using a key through the -<code>pthread_getspecific()</code> and <code>pthread_setspecific()</code> -calls to read and write to the association table respectively. -</p> - -<p> -Per-thread error stacks are accessed through the key -<code>H5_errstk_key_g</code> which is initialized by the first thread -initialization call (see section 4.2). -</p> - -<p> -In the non-threadsafe version of the library, there is a global stack -variable <code>H5E_stack_g[1]</code> which is no longer defined in the -threadsafe version. At the same time, the macro call to gain access to -the error stack <code>H5E_get_my_stack</code> is changed from: -</p> - -<blockquote> - <pre> - #define H5E_get_my_stack() (H5E_stack_g+0) - </pre> -</blockquote> - -<p> -to: -</p> - -<blockquote> - <pre> - #define H5E_get_my_stack() H5E_get_stack() - </pre> -</blockquote> - -<p> -where <code>H5E_get_stack()</code> is a surrogate function that does the -following operations: -</p> - -<ol> - <li>if a thread is attempting to get an error stack for the first - time, the error stack is dynamically allocated for the thread and - associated with <code>H5_errstk_key_g</code> using - <code>pthread_setspecific()</code>. The way we detect if it is the - first time is through <code>pthread_getspecific()</code> which - returns <code>NULL</code> if no previous value is associated with - the thread using the key.</li> - - <li>if <code>pthread_getspecific()</code> returns a non-null value, - then that is the pointer to the error stack associated with the - thread and the stack can be used as usual.</li> -</ol> - -<p> -A final change to the error reporting routines is as follows; the current -implementation reports errors to always be detected at thread 0. In the -threadsafe implementation, this is changed to report the number returned -by a call to <code>pthread_self()</code>. -</p> - -<p> -The change in code (reflected in <code>H5Eprint</code> of file -<code>H5E.c</code>) is as follows: -</p> - -<blockquote> - <pre> - #ifdef H5_HAVE_THREADSAFE - fprintf (stream, "HDF5-DIAG: Error detected in thread %d." - ,pthread_self()); - #else - fprintf (stream, "HDF5-DIAG: Error detected in thread 0."); - #endif - </pre> -</blockquote> - -<p> -Code for <code>H5E_get_stack()</code> can be found in Appendix C. All the -above changes were made in <code>H5E.c</code>. -</p> - -<h2>4.4 Thread Cancellation safety</h2> - -<p> -To prevent thread cancellations from killing a thread while it is in the -library, we maintain per-thread information about the cancellability -status of the thread before it entered the library so that we can restore -that same status when the thread leaves the library. -</p> - -<p> -By <i>enter</i> and <i>leave</i> the library, we mean the points when a -thread makes an API call from a user application and the time that API -call returns. Other API or callback function calls made from within that -API call are considered <i>within</i> the library. -</p> - -<p> -Because other API calls may be made from within the first API call, we -need to maintain a counter to determine which was the first and -correspondingly the last return. -</p> - -<p> -When a thread makes an API call, the macro <code>H5_API_SET_CANCEL</code> -calls the worker function <code>H5_cancel_count_inc()</code> which does -the following: -</p> - -<ol> - <li>if this is the first time the thread has entered the library, - a new cancellability structure needs to be assigned to it.</li> - <li>if the thread is already within the library when the API call is - made, then cancel_count is simply incremented. Otherwise, we set - the cancellability state to <code>PTHREAD_CANCEL_DISABLE</code> - while storing the previous state into the cancellability structure. - <code>cancel_count</code> is also incremented in this case.</li> -</ol> - -<p> -When a thread leaves an API call, the macro -<code>H5_API_UNSET_CANCEL</code> calls the worker function -<code>H5_cancel_count_dec()</code> which does the following: -</p> - -<ol> - <li>if <code>cancel_count</code> is greater than 1, indicating that the - thread is not yet about to leave the library, then - <code>cancel_count</code> is simply decremented.</li> - <li>otherwise, we reset the cancellability state back to its original - state before it entered the library and decrement the count (back - to zero).</li> -</ol> - -<p> -<code>H5_cancel_count_inc</code> and <code>H5_cancel_count_dec</code> are -described in Appendix D and may be found in <code>H5TS.c</code>. -</p> - -<h1>5. Test programs</h1> - -<p> -Except where stated, all tests involve 16 simultaneous threads that make -use of HDF-5 API calls without any explicit synchronization typically -required in a non-threadsafe environment. -</p> - -<h2>5.1 Data set create and write</h2> - -<p> -The test program sets up 16 threads to simultaneously create 16 -different datasets named from <i>zero</i> to <i>fifteen</i> for a single -file and then writing an integer value into that dataset equal to the -dataset's named value. -</p> - -<p> -The main thread would join with all 16 threads and attempt to match the -resulting HDF-5 file with expected results - that each dataset contains -the correct value (0 for <i>zero</i>, 1 for <i>one</i> etc ...) and all -datasets were correctly created. -</p> - -<p> -The test is implemented in the file <code>ttsafe_dcreate.c</code>. -</p> - -<h2>5.2 Test on error stack</h2> - -<p> -The error stack test is one in which 16 threads simultaneously try to -create datasets with the same name. The result, when properly serialized, -should be equivalent to 16 attempts to create the dataset with the same -name. -</p> - -<p> -The error stack implementation runs correctly if it reports 15 instances -of the dataset name conflict error and finally generates a correct HDF-5 -containing that single dataset. Each thread should report its own stack -of errors with a thread number associated with it. -</p> - -<p> -The test is implemented in the file <code>ttsafe_error.c</code>. -</p> - -<h2>5.3 Test on cancellation safety</h2> - -<p> -The main idea in thread cancellation safety is as follows; a child thread -is spawned to create and write to a dataset. Following that, it makes a -<code>H5Diterate</code> call on that dataset which activates a callback -function. -</p> - -<p> -A deliberate barrier is invoked at the callback function which waits for -both the main and child thread to arrive at that point. After that -happens, the main thread proceeds to make a thread cancel call on the -child thread while the latter sleeps for 3 seconds before proceeding to -write a new value to the dataset. -</p> - -<p> -After the iterate call, the child thread logically proceeds to wait -another 3 seconds before writing another newer value to the dataset. -</p> - -<p> -The test is correct if the main thread manages to read the second value -at the end of the test. This means that cancellation did not take place -until the end of the iteration call despite of the 3 second wait within -the iteration callback and the extra dataset write operation. -Furthermore, the cancellation should occur before the child can proceed -to write the last value into the dataset. -</p> - -<h2>5.4 Test on attribute creation</h2> - -<p> -A main thread makes 16 threaded calls to <code>H5Acreate</code> with a -generated name for each attribute. Sixteen attributes should be created -for the single dataset in random (chronological) order and receive values -depending on its generated attribute name (e.g. <i>attrib010</i> would -receive the value 10). -</p> - -<p> -After joining with all child threads, the main thread proceeds to read -each attribute by generated name to see if the value tallies. Failure is -detected if the attribute name does not exist (meaning they were never -created) or if the wrong values were read back. -</p> - -<h1>A. Recursive Lock implementation code</h1> - -<blockquote> - <pre> - void H5_mutex_init(H5_mutex_t *H5_mutex) - { - H5_mutex->owner_thread = NULL; - pthread_mutex_init(&H5_mutex->atomic_lock, NULL); - pthread_cond_init(&H5_mutex->cond_var, NULL); - H5_mutex->lock_count = 0; - } - - void H5_mutex_lock(H5_mutex_t *H5_mutex) - { - pthread_mutex_lock(&H5_mutex->atomic_lock); - - if (pthread_equal(pthread_self(), H5_mutex->owner_thread)) { - /* already owned by self - increment count */ - H5_mutex->lock_count++; - } else { - if (H5_mutex->owner_thread == NULL) { - /* no one else has locked it - set owner and grab lock */ - H5_mutex->owner_thread = pthread_self(); - H5_mutex->lock_count = 1; - } else { - /* if already locked by someone else */ - while (1) { - pthread_cond_wait(&H5_mutex->cond_var, &H5_mutex->atomic_lock); - - if (H5_mutex->owner_thread == NULL) { - H5_mutex->owner_thread = pthread_self(); - H5_mutex->lock_count = 1; - break; - } /* else do nothing and loop back to wait on condition*/ - } - } - } - - pthread_mutex_unlock(&H5_mutex->atomic_lock); - } - - void H5_mutex_unlock(H5_mutex_t *H5_mutex) - { - pthread_mutex_lock(&H5_mutex->atomic_lock); - H5_mutex->lock_count--; - - if (H5_mutex->lock_count == 0) { - H5_mutex->owner_thread = NULL; - pthread_cond_signal(&H5_mutex->cond_var); - } - pthread_mutex_unlock(&H5_mutex->atomic_lock); - } - </pre> -</blockquote> - -<h1>B. First thread initialization</h1> - -<blockquote> - <pre> - void H5_first_thread_init(void) - { - /* initialize global API mutex lock */ - H5_g.H5_libinit_g = FALSE; - H5_g.init_lock.owner_thread = NULL; - pthread_mutex_init(&H5_g.init_lock.atomic_lock, NULL); - pthread_cond_init(&H5_g.init_lock.cond_var, NULL); - H5_g.init_lock.lock_count = 0; - - /* initialize key for thread-specific error stacks */ - pthread_key_create(&H5_errstk_key_g, NULL); - - /* initialize key for thread cancellability mechanism */ - pthread_key_create(&H5_cancel_key_g, NULL); - } - </pre> -</blockquote> - - -<h1>C. Per-thread error stack acquisition</h1> - -<blockquote> - <pre> - H5E_t *H5E_get_stack(void) - { - H5E_t *estack; - - if (estack = pthread_getspecific(H5_errstk_key_g)) { - return estack; - } else { - /* no associated value with current thread - create one */ - estack = (H5E_t *)malloc(sizeof(H5E_t)); - pthread_setspecific(H5_errstk_key_g, (void *)estack); - return estack; - } - } - </pre> -</blockquote> - -<h1>D. Thread cancellation mechanisms</h1> - -<blockquote> - <pre> - void H5_cancel_count_inc(void) - { - H5_cancel_t *cancel_counter; - - if (cancel_counter = pthread_getspecific(H5_cancel_key_g)) { - /* do nothing here */ - } else { - /* - * first time thread calls library - create new counter and - * associate with key - */ - cancel_counter = (H5_cancel_t *)malloc(sizeof(H5_cancel_t)); - cancel_counter->cancel_count = 0; - pthread_setspecific(H5_cancel_key_g, (void *)cancel_counter); - } - - if (cancel_counter->cancel_count == 0) { - /* thread entering library */ - pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, - &(cancel_counter->previous_state)); - } - - cancel_counter->cancel_count++; - } - - void H5_cancel_count_dec(void) - { - H5_cancel_t *cancel_counter = pthread_getspecific(H5_cancel_key_g); - - if (cancel_counter->cancel_count == 1) - pthread_setcancelstate(cancel_counter->previous_state, NULL); - - cancel_counter->cancel_count--; - } - </pre> -</blockquote> - -<h1>E. Macro expansion codes</h1> - -<h2>E.1 <code>FUNC_ENTER</code></h2> - -<blockquote> - <pre> - /* Initialize the library */ \ - H5_FIRST_THREAD_INIT \ - H5_API_UNSET_CANCEL \ - H5_API_LOCK_BEGIN \ - if (!(H5_INIT_GLOBAL)) { \ - H5_INIT_GLOBAL = TRUE; \ - if (H5_init_library() < 0) { \ - HRETURN_ERROR (H5E_FUNC, H5E_CANTINIT, err, \ - "library initialization failed"); \ - } \ - } \ - H5_API_LOCK_END \ - : - : - : - </pre> -</blockquote> - -<h2>E.2 <code>H5_FIRST_THREAD_INIT</code></h2> - -<blockquote> - <pre> - /* Macro for first thread initialization */ - #define H5_FIRST_THREAD_INIT \ - pthread_once(&H5_first_init_g, H5_first_thread_init); - </pre> -</blockquote> - - -<h2>E.3 <code>H5_API_UNSET_CANCEL</code></h2> - -<blockquote> - <pre> - #define H5_API_UNSET_CANCEL \ - if (H5_IS_API(FUNC)) { \ - H5_cancel_count_inc(); \ - } - </pre> -</blockquote> - - -<h2>E.4 <code>H5_API_LOCK_BEGIN</code></h2> - -<blockquote> - <pre> - #define H5_API_LOCK_BEGIN \ - if (H5_IS_API(FUNC)) { \ - H5_mutex_lock(&H5_g.init_lock); - </pre> -</blockquote> - - -<h2>E.5 <code>H5_API_LOCK_END</code></h2> - -<blockquote> - <pre> - #define H5_API_LOCK_END } - </pre> -</blockquote> - - -<h2>E.6 <code>HRETURN</code> and <code>HRETURN_ERROR</code></h2> - -<blockquote> - <pre> - : - : - H5_API_UNLOCK_BEGIN \ - H5_API_UNLOCK_END \ - H5_API_SET_CANCEL \ - return ret_val; \ - } - </pre> -</blockquote> - -<h2>E.7 <code>H5_API_UNLOCK_BEGIN</code></h2> - -<blockquote> - <pre> - #define H5_API_UNLOCK_BEGIN \ - if (H5_IS_API(FUNC)) { \ - H5_mutex_unlock(&H5_g.init_lock); - </pre> -</blockquote> - -<h2>E.8 <code>H5_API_UNLOCK_END</code></h2> - -<blockquote> - <pre> - #define H5_API_UNLOCK_END } - </pre> -</blockquote> - - -<h2>E.9 <code>H5_API_SET_CANCEL</code></h2> - -<blockquote> - <pre> - #define H5_API_SET_CANCEL \ - if (H5_IS_API(FUNC)) { \ - H5_cancel_count_dec(); \ - } - </pre> -</blockquote> - -<h2>By Chee Wai Lee</h2> -<h4>By Bill Wendling</h4> -<h4>27. October 2000</h4> - -</body> -</html> diff --git a/doc/html/TechNotes/VFL.html b/doc/html/TechNotes/VFL.html deleted file mode 100644 index 5674cdb..0000000 --- a/doc/html/TechNotes/VFL.html +++ /dev/null @@ -1,1543 +0,0 @@ -<HTML> -<HEAD> -<!-- This HTML file has been created by texi2html 1.51 - from VFL.texi on 18 November 1999 --> - -<TITLE>HDF5 Virtual File Layer</TITLE> -</HEAD> -<BODY> -<H1>HDF5</H1> -<H2>Virtual File Layer</H2> -<H2>Proposal 1999-08-11</H2> -<ADDRESS>Robb Matzke</ADDRESS> -<P> -<P><HR><P> -<H1>Table of Contents</H1> -<UL> -<LI><A NAME="TOC1" HREF="VFL.html#SEC1">Introduction</A> -<LI><A NAME="TOC2" HREF="VFL.html#SEC2">Using a File Driver</A> -<UL> -<LI><A NAME="TOC3" HREF="VFL.html#SEC3">Driver Header Files</A> -<LI><A NAME="TOC4" HREF="VFL.html#SEC4">Creating and Opening Files</A> -<LI><A NAME="TOC5" HREF="VFL.html#SEC5">Performing I/O</A> -<LI><A NAME="TOC6" HREF="VFL.html#SEC6">File Driver Interchangeability</A> -</UL> -<LI><A NAME="TOC7" HREF="VFL.html#SEC7">Implementation of a Driver</A> -<UL> -<LI><A NAME="TOC8" HREF="VFL.html#SEC8">Mode Functions</A> -<LI><A NAME="TOC9" HREF="VFL.html#SEC9">File Functions</A> -<UL> -<LI><A NAME="TOC10" HREF="VFL.html#SEC10">Opening Files</A> -<LI><A NAME="TOC11" HREF="VFL.html#SEC11">Closing Files</A> -<LI><A NAME="TOC12" HREF="VFL.html#SEC12">File Keys</A> -<LI><A NAME="TOC13" HREF="VFL.html#SEC13">Saving Modes Across Opens</A> -</UL> -<LI><A NAME="TOC14" HREF="VFL.html#SEC14">Address Space Functions</A> -<UL> -<LI><A NAME="TOC15" HREF="VFL.html#SEC15">Userblock and Superblock</A> -<LI><A NAME="TOC16" HREF="VFL.html#SEC16">Allocation of Format Regions</A> -<LI><A NAME="TOC17" HREF="VFL.html#SEC17">Freeing Format Regions</A> -<LI><A NAME="TOC18" HREF="VFL.html#SEC18">Querying Address Range</A> -</UL> -<LI><A NAME="TOC19" HREF="VFL.html#SEC19">Data Functions</A> -<UL> -<LI><A NAME="TOC20" HREF="VFL.html#SEC20">Contiguous I/O Functions</A> -<LI><A NAME="TOC21" HREF="VFL.html#SEC21">Flushing Cached Data</A> -</UL> -<LI><A NAME="TOC22" HREF="VFL.html#SEC22">Optimization Functions</A> -<LI><A NAME="TOC23" HREF="VFL.html#SEC23">Registration of a Driver</A> -<LI><A NAME="TOC24" HREF="VFL.html#SEC24">Querying Driver Information</A> -</UL> -<LI><A NAME="TOC25" HREF="VFL.html#SEC25">Miscellaneous</A> -</UL> -<P><HR><P> - - -<H1><A NAME="SEC1" HREF="VFL.html#TOC1">Introduction</A></H1> - -<P> -The HDF5 file format describes how HDF5 data structures and dataset raw -data are mapped to a linear <STRONG>format address space</STRONG> and the HDF5 -library implements that bidirectional mapping in terms of an -API. However, the HDF5 format specifications do <EM>not</EM> indicate how -the format address space is mapped onto storage and HDF (version 5 and -earlier) simply mapped the format address space directly onto a single -file by convention. - -</P> -<P> -Since early versions of HDF5 it became apparent that users want the ability to -map the format address space onto different types of storage (a single file, -multiple files, local memory, global memory, network distributed global -memory, a network protocol, <I>etc</I>.) with various types of maps. For -instance, some users want to be able to handle very large format address -spaces on operating systems that support only 2GB files by partitioning the -format address space into equal-sized parts each served by a separate -file. Other users want the same multi-file storage capability but want to -partition the address space according to purpose (raw data in one file, object -headers in another, global heap in a third, <I>etc.</I>) in order to improve I/O -speeds. - -</P> -<P> -In fact, the number of storage variations is probably larger than the -number of methods that the HDF5 team is capable of implementing and -supporting. Therefore, a <STRONG>Virtual File Layer</STRONG> API is being -implemented which will allow application teams or departments to design -and implement their own mapping between the HDF5 format address space -and storage, with each mapping being a separate <STRONG>file driver</STRONG> -(possibly written in terms of other file drivers). The HDF5 team will -provide a small set of useful file drivers which will also serve as -examples for those who which to write their own: - -</P> -<DL COMPACT> - -<DT><CODE>H5FD_SEC2</CODE> -<DD> -This is the default driver which uses Posix file-system functions like -<CODE>read</CODE> and <CODE>write</CODE> to perform I/O to a single file. All I/O -requests are unbuffered although the driver does optimize file seeking -operations to some extent. - -<DT><CODE>H5FD_STDIO</CODE> -<DD> -This driver uses functions from <TT>`stdio.h'</TT> to perform buffered I/O -to a single file. - -<DT><CODE>H5FD_CORE</CODE> -<DD> -This driver performs I/O directly to memory and can be used to create small -temporary files that never exist on permanent storage. This type of storage is -generally very fast since the I/O consists only of memory-to-memory copy -operations. - -<DT><CODE>H5FD_MPIIO</CODE> -<DD> -This is the driver of choice for accessing files in parallel using MPI and -MPI-IO. It is only predefined if the library is compiled with parallel I/O -support. - -<DT><CODE>H5FD_FAMILY</CODE> -<DD> -Large format address spaces are partitioned into more manageable pieces and -sent to separate storage locations using an underlying driver of the user's -choice. The <CODE>h5repart</CODE> tool can be used to change the sizes of the -family members when stored as files or to convert a family of files to a -single file or vice versa. - -<DT><CODE>H5FD_SPLIT</CODE> -<DD> -The format address space is split into meta data and raw data and each is -mapped onto separate storage using underlying drivers of the user's -choice. The meta data storage can be read by itself (for limited -functionality) or both files can be accessed together. -</DL> - - - -<H1><A NAME="SEC2" HREF="VFL.html#TOC2">Using a File Driver</A></H1> - -<P> -Most application writers will use a driver defined by the HDF5 library or -contributed by another programming team. This chapter describes how existing -drivers are used. - -</P> - - - -<H2><A NAME="SEC3" HREF="VFL.html#TOC3">Driver Header Files</A></H2> - -<P> -Each file driver is defined in its own public header file which should -be included by any application which plans to use that driver. The -predefined drivers are in header files whose names begin with -<SAMP>`H5FD'</SAMP> followed by the driver name and <SAMP>`.h'</SAMP>. The <TT>`hdf5.h'</TT> -header file includes all the predefined driver header files. - -</P> -<P> -Once the appropriate header file is included a symbol of the form -<SAMP>`H5FD_'</SAMP> followed by the upper-case driver name will be the driver -identification number.<A NAME="DOCF1" HREF="VFL.html#FOOT1">(1)</A> However, the -value may change if the library is closed (<I>e.g.</I>, by calling -<CODE>H5close</CODE>) and the symbol is referenced again. - -</P> - - -<H2><A NAME="SEC4" HREF="VFL.html#TOC4">Creating and Opening Files</A></H2> - -<P> -In order to create or open a file one must define the method by which the -storage is accessed<A NAME="DOCF2" HREF="VFL.html#FOOT2">(2)</A> and does so by creating a file access property list<A NAME="DOCF3" HREF="VFL.html#FOOT3">(3)</A> which is passed to the <CODE>H5Fcreate</CODE> or -<CODE>H5Fopen</CODE> function. A default file access property list is created by -calling <CODE>H5Pcreate</CODE> and then the file driver information is inserted by -calling a driver initialization function such as <CODE>H5Pset_fapl_family</CODE>: - -</P> - -<PRE> -hid_t fapl = H5Pcreate(H5P_FILE_ACCESS); -size_t member_size = 100*1024*1024; /*100MB*/ -H5Pset_fapl_family(fapl, member_size, H5P_DEFAULT); -hid_t file = H5Fcreate("foo%05d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl); -H5Pclose(fapl); -</PRE> - -<P> -Each file driver will have its own initialization function -whose name is <CODE>H5Pset_fapl_</CODE> followed by the driver name and which -takes a file access property list as the first argument followed by -additional driver-dependent arguments. - -</P> -<P> -An alternative to using the driver initialization function is to set the -driver directly using the <CODE>H5Pset_driver</CODE> function.<A NAME="DOCF4" HREF="VFL.html#FOOT4">(4)</A> Its second argument is the file driver identifier, which may -have a different numeric value from run to run depending on the order in which -the file drivers are registered with the library. The third argument -encapsulates the additional arguments of the driver initialization -function. This method only works if the file driver writer has made the -driver-specific property list structure a public datatype, which is -often not the case. - -</P> - -<PRE> -hid_t fapl = H5Pcreate(H5P_FILE_ACCESS); -static H5FD_family_fapl_t fa = {100*1024*1024, H5P_DEFAULT}; -H5Pset_driver(fapl, H5FD_FAMILY, &fa); -hid_t file = H5Fcreate("foo.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl); -H5Pclose(fapl); -</PRE> - -<P> -It is also possible to query the file driver information from a file access -property list by calling <CODE>H5Pget_driver</CODE> to determine the driver and then -calling a driver-defined query function to obtain the driver information: - -</P> - -<PRE> -hid_t driver = H5Pget_driver(fapl); -if (H5FD_SEC2==driver) { - /*nothing further to get*/ -} else if (H5FD_FAMILY==driver) { - hid_t member_fapl; - haddr_t member_size; - H5Pget_fapl_family(fapl, &member_size, &member_fapl); -} else if (....) { - .... -} -</PRE> - - - -<H2><A NAME="SEC5" HREF="VFL.html#TOC5">Performing I/O</A></H2> - -<P> -The <CODE>H5Dread</CODE> and <CODE>H5Dwrite</CODE> functions transfer data between -application memory and the file. They both take an optional data transfer -property list which has some general driver-independent properties and -optional driver-defined properties. An application will typically perform I/O -in one of three styles via the <CODE>H5Dread</CODE> or <CODE>H5Dwrite</CODE> function: - -</P> -<P> -Like file access properties in the previous section, data transfer properties -can be set using a driver initialization function or a general purpose -function. For example, to set the MPI-IO driver to use independent access for -I/O operations one would say: - -</P> - -<PRE> -hid_t dxpl = H5Pcreate(H5P_DATA_XFER); -H5Pset_dxpl_mpio(dxpl, H5FD_MPIO_INDEPENDENT); -H5Dread(dataset, type, mspace, fspace, buffer, dxpl); -H5Pclose(dxpl); -</PRE> - -<P> -The alternative is to initialize a driver defined C <CODE>struct</CODE> and pass it -to the <CODE>H5Pset_driver</CODE> function: - -</P> - -<PRE> -hid_t dxpl = H5Pcreate(H5P_DATA_XFER); -static H5FD_mpio_dxpl_t dx = {H5FD_MPIO_INDEPENDENT}; -H5Pset_driver(dxpl, H5FD_MPIO, &dx); -H5Dread(dataset, type, mspace, fspace, buffer, dxpl); -</PRE> - -<P> -The transfer propery list can be queried in a manner similar to the file -access property list: the driver provides a function (or functions) to return -various information about the transfer property list: - -</P> - -<PRE> -hid_t driver = H5Pget_driver(dxpl); -if (H5FD_MPIO==driver) { - H5FD_mpio_xfer_t xfer_mode; - H5Pget_dxpl_mpio(dxpl, &xfer_mode); -} else { - .... -} -</PRE> - - - -<H2><A NAME="SEC6" HREF="VFL.html#TOC6">File Driver Interchangeability</A></H2> - -<P> -The HDF5 specifications describe two things: the mapping of data onto a linear -<STRONG>format address space</STRONG> and the C API which performs the mapping. -However, the mapping of the format address space onto storage intentionally -falls outside the scope of the HDF5 specs. This is a direct result of the fact -that it is not generally possible to store information about how to access -storage inside the storage itself. For instance, given only the file name -<TT>`/arborea/1225/work/f%03d'</TT> the HDF5 library is unable to tell whether the -name refers to a file on the local file system, a family of files on the local -file system, a file on host <SAMP>`arborea'</SAMP> port 1225, a family of files on a -remote system, <I>etc</I>. - -</P> -<P> -Two ways which library could figure out where the storage is located are: -storage access information can be provided by the user, or the library can try -all known file access methods. This implementation uses the former method. - -</P> -<P> -In general, if a file was created with one driver then it isn't possible to -open it with another driver. There are of course exceptions: a file created -with MPIO could probably be opened with the sec2 driver, any file created -by the sec2 driver could be opened as a family of files with one member, -<I>etc</I>. In fact, sometimes a file must not only be opened with the same -driver but also with the same driver properties. The predefined drivers are -written in such a way that specifying the correct driver is sufficient for -opening a file. - -</P> - - -<H1><A NAME="SEC7" HREF="VFL.html#TOC7">Implementation of a Driver</A></H1> - -<P> -A driver is simply a collection of functions and data structures which are -registered with the HDF5 library at runtime. The functions fall into these -categories: - -</P> - -<UL> -<LI>Functions which operate on modes - -<LI>Functions which operate on files - -<LI>Functions which operate on the address space - -<LI>Functions which operate on data - -<LI>Functions for driver initialization - -<LI>Optimization functions - -</UL> - - - -<H2><A NAME="SEC8" HREF="VFL.html#TOC8">Mode Functions</A></H2> - -<P> -Some drivers need information about file access and data transfers which are -very specific to the driver. The information is usually implemented as a pair -of pointers to C structs which are allocated and initialized as part of an -HDF5 property list and passed down to various driver functions. There are two -classes of settings: file access modes that describe how to access the file -through the driver, and data transfer modes which are settings that control -I/O operations. Each file opened by a particular driver may have a different -access mode; each dataset I/O request for a particular file may have a -different data transfer mode. - -</P> -<P> -Since each driver has its own particular requirements for various settings, -each driver is responsible for defining the mode structures that it -needs. Higher layers of the library treat the structures as opaque but must be -able to copy and free them. Thus, the driver provides either the size of the -structure or a pair of function pointers for each of the mode types. - -</P> -<P> -<STRONG>Example:</STRONG> The family driver needs to know how the format address -space is partitioned and the file access property list to use for the -family members. - -</P> - -<PRE> -/* Driver-specific file access properties */ -typedef struct H5FD_family_fapl_t { - hsize_t memb_size; /*size of each member */ - hid_t memb_fapl_id; /*file access property list of each memb*/ -} H5FD_family_fapl_t; - -/* Driver specific data transfer properties */ -typedef struct H5FD_family_dxpl_t { - hid_t memb_dxpl_id; /*data xfer property list of each memb */ -} H5FD_family_dxpl_t; -</PRE> - -<P> -In order to copy or free one of these structures the member file access -or data transfer properties must also be copied or freed. This is done -by providing a copy and close function for each structure: - -</P> -<P> -<STRONG>Example:</STRONG> The file access property list copy and close functions -for the family driver: - -</P> - -<PRE> -static void * -H5FD_family_fapl_copy(const void *_old_fa) -{ - const H5FD_family_fapl_t *old_fa = (const H5FD_family_fapl_t*)_old_fa; - H5FD_family_fapl_t *new_fa = malloc(sizeof(H5FD_family_fapl_t)); - assert(new_fa); - - memcpy(new_fa, old_fa, sizeof(H5FD_family_fapl_t)); - new_fa->memb_fapl_id = H5Pcopy(old_fa->memb_fapl_id); - return new_fa; -} - -static herr_t -H5FD_family_fapl_free(void *_fa) -{ - H5FD_family_fapl_t *fa = (H5FD_family_fapl_t*)_fa; - H5Pclose(fa->memb_fapl_id); - free(fa); - return 0; -} -</PRE> - -<P> -Generally when a file is created or opened the file access properties -for the driver are copied into the file pointer which is returned and -they may be modified from their original value (for instance, the file -family driver modifies the member size property when opening an existing -family). In order to support the <CODE>H5Fget_access_plist</CODE> function the -driver must provide a <CODE>fapl_get</CODE> callback which creates a copy of -the driver-specific properties based on a particular file. - -</P> -<P> -<STRONG>Example:</STRONG> The file family driver copies the member size file -access property list into the return value: - -</P> - -<PRE> -static void * -H5FD_family_fapl_get(H5FD_t *_file) -{ - H5FD_family_t *file = (H5FD_family_t*)_file; - H5FD_family_fapl_t *fa = calloc(1, sizeof(H5FD_family_fapl_t*)); - - fa->memb_size = file->memb_size; - fa->memb_fapl_id = H5Pcopy(file->memb_fapl_id); - return fa; -} -</PRE> - - - -<H2><A NAME="SEC9" HREF="VFL.html#TOC9">File Functions</A></H2> - -<P> -The higher layers of the library expect files to have a name and allow the -file to be accessed in various modes. The driver must be able to create a new -file, replace an existing file, or open an existing file. Opening or creating -a file should return a handle, a pointer to a specialization of the -<CODE>H5FD_t</CODE> struct, which allows read-only or read-write access and which -will be passed to the other driver functions as they are -called.<A NAME="DOCF5" HREF="VFL.html#FOOT5">(5)</A> - -</P> - -<PRE> -typedef struct { - /* Public fields */ - H5FD_class_t *cls; /*class data defined below*/ - - /* Private fields -- driver-defined */ - -} H5FD_t; -</PRE> - -<P> -<STRONG>Example:</STRONG> The family driver requires handles to the underlying -storage, the size of the members for this particular file (which might be -different than the member size specified in the file access property list if -an existing file family is being opened), the name used to open the file in -case additional members must be created, and the flags to use for creating -those additional members. The <CODE>eoa</CODE> member caches the size of the format -address space so the family members don't have to be queried in order to find -it. - -</P> - -<PRE> -/* The description of a file belonging to this driver. */ -typedef struct H5FD_family_t { - H5FD_t pub; /*public stuff, must be first */ - hid_t memb_fapl_id; /*file access property list for members */ - hsize_t memb_size; /*maximum size of each member file */ - int nmembs; /*number of family members */ - int amembs; /*number of member slots allocated */ - H5FD_t **memb; /*dynamic array of member pointers */ - haddr_t eoa; /*end of allocated addresses */ - char *name; /*name generator printf format */ - unsigned flags; /*flags for opening additional members */ -} H5FD_family_t; -</PRE> - -<P> -<STRONG>Example:</STRONG> The sec2 driver needs to keep track of the underlying Unix -file descriptor and also the end of format address space and current Unix file -size. It also keeps track of the current file position and last operation -(read, write, or unknown) in order to optimize calls to <CODE>lseek</CODE>. The -<CODE>device</CODE> and <CODE>inode</CODE> fields are defined on Unix in order to uniquely -identify the file and will be discussed below. - -</P> - -<PRE> -typedef struct H5FD_sec2_t { - H5FD_t pub; /*public stuff, must be first */ - int fd; /*the unix file */ - haddr_t eoa; /*end of allocated region */ - haddr_t eof; /*end of file; current file size*/ - haddr_t pos; /*current file I/O position */ - int op; /*last operation */ - dev_t device; /*file device number */ - ino_t inode; /*file i-node number */ -} H5FD_sec2_t; -</PRE> - - - -<H3><A NAME="SEC10" HREF="VFL.html#TOC10">Opening Files</A></H3> - -<P> -All drivers must define a function for opening/creating a file. This -function should have a prototype which is: - -</P> -<P> -<DL> -<DT><U>Function:</U> static H5FD_t * <B>open</B> <I>(const char *<VAR>name</VAR>, unsigned <VAR>flags</VAR>, hid_t <VAR>fapl</VAR>, haddr_t <VAR>maxaddr</VAR>)</I> -<DD><A NAME="IDX1"></A> - -</P> -<P> -The file name <VAR>name</VAR> and file access property list <VAR>fapl</VAR> are -the same as were specified in the <CODE>H5Fcreate</CODE> or <CODE>H5Fopen</CODE> -call. The <VAR>flags</VAR> are the same as in those calls also except the -flag <CODE>H5F_ACC_CREATE</CODE> is also present if the call was to -<CODE>H5Fcreate</CODE> and they are documented in the <TT>`H5Fpublic.h'</TT> -file. The <VAR>maxaddr</VAR> argument is the maximum format address that the -driver should be prepared to handle (the minimum address is always -zero). -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver opens a Unix file with the requested name -and saves information which uniquely identifies the file (the Unix device -number and inode). - -</P> - -<PRE> -static H5FD_t * -H5FD_sec2_open(const char *name, unsigned flags, hid_t fapl_id/*unused*/, - haddr_t maxaddr) -{ - unsigned o_flags; - int fd; - struct stat sb; - H5FD_sec2_t *file=NULL; - - /* Check arguments */ - if (!name || !*name) return NULL; - if (0==maxaddr || HADDR_UNDEF==maxaddr) return NULL; - if (ADDR_OVERFLOW(maxaddr)) return NULL; - - /* Build the open flags */ - o_flags = (H5F_ACC_RDWR & flags) ? O_RDWR : O_RDONLY; - if (H5F_ACC_TRUNC & flags) o_flags |= O_TRUNC; - if (H5F_ACC_CREAT & flags) o_flags |= O_CREAT; - if (H5F_ACC_EXCL & flags) o_flags |= O_EXCL; - - /* Open the file */ - if ((fd=open(name, o_flags, 0666))<0) return NULL; - if (fstat(fd, &sb)<0) { - close(fd); - return NULL; - } - - /* Create the new file struct */ - file = calloc(1, sizeof(H5FD_sec2_t)); - file->fd = fd; - file->eof = sb.st_size; - file->pos = HADDR_UNDEF; - file->op = OP_UNKNOWN; - file->device = sb.st_dev; - file->inode = sb.st_ino; - - return (H5FD_t*)file; -} -</PRE> - - - -<H3><A NAME="SEC11" HREF="VFL.html#TOC11">Closing Files</A></H3> - -<P> -Closing a file simply means that all cached data should be flushed to the next -lower layer, the file should be closed at the next lower layer, and all -file-related data structures should be freed. All information needed by the -close function is already present in the file handle. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>close</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX2"></A> - -</P> -<P> -The <VAR>file</VAR> argument is the handle which was returned by the <CODE>open</CODE> -function, and the <CODE>close</CODE> should free only memory associated with the -driver-specific part of the handle (the public parts will have already been released by HDF5's virtual file layer). -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver just closes the underlying Unix file, -making sure that the actual file size is the same as that known to the -library by writing a zero to the last file position it hasn't been -written by some previous operation (which happens in the same code which -flushes the file contents and is shown below). - -</P> - -<PRE> -static herr_t -H5FD_sec2_close(H5FD_t *_file) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - - if (H5FD_sec2_flush(_file)<0) return -1; - if (close(file->fd)<0) return -1; - free(file); - return 0; -} -</PRE> - - - -<H3><A NAME="SEC12" HREF="VFL.html#TOC12">File Keys</A></H3> - -<P> -Occasionally an application will attempt to open a single file more than one -time in order to obtain multiple handles to the file. HDF5 allows the files to -share information<A NAME="DOCF6" HREF="VFL.html#FOOT6">(6)</A> but in order to -accomplish this HDF5 must be able to tell when two names refer to the same -file. It does this by associating a driver-defined key with each file opened -by a driver and comparing the key for an open request with the keys for all -other files currently open by the same driver. - -</P> -<P> -<DL> -<DT><U>Function:</U> const int <B>cmp</B> <I>(const H5FD_t *<VAR>f1</VAR>, const H5FD_t *<VAR>f2</VAR>)</I> -<DD><A NAME="IDX3"></A> - -</P> -<P> -The driver may provide a function which compares two files <VAR>f1</VAR> and -<VAR>f2</VAR> belonging to the same driver and returns a negative, positive, or -zero value <I>a la</I> the <CODE>strcmp</CODE> function.<A NAME="DOCF7" HREF="VFL.html#FOOT7">(7)</A> If this -function is not provided then HDF5 assumes that all calls to the <CODE>open</CODE> -callback return unique files regardless of the arguments and it is up to the -application to avoid doing this if that assumption is incorrect. -</DL> - -</P> -<P> -Each time a file is opened the library calls the <CODE>cmp</CODE> function to -compare that file with all other files currently open by the same driver and -if one of them matches (at most one can match) then the file which was just -opened is closed and the previously opened file is used instead. - -</P> -<P> -Opening a file twice with incompatible flags will result in failure. For -instance, opening a file with the truncate flag is a two step process which -first opens the file without truncation so keys can be compared, and if no -matching file is found already open then the file is closed and immediately -reopened with the truncation flag set (if a matching file is already open then -the truncating open will fail). - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver uses the Unix device and i-node as the -key. They were initialized when the file was opened. - -</P> - -<PRE> -static int -H5FD_sec2_cmp(const H5FD_t *_f1, const H5FD_t *_f2) -{ - const H5FD_sec2_t *f1 = (const H5FD_sec2_t*)_f1; - const H5FD_sec2_t *f2 = (const H5FD_sec2_t*)_f2; - - if (f1->device < f2->device) return -1; - if (f1->device > f2->device) return 1; - - if (f1->inode < f2->inode) return -1; - if (f1->inode > f2->inode) return 1; - - return 0; -} -</PRE> - - - -<H3><A NAME="SEC13" HREF="VFL.html#TOC13">Saving Modes Across Opens</A></H3> - -<P> -Some drivers may also need to store certain information in the file superblock -in order to be able to reliably open the file at a later date. This is done by -three functions: one to determine how much space will be necessary to store -the information in the superblock, one to encode the information, and one to -decode the information. These functions are optional, but if any one is -defined then the other two must also be defined. - -</P> -<P> -<DL> -<DT><U>Function:</U> static hsize_t <B>sb_size</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX4"></A> -<DT><U>Function:</U> static herr_t <B>sb_encode</B> <I>(H5FD_t *<VAR>file</VAR>, char *<VAR>name</VAR>, unsigned char *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX5"></A> -<DT><U>Function:</U> static herr_t <B>sb_decode</B> <I>(H5FD_t *<VAR>file</VAR>, const char *<VAR>name</VAR>, const unsigned char *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX6"></A> - -</P> -<P> -The <CODE>sb_size</CODE> function returns the number of bytes necessary to encode -information needed later if the file is reopened. The <CODE>sb_encode</CODE> -function encodes information from the file into buffer <VAR>buf</VAR> -allocated by the caller. It also writes an 8-character (plus null -termination) into the <CODE>name</CODE> argument, which should be a unique -identification for the driver. The <CODE>sb_decode</CODE> function looks at -the <VAR>name</VAR> - -</P> -<P> - decodes -data from the buffer <VAR>buf</VAR> and updates the <VAR>file</VAR> argument with the new information, -advancing <VAR>*p</VAR> in the process. -</DL> - -</P> -<P> -The part of this which is somewhat tricky is that the file must be readable -before the superblock information is decoded. File access modes fall outside -the scope of the HDF5 file format, but they are placed inside the boot block -for convenience.<A NAME="DOCF8" HREF="VFL.html#FOOT8">(8)</A> - -</P> -<P> -<STRONG>Example:</STRONG> <EM>To be written later.</EM> - -</P> - - -<H2><A NAME="SEC14" HREF="VFL.html#TOC14">Address Space Functions</A></H2> - -<P> -HDF5 does not assume that a file is a linear address space of bytes. Instead, -the library will call functions to allocate and free portions of the HDF5 -format address space, which in turn map onto functions in the file driver to -allocate and free portions of file address space. The library tells the file -driver how much format address space it wants to allocate and the driver -decides what format address to use and how that format address is mapped onto -the file address space. Usually the format address is chosen so that the file -address can be calculated in constant time for data I/O operations (which are -always specified by format addresses). - -</P> - - - -<H3><A NAME="SEC15" HREF="VFL.html#TOC15">Userblock and Superblock</A></H3> - -<P> -The HDF5 format allows an optional userblock to appear before the actual HDF5 -data in such a way that if the userblock is <STRONG>sucked out</STRONG> of the file and -everything remaining is shifted downward in the file address space, then the -file is still a valid HDF5 file. The userblock size can be zero or any -multiple of two greater than or equal to 512 and the file superblock begins -immediately after the userblock. - -</P> -<P> -HDF5 allocates space for the userblock and superblock by calling an -allocation function defined below, which must return a chunk of memory at -format address zero on the first call. - -</P> - - -<H3><A NAME="SEC16" HREF="VFL.html#TOC16">Allocation of Format Regions</A></H3> - -<P> -The library makes many types of allocation requests: - -</P> -<DL COMPACT> - -<DT><CODE>H5FD_MEM_SUPER</CODE> -<DD> -An allocation request for the userblock and/or superblock. -<DT><CODE>H5FD_MEM_BTREE</CODE> -<DD> -An allocation request for a node of a B-tree. -<DT><CODE>H5FD_MEM_DRAW</CODE> -<DD> -An allocation request for the raw data of a dataset. -<DT><CODE>H5FD_MEM_META</CODE> -<DD> -An allocation request for the raw data of a dataset which -the user has indicated will be relatively small. -<DT><CODE>H5FD_MEM_GROUP</CODE> -<DD> -An allocation request for a group leaf node (internal nodes of the group tree -are allocated as H5MF_BTREE). -<DT><CODE>H5FD_MEM_GHEAP</CODE> -<DD> -An allocation request for a global heap collection. Global heaps are used to -store certain types of references such as dataset region references. The set -of all global heap collections can become quite large. -<DT><CODE>H5FD_MEM_LHEAP</CODE> -<DD> -An allocation request for a local heap. Local heaps are used to store the -names which are members of a group. The combined size of all local heaps is a -function of the number of object names in the file. -<DT><CODE>H5FD_MEM_OHDR</CODE> -<DD> -An allocation request for (part of) an object header. Object headers are -relatively small and include meta information about objects (like the data -space and type of a dataset) and attributes. -</DL> - -<P> -When a chunk of memory is freed the library adds it to a free list and -allocation requests are satisfied from the free list before requesting memory -from the file driver. Each type of allocation request enumerated above has its -own free list, but the file driver can specify that certain object types can -share a free list. It does so by providing an array which maps a request type -to a free list. If any value of the map is <CODE>H5MF_DEFAULT</CODE> (zero) then the -object's own free list is used. The special value <CODE>H5MF_NOLIST</CODE> indicates -that the library should not attempt to maintain a free list for that -particular object type, instead calling the file driver each time an object of -that type is freed. - -</P> -<P> -Mappings predefined in the <TT>`H5FDpublic.h'</TT> file are: -<DL COMPACT> - -<DT><CODE>H5FD_FLMAP_SINGLE</CODE> -<DD> -All memory usage types are mapped to a single free list. -<DT><CODE>H5FD_FLMAP_DICHOTOMY</CODE> -<DD> -Memory usage is segregated into meta data and raw data for the purposes of -memory management. -<DT><CODE>H5FD_FLMAP_DEFAULT</CODE> -<DD> -Each memory usage type has its own free list. -</DL> - -<P> -<STRONG>Example:</STRONG> To make a map that manages object headers on one free list -and everything else on another free list one might initialize the map with the -following code: (the use of <CODE>H5FD_MEM_SUPER</CODE> is arbitrary) - -</P> - -<PRE> -H5FD_mem_t mt, map[H5FD_MEM_NTYPES]; - -for (mt=0; mt<H5FD_MEM_NTYPES; mt++) { - map[mt] = (H5FD_MEM_OHDR==mt) ? mt : H5FD_MEM_SUPER; -} -</PRE> - -<P> -If an allocation request cannot be satisfied from the free list then one of -two things happen. If the driver defines an allocation callback then it is -used to allocate space; otherwise new memory is allocated from the end of the -format address space by incrementing the end-of-address marker. - -</P> -<P> -<DL> -<DT><U>Function:</U> static haddr_t <B>alloc</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, hsize_t <VAR>size</VAR>)</I> -<DD><A NAME="IDX7"></A> - -</P> -<P> -The <VAR>file</VAR> argument is the file from which space is to be allocated, -<VAR>type</VAR> is the type of memory being requested (from the list above) without -being mapped according to the freelist map and <VAR>size</VAR> is the number of -bytes being requested. The library is allowed to allocate large chunks of -storage and manage them in a layer above the file driver (although the current -library doesn't do that). The allocation function should return a format -address for the first byte allocated. The allocated region extends from that -address for <VAR>size</VAR> bytes. If the request cannot be honored then the -undefined address value is returned (<CODE>HADDR_UNDEF</CODE>). The first call to -this function for a file which has never had memory allocated <EM>must</EM> -return a format address of zero or <CODE>HADDR_UNDEF</CODE> since this is how the -library allocates space for the userblock and/or superblock. -</DL> - -</P> - -<P> -<STRONG>Example:</STRONG> <EM>To be written later.</EM> - -</P> - - -<H3><A NAME="SEC17" HREF="VFL.html#TOC17">Freeing Format Regions</A></H3> - -<P> -When the library is finished using a certain region of the format address -space it will return the space to the free list according to the type of -memory being freed and the free list map described above. If the free list has -been disabled for a particular memory usage type (according to the free list -map) and the driver defines a <CODE>free</CODE> callback then it will be -invoked. The <CODE>free</CODE> callback is also invoked for all entries on the free -list when the file is closed. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>free</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>)</I> -<DD><A NAME="IDX8"></A> - -</P> -<P> -The <VAR>file</VAR> argument is the file for which space is being freed; <VAR>type</VAR> -is the type of object being freed (from the list above) without being mapped -according to the freelist map; <VAR>addr</VAR> is the first format address to free; -and <VAR>size</VAR> is the size in bytes of the region being freed. The region -being freed may refer to just part of the region originally allocated and/or -may cross allocation boundaries provided all regions being freed have the same -usage type. However, the library will never attempt to free regions which have -already been freed or which have never been allocated. -</DL> - -</P> -<P> -A driver may choose to not define the <CODE>free</CODE> function, in which case -format addresses will be leaked. This isn't normally a huge problem since the -library contains a simple free list of its own and freeing parts of the format -address space is not a common occurrence. - -</P> -<P> -<STRONG>Example:</STRONG> <EM>To be written later.</EM> - -</P> - - -<H3><A NAME="SEC18" HREF="VFL.html#TOC18">Querying Address Range</A></H3> - -<P> -Each file driver must have some mechanism for setting and querying the end of -address, or <STRONG>EOA</STRONG>, marker. The EOA marker is the first format address -after the last format address ever allocated. If the last part of the -allocated address range is freed then the driver may optionally decrease the -eoa marker. - -</P> -<P> -<DL> -<DT><U>Function:</U> static haddr_t <B>get_eoa</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX9"></A> - -</P> -<P> -This function returns the current value of the EOA marker for the specified -file. -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver just returns the current eoa marker value -which is cached in the file structure: - -</P> - -<PRE> -static haddr_t -H5FD_sec2_get_eoa(H5FD_t *_file) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - return file->eoa; -} -</PRE> - -<P> -The eoa marker is initially zero when a file is opened and the library may set -it to some other value shortly after the file is opened (after the superblock -is read and the saved eoa marker is determined) or when allocating additional -memory in the absence of an <CODE>alloc</CODE> callback (described above). - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver simply caches the eoa marker in the file -structure and does not extend the underlying Unix file. When the file is -flushed or closed then the Unix file size is extended to match the eoa marker. - -</P> - -<PRE> -static herr_t -H5FD_sec2_set_eoa(H5FD_t *_file, haddr_t addr) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - file->eoa = addr; - return 0; -} -</PRE> - - - -<H2><A NAME="SEC19" HREF="VFL.html#TOC19">Data Functions</A></H2> - -<P> -These functions operate on data, transferring a region of the format address -space between memory and files. - -</P> - - - -<H3><A NAME="SEC20" HREF="VFL.html#TOC20">Contiguous I/O Functions</A></H3> - -<P> -A driver must specify two functions to transfer data from the library to the -file and vice versa. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>read</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, void *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX10"></A> -<DT><U>Function:</U> static herr_t <B>write</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, const void *<VAR>buf</VAR>)</I> -<DD><A NAME="IDX11"></A> - -</P> -<P> -The <CODE>read</CODE> function reads data from file <VAR>file</VAR> beginning at address -<VAR>addr</VAR> and continuing for <VAR>size</VAR> bytes into the buffer <VAR>buf</VAR> -supplied by the caller. The <CODE>write</CODE> function transfers data in the -opposite direction. Both functions take a data transfer property list -<VAR>dxpl</VAR> which indicates the fine points of how the data is to be -transferred and which comes directly from the <CODE>H5Dread</CODE> or -<CODE>H5Dwrite</CODE> function. Both functions receive <VAR>type</VAR> of -data being written, which may allow a driver to tune it's behavior for -different kinds of data. -</DL> - -</P> -<P> -Both functions should return a negative value if they fail to transfer the -requested data, or non-negative if they succeed. The library will never -attempt to read from unallocated regions of the format address space. - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver just makes system calls. It tries not to -call <CODE>lseek</CODE> if the current operation is the same as the previous -operation and the file position is correct. It also fills the output buffer -with zeros when reading between the current EOF and EOA markers and restarts -system calls which were interrupted. - -</P> - -<PRE> -static herr_t -H5FD_sec2_read(H5FD_t *_file, H5FD_mem_t type/*unused*/, hid_t dxpl_id/*unused*/, - haddr_t addr, hsize_t size, void *buf/*out*/) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - ssize_t nbytes; - - assert(file && file->pub.cls); - assert(buf); - - /* Check for overflow conditions */ - if (REGION_OVERFLOW(addr, size)) return -1; - if (addr+size>file->eoa) return -1; - - /* Seek to the correct location */ - if ((addr!=file->pos || OP_READ!=file->op) && - file_seek(file->fd, (file_offset_t)addr, SEEK_SET)<0) { - file->pos = HADDR_UNDEF; - file->op = OP_UNKNOWN; - return -1; - } - - /* - * Read data, being careful of interrupted system calls, partial results, - * and the end of the file. - */ - while (size>0) { - do nbytes = read(file->fd, buf, size); - while (-1==nbytes && EINTR==errno); - if (-1==nbytes) { - /* error */ - file->pos = HADDR_UNDEF; - file->op = OP_UNKNOWN; - return -1; - } - if (0==nbytes) { - /* end of file but not end of format address space */ - memset(buf, 0, size); - size = 0; - } - assert(nbytes>=0); - assert((hsize_t)nbytes<=size); - size -= (hsize_t)nbytes; - addr += (haddr_t)nbytes; - buf = (char*)buf + nbytes; - } - - /* Update current position */ - file->pos = addr; - file->op = OP_READ; - return 0; -} -</PRE> - -<P> -<STRONG>Example:</STRONG> The sec2 <CODE>write</CODE> callback is similar except it updates -the file EOF marker when extending the file. - -</P> - - -<H3><A NAME="SEC21" HREF="VFL.html#TOC21">Flushing Cached Data</A></H3> - -<P> -Some drivers may desire to cache data in memory in order to make larger I/O -requests to the underlying file and thus improving bandwidth. Such drivers -should register a cache flushing function so that the library can insure that -data has been flushed out of the drivers in response to the application -calling <CODE>H5Fflush</CODE>. - -</P> -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>flush</B> <I>(H5FD_t *<VAR>file</VAR>)</I> -<DD><A NAME="IDX12"></A> - -</P> -<P> -Flush all data for file <VAR>file</VAR> to storage. -</DL> - -</P> -<P> -<STRONG>Example:</STRONG> The sec2 driver doesn't cache any data but it also doesn't -extend the Unix file as agressively as it should. Therefore, when finalizing a -file it should write a zero to the last byte of the allocated region so that -when reopening the file later the EOF marker will be at least as large as the -EOA marker saved in the superblock (otherwise HDF5 will refuse to open the -file, claiming that the data appears to be truncated). - -</P> - -<PRE> -static herr_t -H5FD_sec2_flush(H5FD_t *_file) -{ - H5FD_sec2_t *file = (H5FD_sec2_t*)_file; - - if (file->eoa>file->eof) { - if (-1==file_seek(file->fd, file->eoa-1, SEEK_SET)) return -1; - if (write(file->fd, "", 1)!=1) return -1; - file->eof = file->eoa; - file->pos = file->eoa; - file->op = OP_WRITE; - } - - return 0; -} -</PRE> - - - -<H2><A NAME="SEC22" HREF="VFL.html#TOC22">Optimization Functions</A></H2> - -<P> -The library is capable of performing several generic optimizations on I/O, but -these types of optimizations may not be appropriate for a given VFL driver. -</P> - -<P> -Each driver may provide a query function to allow the library to query whether -to enable these optimizations. If a driver lacks a query function, the library -will disable all types of optimizations which can be queried. -</P> - -<P> -<DL> -<DT><U>Function:</U> static herr_t <B>query</B> <I>(const H5FD_t *<VAR>file</VAR>, unsigned long *<VAR>flags</VAR>)</I> -<DD><A NAME="IDX17"></A> -</P> -<P> -This function is called by the library to query which optimizations to enable -for I/O to this driver. These are the flags which are currently defined: - -<UL> -<DL> -<DT>H5FD_FEAT_AGGREGATE_METADATA (0x00000001) -<DD>Defining the H5FD_FEAT_AGGREGATE_METADATA for a VFL driver means that -the library will attempt to allocate a larger block for metadata and -then sub-allocate each metadata request from that larger block. -<DT>H5FD_FEAT_ACCUMULATE_METADATA (0x00000002) -<DD>Defining the H5FD_FEAT_ACCUMULATE_METADATA for a VFL driver means that -the library will attempt to cache metadata as it is written to the file -and build up a larger block of metadata to eventually pass to the VFL -'write' routine. -<DT>H5FD_FEAT_DATA_SIEVE (0x00000004) -<DD>Defining the H5FD_FEAT_DATA_SIEVE for a VFL driver means that -the library will attempt to cache raw data as it is read from/written to -a file in a "data sieve" buffer. See Rajeev Thakur's papers: - <UL> - <DL> - <DT>http://www.mcs.anl.gov/~thakur/papers/romio-coll.ps.gz - <DT>http://www.mcs.anl.gov/~thakur/papers/mpio-high-perf.ps.gz - </DL> - </UL> -</DL> -</UL> -</P> - -</DL> -</P> - -<H2><A NAME="SEC23" HREF="VFL.html#TOC23">Registration of a Driver</A></H2> - -<P> -Before a driver can be used the HDF5 library needs to be told of its -existence. This is done by registering the driver, which results in a driver -identification number. Instead of passing many arguments to the registration -function, the driver information is entered into a structure and the address -of the structure is passed to the registration function where it is -copied. This allows the HDF5 API to be extended while providing backward -compatibility at the source level. - -</P> -<P> -<DL> -<DT><U>Function:</U> hid_t <B>H5FDregister</B> <I>(H5FD_class_t *<VAR>cls</VAR>)</I> -<DD><A NAME="IDX13"></A> - -</P> -<P> -The driver described by struct <VAR>cls</VAR> is registered with the library and an -ID number for the driver is returned. -</DL> - -</P> -<P> -The <CODE>H5FD_class_t</CODE> type is a struct with the following fields: - -</P> -<DL COMPACT> - -<DT><CODE>const char *name</CODE> -<DD> -A pointer to a constant, null-terminated driver name to be used for debugging -purposes. -<DT><CODE>size_t fapl_size</CODE> -<DD> -The size in bytes of the file access mode structure or zero if the driver -supplies a copy function or doesn't define the structure. -<DT><CODE>void *(*fapl_copy)(const void *fapl)</CODE> -<DD> -An optional function which copies a driver-defined file access mode structure. -This field takes precedence over <CODE>fm_size</CODE> when both are defined. -<DT><CODE>void (*fapl_free)(void *fapl)</CODE> -<DD> -An optional function to free the driver-defined file access mode structure. If -null, then the library calls the C <CODE>free</CODE> function to free the -structure. -<DT><CODE>size_t dxpl_size</CODE> -<DD> -The size in bytes of the data transfer mode structure or zero if the driver -supplies a copy function or doesn't define the structure. -<DT><CODE>void *(*dxpl_copy)(const void *dxpl)</CODE> -<DD> -An optional function which copies a driver-defined data transfer mode -structure. This field takes precedence over <CODE>xm_size</CODE> when both are -defined. -<DT><CODE>void (*dxpl_free)(void *dxpl)</CODE> -<DD> -An optional function to free the driver-defined data transfer mode -structure. If null, then the library calls the C <CODE>free</CODE> function to -free the structure. -<DT><CODE>H5FD_t *(*open)(const char *name, unsigned flags, hid_t fapl, haddr_t maxaddr)</CODE> -<DD> -The function which opens or creates a new file. -<DT><CODE>herr_t (*close)(H5FD_t *file)</CODE> -<DD> -The function which ends access to a file. -<DT><CODE>int (*cmp)(const H5FD_t *f1, const H5FD_t *f2)</CODE> -<DD> -An optional function to determine whether two open files have the same key. If -this function is not present then the library assumes that two files will -never be the same. -<DT><CODE>int (*query)(const H5FD_t *f, unsigned long *flags)</CODE> -<DD> -An optional function to determine which library optimizations a driver can -support. -<DT><CODE>haddr_t (*alloc)(H5FD_t *file, H5FD_mem_t type, hsize_t size)</CODE> -<DD> -An optional function to allocate space in the file. -<DT><CODE>herr_t (*free)(H5FD_t *file, H5FD_mem_t type, haddr_t addr, hsize_t size)</CODE> -<DD> -An optional function to free space in the file. -<DT><CODE>haddr_t (*get_eoa)(H5FD_t *file)</CODE> -<DD> -A function to query how much of the format address space has been allocated. -<DT><CODE>herr_t (*set_eoa)(H5FD_t *file, haddr_t)</CODE> -<DD> -A function to set the end of address space. -<DT><CODE>haddr_t (*get_eof)(H5FD_t *file)</CODE> -<DD> -A function to return the current end-of-file marker value. -<DT><CODE>herr_t (*read)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, void *buffer)</CODE> -<DD> -A function to read data from a file. -<DT><CODE>herr_t (*write)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, const void *buffer)</CODE> -<DD> -A function to write data to a file. -<DT><CODE>herr_t (*flush)(H5FD_t *file)</CODE> -<DD> -A function which flushes cached data to the file. -<DT><CODE>H5FD_mem_t fl_map[H5FD_MEM_NTYPES]</CODE> -<DD> -An array which maps a file allocation request type to a free list. -</DL> - -<P> -<STRONG>Example:</STRONG> The sec2 driver would be registered as: - -</P> - -<PRE> -static const H5FD_class_t H5FD_sec2_g = { - "sec2", /*name */ - MAXADDR, /*maxaddr */ - NULL, /*sb_size */ - NULL, /*sb_encode */ - NULL, /*sb_decode */ - 0, /*fapl_size */ - NULL, /*fapl_get */ - NULL, /*fapl_copy */ - NULL, /*fapl_free */ - 0, /*dxpl_size */ - NULL, /*dxpl_copy */ - NULL, /*dxpl_free */ - H5FD_sec2_open, /*open */ - H5FD_sec2_close, /*close */ - H5FD_sec2_cmp, /*cmp */ - H5FD_sec2_query, /*query */ - NULL, /*alloc */ - NULL, /*free */ - H5FD_sec2_get_eoa, /*get_eoa */ - H5FD_sec2_set_eoa, /*set_eoa */ - H5FD_sec2_get_eof, /*get_eof */ - H5FD_sec2_read, /*read */ - H5FD_sec2_write, /*write */ - H5FD_sec2_flush, /*flush */ - H5FD_FLMAP_SINGLE, /*fl_map */ -}; - -hid_t -H5FD_sec2_init(void) -{ - if (!H5FD_SEC2_g) { - H5FD_SEC2_g = H5FDregister(&H5FD_sec2_g); - } - return H5FD_SEC2_g; -} -</PRE> - -<P> -A driver can be removed from the library by unregistering it - -</P> -<P> -<DL> -<DT><U>Function:</U> herr_t <B>H5Dunregister</B> <I>(hid_t <VAR>driver</VAR>)</I> -<DD><A NAME="IDX14"></A> -Where <VAR>driver</VAR> is the ID number returned when the driver was registered. -</DL> - -</P> -<P> -Unregistering a driver makes it unusable for creating new file access or data -transfer property lists but doesn't affect any property lists or files that -already use that driver. - -</P> - - - -<H2><A NAME="SEC24" HREF="VFL.html#TOC24">Querying Driver Information</A></H2> - -<P> -<DL> -<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fapl</VAR>)</I> -<DD><A NAME="IDX15"></A> -<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fxpl</VAR>)</I> -<DD><A NAME="IDX16"></A> - -</P> -<P> -This function is intended to be used by driver functions, not applications. -It returns a pointer directly into the file access property list -<CODE><VAR>fapl</VAR></CODE> which is a copy of the driver's file access mode originally -provided to the <CODE>H5Pset_driver</CODE> function. If its argument is a data -transfer property list <CODE>fxpl</CODE> then it returns a pointer to the -driver-specific data transfer information instead. -</DL> - -</P> - - -<H1><A NAME="SEC25" HREF="VFL.html#TOC25">Miscellaneous</A></H1> - -<P> -The various private <CODE>H5F_low_*</CODE> functions will be replaced by public -<CODE>H5FD*</CODE> functions so they can be called from drivers. - -</P> -<P> -All private functions <CODE>H5F_addr_*</CODE> which operate on addresses will be -renamed as public functions by removing the first underscore so they can be -called by drivers. - -</P> -<P> -The <CODE>haddr_t</CODE> address data type will be passed by value throughout the -library. The original intent was that this type would eventually be a union of -file address types for the various drivers and may become quite large, but -that was back when drivers were part of HDF5. It will become an alias for an -unsigned integer type (32 or 64 bits depending on how the library was -configured). - -</P> -<P> -The various <CODE>H5F*.c</CODE> driver files will be renamed <CODE>H5FD*.c</CODE> and each -will have a corresponding header file. All driver functions except the -initializer and API will be declared static. - -</P> -<P> -This documentation didn't cover optimization functions which would be useful -to drivers like MPI-IO. Some drivers may be able to perform data pipeline -operations more efficiently than HDF5 and need to be given a chance to -override those parts of the pipeline. The pipeline would be designed to call -various H5FD optimization functions at various points which return one of -three values: the operation is not implemented by the driver, the operation is -implemented but failed in a non-recoverable manner, the operation is -implemented and succeeded. - -</P> -<P> -Various parts of HDF5 check the only the top-level file driver and do -something special if it is the MPI-IO driver. However, we might want to be -able to put the MPI-IO driver under other drivers such as the raw part of a -split driver or under a debug driver whose sole purpose is to accumulate -statistics as it passes all requests through to the MPI-IO driver. Therefore -we will probably need a function which takes a format address and or object -type and returns the driver which would have been used at the lowest level to -process the request. - -</P> - -<P><HR><P> -<H1>Footnotes</H1> -<H3><A NAME="FOOT1" HREF="VFL.html#DOCF1">(1)</A></H3> -<P>The driver name is by convention and might -not apply to drivers which are not distributed with HDF5. -<H3><A NAME="FOOT2" HREF="VFL.html#DOCF2">(2)</A></H3> -<P>The access method also indicates how to translate -the storage name to a storage server such as a file, network protocol, or -memory. -<H3><A NAME="FOOT3" HREF="VFL.html#DOCF3">(3)</A></H3> -<P>The term -"<EM>file</EM> access property list" is a misnomer since storage isn't -required to be a file. -<H3><A NAME="FOOT4" HREF="VFL.html#DOCF4">(4)</A></H3> -<P>This -function is overloaded to operate on data transfer property lists also, as -described below. -<H3><A NAME="FOOT5" HREF="VFL.html#DOCF5">(5)</A></H3> -<P>Read-only access is only appropriate when opening an existing -file. -<H3><A NAME="FOOT6" HREF="VFL.html#DOCF6">(6)</A></H3> -<P>For instance, writing data to one handle will cause -the data to be immediately visible on the other handle. -<H3><A NAME="FOOT7" HREF="VFL.html#DOCF7">(7)</A></H3> -<P>The ordering is -arbitrary as long as it's consistent within a particular file driver. -<H3><A NAME="FOOT8" HREF="VFL.html#DOCF8">(8)</A></H3> -<P>File access modes do not describe data, but rather -describe how the HDF5 format address space is mapped to the underlying -file(s). Thus, in general the mapping must be known before the file superblock -can be read. However, the user usually knows enough about the mapping for the -superblock to be readable and once the superblock is read the library can fill -in the missing parts of the mapping. -<P><HR><P> -This document was generated on 18 November 1999 using the -<A HREF="http://wwwcn.cern.ch/dci/texi2html/">texi2html</A> -translator version 1.51.</P> -<P> -Updated on 10/24/00 by hand, Quincey Koziol -</P> -</BODY> -</HTML> diff --git a/doc/html/TechNotes/VFLfunc.html b/doc/html/TechNotes/VFLfunc.html deleted file mode 100644 index 1e33593..0000000 --- a/doc/html/TechNotes/VFLfunc.html +++ /dev/null @@ -1,64 +0,0 @@ -<html> -<head> -<title>VFL Functions</title> -</head> - -<body> - -<h1>List of HDF5 VFL Functions</h1> - -<pre> -The following functions support the HDF5 virtual file layer (VFL), enabling -the creation of customized I/O drivers. - -At this time, these functions are documented only in the <a href="VFL.html">HDF5 Virtual File -Layer</a> design document and in the source code. - - - -herr_t <font color=red>H5Pset_driver</font>(hid_t plist_id, hid_t driver_id, - const void *driver_info) - -void *<font color=red>H5Pget_driver_info</font>(hid_t plist_id) - -hid_t <font color=red>H5FDregister</font>(const H5FD_class_t *cls); - -herr_t <font color=red>H5FDunregister</font>(hid_t driver_id); - -H5FD_t *<font color=red>H5FDopen</font>(const char *name, unsigned flags, hid_t fapl_id, - haddr_t maxaddr); - -herr_t <font color=red>H5FDclose</font>(H5FD_t *file); - -int <font color=red>H5FDcmp</font>(const H5FD_t *f1, const H5FD_t *f2); - -int <font color=red>H5FDquery</font>(const H5FD_t *f, unsigned long *flags); - -haddr_t <font color=red>H5FDalloc</font>(H5FD_t *file, H5FD_mem_t type, hsize_t size); - -herr_t <font color=red>H5FDfree</font>(H5FD_t *file, H5FD_mem_t type, haddr_t addr, hsize_t size); - -haddr_t <font color=red>H5FDrealloc</font>(H5FD_t *file, H5FD_mem_t type, haddr_t addr, - hsize_t old_size, hsize_t new_size); - -haddr_t <font color=red>H5FDget_eoa</font>(H5FD_t *file); - -herr_t <font color=red>H5FDset_eoa</font>(H5FD_t *file, haddr_t eof); - -haddr_t <font color=red>H5FDget_eof</font>(H5FD_t *file); - -herr_t <font color=red>H5FDread</font>(H5FD_t *file, H5FD_mem_t type, hid_t dxpl_id, haddr_t addr, - size_t size, void *buf/*out*/); - -herr_t <font color=red>H5FDwrite</font>(H5FD_t *file, H5FD_mem_t type, hid_t dxpl_id, - haddr_t addr, size_t size, const void *buf); - -herr_t <font color=red>H5FDflush</font></font>(H5FD_t *file, unsigned closing); - -=========================================== -Last modified: 25 June 2002 -HDF Help Desk: hdfhelp@ncsa.uiuc.edu - -</pre> -</body> -</html> diff --git a/doc/html/TechNotes/VLTypes.html b/doc/html/TechNotes/VLTypes.html deleted file mode 100644 index 8a41c10..0000000 --- a/doc/html/TechNotes/VLTypes.html +++ /dev/null @@ -1,150 +0,0 @@ -<html> - <head> - <title> - Variable-Length Datatypes in HDF5 - </title> - - <STYLE TYPE="text/css"> - - P { text-indent: 2em} - P.item { margin-left: 2em; text-indent: -2em} - P.item2 { margin-left: 2em; text-indent: 2em} - - TABLE.format { border:solid; border-collapse:collapse; caption-side:top; text-align:center; width:80%;} - TABLE.format TH { border:ridge; padding:4px; width:25%;} - TABLE.format TD { border:ridge; padding:4px; } - TABLE.format CAPTION { font-weight:bold; font-size:larger;} - - TABLE.note {border:none; text-align:right; width:80%;} - - TABLE.desc { border:solid; border-collapse:collapse; caption-size:top; text-align:left; width:80%;} - TABLE.desc TR { vertical-align:top;} - TABLE.desc TH { border-style:ridge; font-size:larger; padding:4px; text-decoration:underline;} - TABLE.desc TD { border-style:ridge; padding:4px; } - TABLE.desc CAPTION { font-weight:bold; font-size:larger;} - - TABLE.list { border:none; } - TABLE.list TR { vertical-align:top;} - TABLE.list TH { border:none; text-decoration:underline;} - TABLE.list TD { border:none; } - - </STYLE> - - <!-- #BeginLibraryItem "/ed_libs/styles_Format.lbi" --> -<!-- - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * - * Copyright by the Board of Trustees of the University of Illinois. * - * All rights reserved. * - * * - * This file is part of HDF5. The full HDF5 copyright notice, including * - * terms governing use, modification, and redistribution, is contained in * - * the files COPYING and Copyright.html. COPYING can be found at the root * - * of the source code distribution tree; Copyright.html can be found at the * - * root level of an installed copy of the electronic HDF5 document set and * - * is linked from the top-level documents page. It can also be found at * - * http://hdf.ncsa.uiuc.edu/HDF5/doc/Copyright.html. If you do not have * - * access to either file, you may request a copy from hdfhelp@ncsa.uiuc.edu. * - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * - --> - -<link href="../ed_styles/FormatElect.css" rel="stylesheet" type="text/css"> -<!-- #EndLibraryItem --></head> - - <body bgcolor="#FFFFFF"> - <H3>Introduction</H3> - <P>Variable-length (VL) datatypes have a great deal of flexibility, but can - be over- or mis-used. VL datatypes are ideal at capturing the notion - that elements in an HDF5 dataset (or attribute) can have different - amounts of information (VL strings are the canonical example), - but they have some drawbacks that this document attempts - to address. - </P> - - <H3>Background</H3> - <P>Because fast random access to dataset elements requires that each - element be a fixed size, the information stored for VL datatype elements - is actually information to locate the VL information, not - the information itself. - </P> - - <H3>When to use VL datatypes</H3> - <P>VL datatypes are designed allow the amount of data stored in each - element of a dataset to vary. This change could be - over time as new values, with different lengths, were written to the - element. Or, the change can be over "space" - the dataset's space, - with each element in the dataset having the same fundamental type, but - different lengths. "Ragged arrays" are the classic example of elements - that change over the "space" of the dataset. If the elements of a - dataset are not going to change over "space" or time, a VL datatype - should probably not be used. - </P> - - <H3>Access Time Penalty</H3> - <P>Accessing VL information requires reading the element in the file, then - using that element's location information to retrieve the VL - information itself. - In the worst case, this obviously doubles the number of disk accesses - required to access the VL information. - </P> - <P>However, in order to avoid this extra disk access overhead, the HDF5 - library groups VL information together into larger blocks on disk and - performs I/O only on those larger blocks. Additionally, these blocks of - information are cached in memory as long as possible. For most access - patterns, this amortizes the extra disk accesses over enough pieces of - VL information to hide the extra overhead involved. - </P> - - <H3>Storage Space Penalty</H3> - <P>Because VL information must be located and retrieved from another - location in the file, extra information must be stored in the file to - locate - each item of VL information (i.e. each element in a dataset or each - VL field in a compound datatype, etc.). - Currently, that extra information amounts to 32 bytes per VL item. - </P> - <P> - With some judicious re-architecting of the library and file format, - this could be reduced to 18 bytes per VL item with no loss in - functionality or additional time penalties. With some additional - effort, the space could perhaps could be pushed down as low as 8-10 - bytes per VL item with no loss in functionality, but potentially a - small time penalty. - </P> - - <H3>Chunking and Filters</H3> - <P>Storing data as VL information has some affects on chunked storage and - the filters that can be applied to chunked data. Because the data that - is stored in each chunk is the location to access the VL information, - the actual VL information is not broken up into chunks in the same way - as other data stored in chunks. Additionally, because the - actual VL information is not stored in the chunk, any filters which - operate on a chunk will operate on the information to - locate the VL information, not the VL information itself. - </P> - - <H3>File Drivers</H3> - <P>Because the parallel I/O file drivers (MPI-I/O and MPI-posix) don't - allow objects with varying sizes to be created in the file, attemping - to create - a dataset or attribute with a VL datatype in a file managed by those - drivers will cause the creation call to fail. - </P> - <P>Additionally, using - VL datatypes and the 'multi' and 'split' file drivers may not operate - in the manner desired. The HDF5 library currently categorizes the - "blocks of VL information" stored in the file as a type of metadata, - which means that they may not be stored with the other raw data for - the file. - </P> - - <H3>Rewriting</H3> - <P>When VL information in the file is re-written, the old VL information - must be releases, space for the new VL information allocated and - the new VL information must be written to the file. This may cause - additional I/O accesses. - </P> - - </body> - -</html> - diff --git a/doc/html/TechNotes/Version.html b/doc/html/TechNotes/Version.html deleted file mode 100644 index 0e0853b..0000000 --- a/doc/html/TechNotes/Version.html +++ /dev/null @@ -1,137 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> -<html> - <head> - <title>Version Numbers</title> - </head> - - <body> - <h1>HDF5 Release Version Numbers</h1> - - <h2>1. Introduction</h2> - - <p>The HDF5 version number is a set of three integer values - written as either <code>hdf5-1.2.3</code> or <code>hdf5 version - 1.2 release 3</code>. - - <p>The <code>5</code> is part of the library name and will only - change if the entire file format and library are redesigned - similar in scope to the changes between HDF4 and HDF5. - - <p>The <code>1</code> is the <em>major version number</em> and - changes when there is an extensive change to the file format or - library API. Such a change will likely require files to be - translated and applications to be modified. This number is not - expected to change frequently. - - <p>The <code>2</code> is the <em>minor version number</em> and is - incremented by each public release that presents new features. - Even numbers are reserved for stable public versions of the - library while odd numbers are reserved for developement - versions. See the diagram below for examples. - - <p>The <code>3</code> is the <em>release number</em>. For public - versions of the library, the release number is incremented each - time a bug is fixed and the fix is made available to the public. - For development versions, the release number is incremented more - often (perhaps almost daily). - - <h2>2. Abbreviated Versions</h2> - - <p>It's often convenient to drop the release number when referring - to a version of the library, like saying version 1.2 of HDF5. - The release number can be any value in this case. - - <h2>3. Special Versions</h2> - - <p>Version 1.0.0 was released for alpha testing the first week of - March, 1998. The developement version number was incremented to - 1.0.1 and remained constant until the the last week of April, - when the release number started to increase and development - versions were made available to people outside the core HDF5 - development team. - - <p>Version 1.0.23 was released mid-July as a second alpha - version. - - <p>Version 1.1.0 will be the first official beta release but the - 1.1 branch will also serve as a development branch since we're - not concerned about providing bug fixes separate from normal - development for the beta version. - - <p>After the beta release we rolled back the version number so the - first release is version 1.0 and development will continue on - version 1.1. We felt that an initial version of 1.0 was more - important than continuing to increment the pre-release version - numbers. - - <h2>4. Public versus Development</h2> - - <p>The motivation for separate public and development versions is - that the public version will receive only bug fixes while the - development version will receive new features. This also allows - us to release bug fixes expediently without waiting for the - development version to reach a stable state. - - <p>Eventually, the development version will near completion and a - new development branch will fork while the original one enters a - feature freeze state. When the original development branch is - ready for release the minor version number will be incremented - to an even value. - - <p> - <center> - <img alt="Version Example" src="version.gif"> - <br><b>Fig 1: Version Example</b> - </center> - - <h2>5. Version Support from the Library</h2> - - <p>The library provides a set of macros and functions to query and - check version numbers. - - <dl> - <dt><code>H5_VERS_MAJOR</code> - <dt><code>H5_VERS_MINOR</code> - <dt><code>H5_VERS_RELEASE</code> - <dd>These preprocessor constants are defined in the public - include file and determine the version of the include files. - - <br><br> - <dt><code>herr_t H5get_libversion (unsigned *<em>majnum</em>, unsigned - *<em>minnum</em>, unsigned *<em>relnum</em>)</code> - <dd>This function returns through its arguments the version - numbers for the library to which the application is linked. - - <br><br> - <dt><code>void H5check(void)</code> - <dd>This is a macro that verifies that the version number of the - HDF5 include file used to compile the application matches the - version number of the library to which the application is - linked. This check occurs automatically when the first HDF5 - file is created or opened and is important because a mismatch - between the include files and the library is likely to result - in corrupted data and/or segmentation faults. If a mismatch - is detected the library issues an error message on the - standard error stream and aborts with a core dump. - - <br><br> - <dt><code>herr_t H5check_version (unsigned <em>majnum</em>, - unsigned <em>minnum</em>, unsigned <em>relnum</em>)</code> - <dd>This function is called by the <code>H5check()</code> macro - with the include file version constants. The function - compares its arguments to the result returned by - <code>H5get_libversion()</code> and if a mismatch is detected prints - an error message on the standard error stream and aborts. - </dl> - -<hr> -<address><a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</a></address> -<br> - -<!-- Created: Wed Apr 22 11:24:40 EDT 1998 --> -<!-- hhmts start --> -Last modified: Fri Oct 30 10:32:50 EST 1998 -<!-- hhmts end --> - - </body> -</html> diff --git a/doc/html/TechNotes/openmp-hdf5.c b/doc/html/TechNotes/openmp-hdf5.c deleted file mode 100644 index 6d61c38..0000000 --- a/doc/html/TechNotes/openmp-hdf5.c +++ /dev/null @@ -1,403 +0,0 @@ -Appendix A: OpenMP-HDF5 Programs -------------------------------------------------------------------------- - Program 1 -------------------------------------------------------------------------- -/* - * This example writes 64 datasets to a HDF5 file, using multiple threads - * (OpenMP). Each thread grab the lock while it tries to call HDF5 functions - * to write out dataset. In this way, the HDF5 calls are serialized, while - * the calculation part is in parallel. This is one of the ways to do - * OpenMP computation with HDF. As long as not to do HDF I/O in parallel, - * it is safe to use HDF. - */ - -#include <hdf5.h> -#include <omp.h> -#include <math.h> - -#define NUM_THREADS 4 -#define NUM_MDSET 16 -#define FILE "SDS.h5" -#define NX 5 /* dataset dimensions */ -#define NY 18 -#define RANK 2 - -void CalcWriteData(hid_t, hid_t, hid_t); - -/*Global variable, OpenMP lock*/ -omp_lock_t lock; - - -int -main (void) -{ - hid_t fid; /* file and dataset handles */ - hid_t datatype, dataspace; /* handles */ - hsize_t dimsf[2]; /* dataset dimensions */ - herr_t status; - int i; - - /* - * Create a new file using H5F_ACC_TRUNC access, - * default file creation properties, and default file - * access properties. - */ - fid = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); - - /* - * Describe the size of the array and create the data space for fixed - * size dataset. - */ - dimsf[0] = NX; - dimsf[1] = NY; - dataspace = H5Screate_simple(RANK, dimsf, NULL); - - /* - * Define datatype for the data in the file. - * We will store little endian INT numbers. - */ - datatype = H5Tcopy(H5T_NATIVE_DOUBLE); - status = H5Tset_order(datatype, H5T_ORDER_LE); - - /*Disable dynamic allocation of threads*/ - omp_set_dynamic(0); - - /*Allocate threads*/ - omp_set_num_threads(NUM_THREADS); - - /*Initialize lock*/ - omp_init_lock(&lock); - - /*Each thread grab one iteration in the for loop and call function - * CaclWriteData*/ - #pragma omp parallel default(shared) - { - #pragma omp for - for(i=0; i<NUM_THREADS; i++) { - CalcWriteData(fid, dataspace, datatype); - } - } - - /*Finished lock mechanism, destroy it*/ - omp_destroy_lock(&lock); - - /* - * Close/release resources. - */ - H5Sclose(dataspace); - H5Tclose(datatype); - H5Fclose(fid); - - return 0; -} - -/*Each thread will call this function independantly. They calculate dataset - *and then write it out to hdf, for NUM_MDSET times */ -void CalcWriteData(hid_t fid, hid_t dataspace, hid_t datatype) -{ - double data[NX][NY]; - hid_t dataset; - char dname[16]; - int tid; - int i, j, k; - - tid = omp_get_thread_num(); - - for(i=0; i<NUM_MDSET; i++) { - /*Weird calculation to extend computing time*/ - for(j=0; j<NX; j++) { - for(k=0; k<NY; k++) - data[j][k] = ( pow(sin(tid), 2.0) + pow(cos(tid), 2.0) ) * - ( tid + i ) + - ( pow(123456789.0, 8.0) - pow(123456789.0, 8.0) ); - - } - sprintf(dname, "tid%d-dataset%d", tid, i); - - /*Serialize HDF dataset writing using OpenMP lock*/ - omp_set_lock(&lock); - - dataset = H5Dcreate(fid, dname, datatype, dataspace, H5P_DEFAULT); - H5Dwrite(dataset, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, - data); - H5Dclose(dataset); - - /*Release lock*/ - omp_unset_lock(&lock); - } - -} - - - - -------------------------------------------------------------------------- - Program 2 -------------------------------------------------------------------------- -/* - * This example compute the element values of an array in parallel - * by two threads. Then write this dataset into HDF file. This is - * one of the ways to do OpenMP computation with HDF. As long as - * not to do HDF I/O in parallel, it is safe to use HDF. - * - */ - -#include <hdf5.h> -#include <omp.h> - -#define FILE "SDS.h5" -#define DATASETNAME "IntArray" -#define NX 5 /* dataset dimensions */ -#define NY 6 -#define RANK 2 - -int -main (void) -{ - hid_t file, dataset; /* file and dataset handles */ - hid_t datatype, dataspace; /* handles */ - hsize_t dimsf[2]; /* dataset dimensions */ - herr_t status; - int data[NX][NY]; /* data to write */ - int i, j; - - /* - * Create a new file using H5F_ACC_TRUNC access, - * default file creation properties, and default file - * access properties. - */ - file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); - - /* - * Describe the size of the array and create the data space for fixed - * size dataset. - */ - dimsf[0] = NX; - dimsf[1] = NY; - dataspace = H5Screate_simple(RANK, dimsf, NULL); - - /* - * Define datatype for the data in the file. - * We will store little endian INT numbers. - */ - datatype = H5Tcopy(H5T_NATIVE_INT); - status = H5Tset_order(datatype, H5T_ORDER_LE); - - /* Disable dynamic allocation of threads. */ - omp_set_dynamic(0); - - /* Allocate 2 threads */ - omp_set_num_threads(2); - - - /* Parallel computation. Let 2 threads handle this nested for loops - * in parallel. Only one data array is computed. */ - #pragma omp parallel default(shared) - { - #pragma omp for - for (j = 0; j < NX; j++) { - #pragma omp parallel shared(j, NY) - { - #pragma omp for - for (i = 0; i < NY; i++) - data[j][i] = i + j; - } - } - } - - /* Write this dataset into HDF file */ - dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace, - H5P_DEFAULT); - H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, - H5P_DEFAULT, data); - H5Dclose(dataset); - - /* - * Close/release resources. - */ - H5Sclose(dataspace); - H5Tclose(datatype); - H5Fclose(file); - - return 0; -} - - - -------------------------------------------------------------------------- - Program 3 -------------------------------------------------------------------------- -/* - * This example create two threads. Each thread writes a dataset to - * the HDF5 file in parallel. This program only works occasionally. - */ - -#include <hdf5.h> -#include <omp.h> - -#define FILE "SDS.h5" -#define NX 5 /* dataset dimensions */ -#define NY 6 -#define RANK 2 - -void writeData(int, hid_t, hid_t, hid_t, char*); - -int main (void) -{ - hid_t file; /* file and dataset handles */ - hid_t datatype, dataspace; /* handles */ - hsize_t dimsf[2]; /* dataset dimensions */ - herr_t status; - int id; - char dname[2][10] = {"Array1", "Array2"}; - - /* - * Create a new file using H5F_ACC_TRUNC access, - * default file creation properties, and default file - * access properties. - */ - file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); - - /* - * Describe the size of the array and create the data space for fixed - * size dataset. - */ - dimsf[0] = NX; - dimsf[1] = NY; - dataspace = H5Screate_simple(RANK, dimsf, NULL); - - /* - * Define datatype for the data in the file. - * We will store little endian INT numbers. - */ - datatype = H5Tcopy(H5T_NATIVE_INT); - status = H5Tset_order(datatype, H5T_ORDER_LE); - - /*Disable dynamic allocation of threads*/ - omp_set_dynamic(0); - - /*Allocate 2 threads*/ - omp_set_num_threads(2); - - /*Parallel Part: each thread call function writeData; id is private to - * thread while others are shared */ - #pragma omp parallel shared(file, dataspace, datatype, dname) private(id) - { - id = omp_get_thread_num(); - writeData(id, file, dataspace, datatype, dname[id]); - } - - - /* - * Close/release resources. - */ - H5Sclose(dataspace); - H5Tclose(datatype); - H5Fclose(file); - - return 0; -} - - -/*Each thread call this function to write a dataset into HDF5 file*/ - -void writeData(int id, hid_t file, hid_t dataspace, hid_t datatype, char *dname) -{ - int data[NX][NY]; - hid_t dataset; - int i, j; - - for (j = 0; j < NX; j++) { - for (i = 0; i < NY; i++) - data[j][i] = i + j + id; - } - - dataset = H5Dcreate(file, dname, datatype, dataspace, - H5P_DEFAULT); - H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, - H5P_DEFAULT, data); - H5Dclose(dataset); -} - - -------------------------------------------------------------------------- - Program 4 -------------------------------------------------------------------------- -/* - * This example compute and write two datasets into HDF file in - * parallel. It also only works occasionally. - */ - -#include <hdf5.h> -#include <omp.h> - -#define FILE "SDS.h5" -#define DATASETNAME "IntArray" -#define NX 5 /* dataset dimensions */ -#define NY 6 -#define RANK 2 - -int -main (void) -{ - hid_t file, dataset; /* file and dataset handles */ - hid_t datatype, dataspace; /* handles */ - hsize_t dimsf[2]; /* dataset dimensions */ - herr_t status; - int data[NX][NY]; /* data to write */ - int i, j, id; - char dname[2][10] = {"intArray1", "intArray2"}; - - /* - * Create a new file using H5F_ACC_TRUNC access, - * default file creation properties, and default file - * access properties. - */ - file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); - - /* - * Describe the size of the array and create the data space for fixed - * size dataset. - */ - dimsf[0] = NX; - dimsf[1] = NY; - dataspace = H5Screate_simple(RANK, dimsf, NULL); - - /* - * Define datatype for the data in the file. - * We will store little endian INT numbers. - */ - datatype = H5Tcopy(H5T_NATIVE_INT); - status = H5Tset_order(datatype, H5T_ORDER_LE); - - omp_set_dynamic(0); - omp_set_num_threads(2); - - - /* This part of program compute and write two datasets in parallel. */ - #pragma omp parallel shared(file, datatype, dataspace, dname) private(id, j, i, data, dataset) - { - id = omp_get_thread_num(); - for (j = 0; j < NX; j++) { - for (i = 0; i < NY; i++) - data[j][i] = i + j + id; - } - - dataset = H5Dcreate(file, dname[id], datatype, dataspace, - H5P_DEFAULT); - H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, - H5P_DEFAULT, data); - H5Dclose(dataset); - } - - - /* - * Close/release resources. - */ - H5Sclose(dataspace); - H5Tclose(datatype); - H5Fclose(file); - - return 0; -} - diff --git a/doc/html/TechNotes/openmp-hdf5.html b/doc/html/TechNotes/openmp-hdf5.html deleted file mode 100644 index ff13b81..0000000 --- a/doc/html/TechNotes/openmp-hdf5.html +++ /dev/null @@ -1,67 +0,0 @@ -<pre> - Using HDF5 with OpenMP - ---------------------- - - -1. Introduction to OpenMP -------------------------- - - - For shared-memory parallelism - - A combination of library and directives - - Available for C/C++ and Fortran - - SGI leading effort - - Information at http://www.openmp.org and - http://www.sgi.com/software/openmp - -2. Programming(SGI MPISpro compiler and C language) ---------------------------------------------------- - - - Turn on compiler '-mp' option - - Include 'omp.h' library in program - - Use library functions, directives and environment variables - - -3. Sample Programs ------------------- - -Appendix A contains four OpenMP-HDF5 test programs. (They are derived from -the hdf5/examples/h5_write.c). The purpose of these program is to -test OpenMP parallelism with the HDF5 library. - -All tests were run on modi4 with SGI MPISpro compiler(cc) and make. -Program 1 and Program 2 are the working programs. Program 3 and Program 4 -work occasionally due to racing conditions. -Follow the following steps to try the programs. - - a. have your hdf5 library compiled, - b. go to hdf5/examples directory, - c. add -mp option to the end of the CFLAGS list in the Makefile. If you - have the compiled program in another directory, you should go to the - examples in that directory. - d. modify the hdf5/examples/h5_write.c according to the program attached - here. - e. use hdf5/tools/h5dump to examine the output file. - - -4. Conclusion -------------- - -It is not safe to invoke HDF5 library calls via multiple threads in an -OpenMP program. But if one serializes HDF5 calls as illustrated in Program 1, -the HDF5 library works correctly with the OpenMP programs. - -The serialization of HDF5 calls will slow down the OpenMP program unnecessarily. -Future study is needed to check possible ways to "un-seralize" the HDF5 calls. -One possibility is that the HDF5 library has a beta-version of Thread-safe -implmentation though it is for Pthreads environment. One can check on the -feasibility of running OpenMP programs with this version of HDF5 Thread-safe -library. - - - -<A HREF="openmp-hdf5.c">Appendix A: OpenMP-HDF5 Programs</A> - -------- -Updated: 2000/11/28 -Contact: hdfhelp@ncsa.uiuc.edu -</pre> diff --git a/doc/html/TechNotes/pipe1.gif b/doc/html/TechNotes/pipe1.gif Binary files differdeleted file mode 100644 index 3b489a6..0000000 --- a/doc/html/TechNotes/pipe1.gif +++ /dev/null diff --git a/doc/html/TechNotes/pipe1.obj b/doc/html/TechNotes/pipe1.obj deleted file mode 100644 index 41f3461..0000000 --- a/doc/html/TechNotes/pipe1.obj +++ /dev/null @@ -1,136 +0,0 @@ -%TGIF 3.0-p5 -state(1,33,100,0,0,0,8,1,9,1,1,0,0,0,0,1,1,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[ -]). -box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 160,160,144,224,160,272,176,224,160,160],1,2,1,25,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 848,160,832,224,848,272,864,224,848,160],1,2,1,34,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -box('black',464,192,496,256,26,1,1,39,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 160,224,464,224],1,2,1,40,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',2,[ - 496,224,848,224],1,2,1,41,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 192,224,176,288,192,336,208,288,192,224],1,2,1,42,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 432,224,416,288,432,336,448,288,432,224],1,2,1,43,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 192,288,432,288],1,2,1,44,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',464,352,496,416,26,1,1,45,0,0,0,0,0,'1',[ -]). -poly('black',5,[ - 528,224,512,288,528,336,544,288,528,224],1,2,1,46,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 816,224,800,288,816,336,832,288,816,224],1,2,1,47,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 528,288,816,288],1,2,1,48,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 464,256,456,304,464,328,488,304,488,256],1,2,1,62,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 480,352,488,304],2,2,1,85,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',912,64,976,256,0,1,1,87,0,0,0,0,0,'1',[ -]). -box('black',928,96,960,224,26,1,1,88,0,0,0,0,0,'1',[ -]). -text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "File"]). -text('black',944,48,'Helvetica',0,17,1,1,0,1,64,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Application"]). -text('black',480,144,'Helvetica',0,17,1,1,0,1,65,15,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5D_read()"]). -text('black',480,128,'Helvetica',0,17,1,1,0,1,58,15,108,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5Dread()"]). -text('black',304,208,'Helvetica',0,17,1,1,0,1,86,15,115,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_arr_read()"]). -text('black',304,192,'Helvetica',0,17,1,1,0,1,99,15,119,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5S_simp_fgath()"]). -text('black',296,288,'Helvetica',0,17,1,1,0,1,101,15,125,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_block_read()"]). -text('black',296,304,'Helvetica',0,17,1,1,0,1,90,15,132,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_low_read()"]). -text('black',296,320,'Helvetica',0,17,1,1,0,1,98,15,136,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_sec2_read()"]). -text('black',296,336,'Helvetica',0,17,1,1,0,1,33,15,140,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "read()"]). -text('black',664,208,'Helvetica',0,17,1,1,0,1,106,15,146,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_stride_copy()"]). -text('black',664,176,'Helvetica',0,17,1,1,0,1,104,15,150,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5S_simp_mscat()"]). -text('black',664,272,'Helvetica',0,17,1,1,0,1,54,15,154,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "memcpy()"]). -text('black',384,392,'Helvetica',0,17,1,1,0,1,105,15,170,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5T_conv_struct()"]). -poly('black',4,[ - 392,384,400,352,440,368,456,336],1,1,1,172,1,0,0,0,8,3,0,0,0,'1','8','3', - "6",[ -]). -text('black',480,176,'Helvetica',0,17,1,1,0,1,44,15,176,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "TCONV"]). -text('black',480,416,'Helvetica',0,17,1,1,0,1,25,15,182,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "BKG"]). -box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[ -]). -poly('black',5,[ - 72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "== Loop"]). -poly('black',3,[ - 48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',480,40,'Helvetica',0,24,1,1,0,1,380,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Fig 1: Internal Contiguous Storage"]). -text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "A"]). -text('black',160,208,'Helvetica',0,17,1,1,0,1,8,15,207,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "B"]). -text('black',192,272,'Helvetica',0,17,1,1,0,1,9,15,211,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "C"]). -text('black',504,208,'Helvetica',0,17,1,1,0,1,8,15,215,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "E"]). -text('black',528,272,'Helvetica',0,17,1,1,0,1,8,15,223,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "F"]). -text('black',464,304,'Helvetica',0,17,1,1,0,1,9,15,231,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "D"]). -text('black',664,192,'Helvetica',0,17,1,1,0,1,107,15,324,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_hyper_copy()"]). diff --git a/doc/html/TechNotes/pipe2.gif b/doc/html/TechNotes/pipe2.gif Binary files differdeleted file mode 100644 index 3a0c947..0000000 --- a/doc/html/TechNotes/pipe2.gif +++ /dev/null diff --git a/doc/html/TechNotes/pipe2.obj b/doc/html/TechNotes/pipe2.obj deleted file mode 100644 index 70d9c18..0000000 --- a/doc/html/TechNotes/pipe2.obj +++ /dev/null @@ -1,168 +0,0 @@ -%TGIF 3.0-p5 -state(1,33,100,0,0,0,8,1,9,1,1,1,1,0,0,1,1,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[ -]). -box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 160,160,144,224,160,272,176,224,160,160],1,2,1,25,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 848,160,832,224,848,272,864,224,848,160],1,2,1,34,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -box('black',464,192,496,256,26,1,1,39,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 160,224,464,224],1,2,1,40,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',2,[ - 496,224,848,224],1,2,1,41,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 192,224,176,288,192,336,208,288,192,224],1,2,1,42,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 432,224,416,288,432,336,448,288,432,224],1,2,1,43,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 192,288,432,288],1,2,1,44,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',464,352,496,416,26,1,1,45,0,0,0,0,0,'1',[ -]). -poly('black',5,[ - 528,224,512,288,528,336,544,288,528,224],1,2,1,46,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 816,224,800,288,816,336,832,288,816,224],1,2,1,47,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 528,288,816,288],1,2,1,48,0,26,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 848,240,848,352,832,384,800,384,496,384],1,2,1,55,1,0,0,0,10,4,0,0,0,'2','10','4', - "70",[ -]). -poly('black',5,[ - 528,384,512,448,528,496,544,448,528,384],1,2,1,57,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 800,384,784,448,800,496,816,448,800,384],1,2,1,58,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 800,448,528,448],1,2,1,61,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',5,[ - 464,256,456,304,464,328,488,304,488,256],1,2,1,62,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 480,352,488,304],0,2,1,85,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',912,64,976,256,0,1,1,87,0,0,0,0,0,'1',[ -]). -box('black',928,96,960,224,26,1,1,88,0,0,0,0,0,'1',[ -]). -text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "File"]). -text('black',944,48,'Helvetica',0,17,1,1,0,1,64,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Application"]). -text('black',480,144,'Helvetica',0,17,1,1,0,1,65,15,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5D_read()"]). -text('black',480,128,'Helvetica',0,17,1,1,0,1,58,15,108,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5Dread()"]). -text('black',304,208,'Helvetica',0,17,1,1,0,1,86,15,115,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_arr_read()"]). -text('black',304,192,'Helvetica',0,17,1,1,0,1,99,15,119,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5S_simp_fgath()"]). -text('black',296,288,'Helvetica',0,17,1,1,0,1,101,15,125,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_block_read()"]). -text('black',296,304,'Helvetica',0,17,1,1,0,1,90,15,132,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_low_read()"]). -text('black',296,320,'Helvetica',0,17,1,1,0,1,98,15,136,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_sec2_read()"]). -text('black',296,336,'Helvetica',0,17,1,1,0,1,33,15,140,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "read()"]). -text('black',664,208,'Helvetica',0,17,1,1,0,1,106,15,146,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_stride_copy()"]). -text('black',664,176,'Helvetica',0,17,1,1,0,1,104,15,150,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5S_simp_mscat()"]). -text('black',664,272,'Helvetica',0,17,1,1,0,1,54,15,154,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "memcpy()"]). -text('black',672,368,'Helvetica',0,17,1,1,0,1,106,15,158,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_stride_copy()"]). -text('black',672,336,'Helvetica',0,17,1,1,0,1,105,15,162,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5S_simp_mgath()"]). -text('black',672,432,'Helvetica',0,17,1,1,0,1,54,15,166,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "memcpy()"]). -text('black',384,392,'Helvetica',0,17,1,1,0,1,105,15,170,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5T_conv_struct()"]). -poly('black',4,[ - 392,384,400,352,440,368,456,336],1,1,1,172,1,0,0,0,8,3,0,0,0,'1','8','3', - "6",[ -]). -text('black',480,176,'Helvetica',0,17,1,1,0,1,44,15,176,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "TCONV"]). -text('black',480,416,'Helvetica',0,17,1,1,0,1,25,15,182,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "BKG"]). -box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[ -]). -poly('black',5,[ - 72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "== Loop"]). -poly('black',3,[ - 48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',480,40,'Helvetica',0,24,1,1,0,1,404,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Fig 2: Partially Initialized Destination"]). -text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "A"]). -text('black',160,208,'Helvetica',0,17,1,1,0,1,8,15,207,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "B"]). -text('black',192,272,'Helvetica',0,17,1,1,0,1,9,15,211,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "C"]). -text('black',504,208,'Helvetica',0,17,1,1,0,1,8,15,215,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "E"]). -text('black',528,272,'Helvetica',0,17,1,1,0,1,8,15,223,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "F"]). -text('black',856,288,'Helvetica',0,17,1,1,0,1,9,15,225,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "G"]). -text('black',800,432,'Helvetica',0,17,1,1,0,1,9,15,229,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H"]). -text('black',464,304,'Helvetica',0,17,1,1,0,1,9,15,231,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "D"]). -poly('black',4,[ - 848,240,848,224,864,224,904,224],0,2,1,318,1,0,0,0,10,4,0,0,0,'2','10','4', - "6",[ -]). -text('black',664,192,'Helvetica',0,17,1,1,0,1,107,15,326,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_hyper_copy()"]). -text('black',672,352,'Helvetica',0,17,1,1,0,1,107,15,334,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_hyper_copy()"]). diff --git a/doc/html/TechNotes/pipe3.gif b/doc/html/TechNotes/pipe3.gif Binary files differdeleted file mode 100644 index 26d82ad..0000000 --- a/doc/html/TechNotes/pipe3.gif +++ /dev/null diff --git a/doc/html/TechNotes/pipe3.obj b/doc/html/TechNotes/pipe3.obj deleted file mode 100644 index cdfef7c..0000000 --- a/doc/html/TechNotes/pipe3.obj +++ /dev/null @@ -1,70 +0,0 @@ -%TGIF 3.0-p5 -state(1,33,100,0,0,0,8,1,9,1,1,0,0,0,0,1,1,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[ -]). -box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',912,64,976,256,0,1,1,87,0,0,0,0,0,'1',[ -]). -box('black',928,96,960,224,26,1,1,88,0,0,0,0,0,'1',[ -]). -text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "File"]). -text('black',944,48,'Helvetica',0,17,1,1,0,1,64,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Application"]). -text('black',480,104,'Helvetica',0,17,1,1,0,1,65,15,99,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5D_read()"]). -text('black',480,88,'Helvetica',0,17,1,1,0,1,58,15,108,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5Dread()"]). -box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[ -]). -poly('black',5,[ - 72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "== Loop"]). -poly('black',3,[ - 48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',480,40,'Helvetica',0,24,1,1,0,1,295,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Fig 3: No Type Conversion"]). -text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "A"]). -poly('black',5,[ - 152,160,136,224,152,272,168,224,152,160],1,2,1,273,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',480,120,'Helvetica',0,17,1,1,0,1,96,15,277,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5S_simp_read()"]). -text('black',480,136,'Helvetica',0,17,1,1,0,1,86,15,281,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_arr_read()"]). -poly('black',5,[ - 880,160,864,224,880,272,896,224,880,160],1,2,1,283,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',2,[ - 152,224,880,224],1,2,1,286,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -text('black',480,232,'Helvetica',0,17,1,1,0,1,101,15,291,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_block_read()"]). -text('black',480,248,'Helvetica',0,17,1,1,0,1,90,15,293,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_low_read()"]). -text('black',480,264,'Helvetica',0,17,1,1,0,1,98,15,309,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_sec2_read()"]). -text('black',480,280,'Helvetica',0,17,1,1,0,1,33,15,311,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "read()"]). -text('black',176,208,'Helvetica',0,17,1,1,0,1,8,15,418,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "B"]). diff --git a/doc/html/TechNotes/pipe4.gif b/doc/html/TechNotes/pipe4.gif Binary files differdeleted file mode 100644 index a3a857b..0000000 --- a/doc/html/TechNotes/pipe4.gif +++ /dev/null diff --git a/doc/html/TechNotes/pipe4.obj b/doc/html/TechNotes/pipe4.obj deleted file mode 100644 index 6f50123..0000000 --- a/doc/html/TechNotes/pipe4.obj +++ /dev/null @@ -1,92 +0,0 @@ -%TGIF 3.0-p5 -state(1,33,100,0,0,0,8,1,9,1,1,1,2,1,0,1,0,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[ -]). -box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',912,96,944,224,26,1,1,88,0,0,0,0,0,'1',[ -]). -text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "File"]). -text('black',928,72,'Helvetica',0,17,1,1,0,1,32,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Buffer"]). -box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[ -]). -poly('black',5,[ - 72,392,56,456,72,504,88,456,72,392],1,2,1,188,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',96,448,'Helvetica',0,17,1,0,0,1,46,15,189,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "== Loop"]). -poly('black',3,[ - 48,384,152,384,152,512],0,1,1,191,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',480,40,'Helvetica',0,24,1,1,0,1,372,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Fig 4: Regularly Chunked Storage"]). -text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "A"]). -text('black',480,104,'Helvetica',0,17,1,1,0,1,86,15,281,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_arr_read()"]). -text('black',480,120,'Helvetica',0,17,1,1,0,1,102,15,349,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_istore_read()"]). -text('black',480,136,'Helvetica',0,17,1,1,0,1,167,15,351,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_istore_copy_hyperslab()"]). -poly('black',5,[ - 160,160,144,224,160,272,176,224,160,160],1,2,1,362,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -poly('black',5,[ - 880,160,864,224,880,272,896,224,880,160],1,2,1,363,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -box('black',448,192,512,256,26,1,1,364,0,0,0,0,0,'1',[ -]). -text('black',480,176,'Helvetica',0,17,1,1,0,1,43,15,367,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "CHUNK"]). -poly('black',2,[ - 160,224,448,224],1,2,1,372,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -poly('black',2,[ - 512,224,880,224],1,2,1,373,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -text('black',288,224,'Helvetica',0,17,1,1,0,1,101,15,385,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_block_read()"]). -text('black',288,240,'Helvetica',0,17,1,1,0,1,90,15,387,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_low_read()"]). -text('black',288,256,'Helvetica',0,17,1,1,0,1,98,15,391,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_sec2_read()"]). -text('black',288,272,'Helvetica',0,17,1,1,0,1,33,15,395,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "read()"]). -poly('black',5,[ - 456,256,448,296,480,320,512,296,504,256],1,2,1,401,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',184,208,'Helvetica',0,17,1,1,0,1,8,15,422,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "B"]). -text('black',520,208,'Helvetica',0,17,1,1,0,1,9,15,434,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "D"]). -text('black',440,272,'Helvetica',0,17,1,1,0,1,9,15,440,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "C"]). -text('black',480,320,'Helvetica',0,17,1,1,0,1,107,15,444,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5Z_uncompress()"]). -text('black',672,224,'Helvetica',0,17,1,1,0,1,107,15,454,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_hyper_copy()"]). -text('black',672,240,'Helvetica',0,17,1,1,0,1,106,15,464,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5V_stride_copy()"]). -text('black',672,256,'Helvetica',0,17,1,1,0,1,54,15,466,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "memcpy()"]). -text('black',168,488,'Helvetica',0,17,1,0,0,1,282,15,471,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "NOTE: H5Z_uncompress() is not implemented yet."]). diff --git a/doc/html/TechNotes/pipe5.gif b/doc/html/TechNotes/pipe5.gif Binary files differdeleted file mode 100644 index 6ae0098..0000000 --- a/doc/html/TechNotes/pipe5.gif +++ /dev/null diff --git a/doc/html/TechNotes/pipe5.obj b/doc/html/TechNotes/pipe5.obj deleted file mode 100644 index 4738bbd..0000000 --- a/doc/html/TechNotes/pipe5.obj +++ /dev/null @@ -1,52 +0,0 @@ -%TGIF 3.0-p5 -state(1,33,100,0,0,0,8,1,9,1,1,1,2,1,0,1,0,'Helvetica',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1408,1088,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -box('black',64,64,128,256,0,1,1,22,0,0,0,0,0,'1',[ -]). -box('black',80,96,112,224,26,1,1,23,0,0,0,0,0,'1',[ -]). -poly('black',2,[ - 128,160,912,160],1,2,1,24,0,0,0,0,10,4,0,0,0,'2','10','4', - "0",[ -]). -box('black',912,96,944,224,26,1,1,88,0,0,0,0,0,'1',[ -]). -text('black',96,48,'Helvetica',0,17,1,1,0,1,21,15,89,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "File"]). -text('black',928,72,'Helvetica',0,17,1,1,0,1,32,15,93,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Buffer"]). -box('black',48,32,992,512,0,1,1,186,0,0,0,0,0,'1',[ -]). -text('black',480,40,'Helvetica',0,24,1,1,0,1,333,29,197,0,24,5,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Fig 5: Reading a Single Chunk"]). -text('black',136,144,'Helvetica',0,17,1,1,0,1,9,15,201,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "A"]). -text('black',480,112,'Helvetica',0,17,1,1,0,1,86,15,281,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_arr_read()"]). -text('black',480,128,'Helvetica',0,17,1,1,0,1,102,15,349,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_istore_read()"]). -text('black',480,144,'Helvetica',0,17,1,1,0,1,167,15,351,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_istore_copy_hyperslab()"]). -text('black',480,160,'Helvetica',0,17,1,1,0,1,101,15,385,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_block_read()"]). -text('black',480,176,'Helvetica',0,17,1,1,0,1,90,15,387,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_low_read()"]). -text('black',480,192,'Helvetica',0,17,1,1,0,1,98,15,391,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5F_sec2_read()"]). -text('black',480,208,'Helvetica',0,17,1,1,0,1,33,15,395,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "read()"]). -text('black',864,240,'Helvetica',0,17,1,1,0,1,107,15,444,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "H5Z_uncompress()"]). -text('black',56,488,'Helvetica',0,17,1,0,0,1,282,15,471,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "NOTE: H5Z_uncompress() is not implemented yet."]). -poly('black',5,[ - 912,176,864,176,840,208,872,232,912,216],1,2,1,490,2,0,0,0,10,4,0,0,0,'2','10','4', - "",[ -]). -text('black',896,184,'Helvetica',0,17,1,0,0,1,8,15,491,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "B"]). diff --git a/doc/html/TechNotes/shuffling-algorithm-report.pdf b/doc/html/TechNotes/shuffling-algorithm-report.pdf Binary files differdeleted file mode 100755 index 459653c..0000000 --- a/doc/html/TechNotes/shuffling-algorithm-report.pdf +++ /dev/null diff --git a/doc/html/TechNotes/version.gif b/doc/html/TechNotes/version.gif Binary files differdeleted file mode 100644 index 41d4401..0000000 --- a/doc/html/TechNotes/version.gif +++ /dev/null diff --git a/doc/html/TechNotes/version.obj b/doc/html/TechNotes/version.obj deleted file mode 100644 index 96b5b7f..0000000 --- a/doc/html/TechNotes/version.obj +++ /dev/null @@ -1,96 +0,0 @@ -%TGIF 3.0-p5 -state(0,33,100,0,0,0,8,1,9,1,1,0,2,1,0,1,0,'Courier',0,17,0,0,0,10,0,0,1,1,0,16,0,0,1,1,1,0,1088,1408,0,0,2880). -% -% @(#)$Header$ -% %W% -% -unit("1 pixel/pixel"). -page(1,"",1). -poly('black',2,[ - 128,128,128,448],0,3,1,0,0,0,0,0,12,5,0,0,0,'3','12','5', - "0",[ -]). -poly('black',2,[ - 128,128,128,64],0,3,1,1,0,0,2,0,12,5,0,0,0,'3','12','5', - "0",[ -]). -poly('black',2,[ - 128,448,128,512],0,3,1,4,0,0,2,0,12,5,0,0,0,'3','12','5', - "0",[ -]). -text('black',144,112,'Courier',0,17,1,0,0,1,42,14,22,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.3.30"]). -text('black',144,144,'Courier',0,17,1,0,0,1,42,14,30,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.3.31"]). -text('black',144,176,'Courier',0,17,1,0,0,1,42,14,32,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.3.32"]). -poly('black',2,[ - 256,208,256,448],0,3,1,34,0,0,0,0,12,5,0,0,0,'3','12','5', - "0",[ -]). -poly('black',2,[ - 256,448,256,512],0,3,1,36,0,0,2,0,12,5,0,0,0,'3','12','5', - "0",[ -]). -poly('black',2,[ - 128,192,256,208],1,1,1,37,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',144,224,'Courier',0,17,1,0,0,1,42,14,41,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.3.33"]). -text('black',144,256,'Courier',0,17,1,0,0,1,42,14,43,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.3.34"]). -text('black',272,224,'Courier',0,17,1,0,0,1,35,14,45,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.5.0"]). -text('black',272,256,'Courier',0,17,1,0,0,1,35,14,47,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.5.1"]). -text('black',272,288,'Courier',0,17,1,0,0,1,35,14,49,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.5.2"]). -text('black',272,320,'Courier',0,17,1,0,0,1,35,14,51,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.5.3"]). -text('black',144,288,'Courier',0,17,1,0,0,1,42,14,53,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.3.35"]). -text('black',144,320,'Courier',0,17,1,0,0,1,35,14,57,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.4.0"]). -text('black',144,368,'Courier',0,17,1,0,0,1,35,14,59,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.4.1"]). -text('black',272,192,'Helvetica',0,17,1,0,0,1,144,15,67,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "New development branch"]). -text('black',144,64,'Helvetica',0,17,1,0,0,1,163,15,69,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Original development branch"]). -text('black',16,208,'Helvetica',0,17,2,0,0,1,87,30,71,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Feature Freeze", - "at this point."]). -text('black',16,320,'Helvetica',0,17,2,0,0,1,84,30,73,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Public Release", - "at this point."]). -poly('black',2,[ - 104,208,128,208],1,1,1,77,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 104,320,128,320],1,1,1,78,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -poly('black',2,[ - 256,336,128,352],1,1,1,79,0,0,0,0,8,3,0,0,0,'1','8','3', - "0",[ -]). -text('black',320,368,'Helvetica',0,17,3,0,0,1,137,45,82,0,12,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "Merge a bug fix from the", - "development branch to", - "the release branch."]). -box('black',312,368,464,416,0,1,1,87,0,0,0,0,0,'1',[ -]). -poly('black',4,[ - 312,392,240,384,296,344,232,344],1,1,1,90,1,0,0,0,8,3,0,0,0,'1','8','3', - "6",[ -]). -box('black',8,208,104,240,0,1,1,95,0,0,0,0,0,'1',[ -]). -box('black',8,320,104,352,0,1,1,98,0,0,0,0,0,'1',[ -]). -text('black',144,408,'Courier',0,17,1,0,0,1,35,14,102,0,11,3,0,0,0,0,0,2,0,0,0,0,"",0,0,0,[ - "1.4.2"]). -box('black',0,40,480,528,0,1,1,104,0,0,0,0,0,'1',[ -]). |
