[svn-r13757]

h5repack and h5diff hyperslab I/O

changed the limit on which  hyperslab I/O is done from 1GB to 128MB

h5repack currently tests this feature by defining a dataset with dimensions of 128MB + 1byte (the datum being 1 byte integer), in which a 1Kb hyperslab was written.

tested: linux
there were some errors in running the tests, but these were in the library, due to a recent check-in that will be fixed shortly (at least we hope :-) )

1 files changed, 42 insertions, 2 deletions

diff --git a/tools/h5repack/h5repack_copy.c b/tools/h5repack/h5repack_copy.c
index 4253678..54e9dd2 100644
--- a/tools/h5repack/h5repack_copy.c
+++ b/tools/h5repack/h5repack_copy.c
@@ -155,7 +155,47 @@ out:
  *                  with the requested filter, use the input one
  *
  *  October 2006:  Read/write using the file type by default.
- *                 Read/write by hyperslabs for big datasets.
+ *
+ *  October 2006:  Read by hyperslabs for big datasets.
+ *
+ *  A threshold of H5TOOLS_MALLOCSIZE (128 MB) is the limit upon which I/O hyperslab is done
+ *  i.e., if the memory needed to read a dataset is greater than this limit, 
+ *  then hyperslab I/O is done instead of one operation I/O 
+ *  For each dataset, the memory needed is calculated according to
+ *
+ *  memory needed = number of elements * size of each element
+ *
+ *  if the memory needed is lower than H5TOOLS_MALLOCSIZE, then the following operations 
+ *  are done
+ *
+ *  H5Dread( input_dataset1 )
+ *  H5Dread( input_dataset2 )
+ *
+ *  with all elements in the datasets selected. If the memory needed is greater than 
+ *  H5TOOLS_MALLOCSIZE, then the following operations are done instead:
+ *
+ *  a strip mine is defined for each dimension k (a strip mine is defined as a 
+ *  hyperslab whose size is memory manageable) according to the formula
+ *
+ *  (1) strip_mine_size[k ] = MIN(dimension[k ], H5TOOLS_BUFSIZE / size of memory type)
+ *
+ *  where H5TOOLS_BUFSIZE is a constant currently defined as 1MB. This formula assures 
+ *  that for small datasets (small relative to the H5TOOLS_BUFSIZE constant), the strip 
+ *  mine size k is simply defined as its dimension k, but for larger datasets the 
+ *  hyperslab size is still memory manageable.
+ *  a cycle is done until the number of elements in the dataset is reached. In each 
+ *  iteration, two parameters are defined for the function H5Sselect_hyperslab, 
+ *  the start and size of each hyperslab, according to
+ *
+ *  (2) hyperslab_size [k] = MIN(dimension[k] - hyperslab_offset[k], strip_mine_size [k])
+ *
+ *  where hyperslab_offset [k] is initially set to zero, and later incremented in 
+ *  hyperslab_size[k] offsets. The reason for the operation 
+ *
+ *  dimension[k] - hyperslab_offset[k]
+ *
+ *  in (2) is that, when using the strip mine size, it assures that the "remaining" part 
+ *  of the dataset that does not fill an entire strip mine is processed.
  *
  *  November 2006:  Use H5Ocopy in the copy of objects. The logic for using 
  *   H5Ocopy or not is if a change of filters or layout is requested by the user 
@@ -273,7 +313,7 @@ int do_copy_objects(hid_t fidin,
        options->all_filter==1 || options->all_layout==1 
        || is_ref
        )
-   {
+   {       
     int j;
 
     if ((dset_in=H5Dopen(fidin,travt->objs[i].name))<0)