[svn-r13757]

h5repack and h5diff hyperslab I/O

changed the limit on which  hyperslab I/O is done from 1GB to 128MB

h5repack currently tests this feature by defining a dataset with dimensions of 128MB + 1byte (the datum being 1 byte integer), in which a 1Kb hyperslab was written.

tested: linux
there were some errors in running the tests, but these were in the library, due to a recent check-in that will be fixed shortly (at least we hope :-) )

2 files changed, 45 insertions, 1 deletions

diff --git a/tools/lib/h5diff_dset.c b/tools/lib/h5diff_dset.c
index ef6d39f..c714d45 100644
--- a/tools/lib/h5diff_dset.c
+++ b/tools/lib/h5diff_dset.c
@@ -152,6 +152,50 @@ error:
  *
  * Date: May 9, 2003
  *
+ * Modifications: 
+ *
+ *
+ * October 2006:  Read by hyperslabs for big datasets.
+ *
+ *  A threshold of H5TOOLS_MALLOCSIZE (128 MB) is the limit upon which I/O hyperslab is done
+ *  i.e., if the memory needed to read a dataset is greater than this limit, 
+ *  then hyperslab I/O is done instead of one operation I/O 
+ *  For each dataset, the memory needed is calculated according to
+ *
+ *  memory needed = number of elements * size of each element
+ *
+ *  if the memory needed is lower than H5TOOLS_MALLOCSIZE, then the following operations 
+ *  are done
+ *
+ *  H5Dread( input_dataset1 )
+ *  H5Dread( input_dataset2 )
+ *
+ *  with all elements in the datasets selected. If the memory needed is greater than 
+ *  H5TOOLS_MALLOCSIZE, then the following operations are done instead:
+ *
+ *  a strip mine is defined for each dimension k (a strip mine is defined as a 
+ *  hyperslab whose size is memory manageable) according to the formula
+ *
+ *  (1) strip_mine_size[k ] = MIN(dimension[k ], H5TOOLS_BUFSIZE / size of memory type)
+ *
+ *  where H5TOOLS_BUFSIZE is a constant currently defined as 1MB. This formula assures 
+ *  that for small datasets (small relative to the H5TOOLS_BUFSIZE constant), the strip 
+ *  mine size k is simply defined as its dimension k, but for larger datasets the 
+ *  hyperslab size is still memory manageable.
+ *  a cycle is done until the number of elements in the dataset is reached. In each 
+ *  iteration, two parameters are defined for the function H5Sselect_hyperslab, 
+ *  the start and size of each hyperslab, according to
+ *
+ *  (2) hyperslab_size [k] = MIN(dimension[k] - hyperslab_offset[k], strip_mine_size [k])
+ *
+ *  where hyperslab_offset [k] is initially set to zero, and later incremented in 
+ *  hyperslab_size[k] offsets. The reason for the operation 
+ *
+ *  dimension[k] - hyperslab_offset[k]
+ *
+ *  in (2) is that, when using the strip mine size, it assures that the "remaining" part 
+ *  of the dataset that does not fill an entire strip mine is processed.
+ *
  *-------------------------------------------------------------------------
  */
 hsize_t diff_datasetid( hid_t did1,
diff --git a/tools/lib/h5tools.h b/tools/lib/h5tools.h
index fa05a9f..a0f54736 100644
--- a/tools/lib/h5tools.h
+++ b/tools/lib/h5tools.h
@@ -45,7 +45,7 @@
 /*
  * Maximum size used in a call to malloc
  */
-#define H5TOOLS_MALLOCSIZE      ((size_t)1024 * 1024 * 1024)
+#define H5TOOLS_MALLOCSIZE      (128 * 1024 * 1024)
 
 /* format for hsize_t */
 #define HSIZE_T_FORMAT   "%"H5_PRINTF_LL_WIDTH"u"