[svn-r16614] 3. #1501 (B1) tools bug if dataset is larger than H5TOOLS_BUFSIZE limit.

ISSUE : the tools use the following formula to read by hyperslabs: hyperslab_size[i] = MIN( dim_size[i], H5TOOLS_BUFSIZE / datum_size) where H5TOOLS_BUFSIZE is a constant defined of 1024K. This is OK as long as the datum_size does not exceed 1024K, otherwise we have a hyperslab size of 0 (since 1024K/(greater than 1024K) = 0). This affects h5dump. h5repack, h5diff
SOLUTION: add a check for a 0 size and define as 1 if so. 
TEST FOR H5DUMP: Defined a case in the h5dump test generator program of such a type (an array type of doubles with a large array dimension, that was the case the user reported).  Since the written file commited in svn would be around 1024K, opted for not writing the data (the part of the code where the hyperslab is defined is executed, since h5dump always reads the files). Defined a macro WRITE_ARRAY to enable such writing if needed. Added a run on the h5dump shell script. Added 2 new files to svn: tools/testfiles/tarray8.ddl, tools/testfiles/tarray8.h5. NOTE: while doing this I thought of adding this dataset case to an existing file, but that would add the large array output to those files (the ddls). The issue is that the file list is increasing.
TEST FOR H5DIFF: for h5diff the check for reading by hyperslabs is H5TOOLS_MALLOCSIZE (128 * H5TOOLS_BUFSIZE) or 128 Mb. This makes it not possible to add such a file to svn, so used the same method as h5dump (only write the dataset if WRITE_ARRAY is defined). As opposed to h5dump, the hyperslab code is NOT executed when the dataset is empty (dataset is not read). Added the new dataset to existing files and shell run (tools/h5diff/testfiles/h5diff_dset1.h5 and tools/h5diff/testfiles/h5diff_dset2.h5 and output in tools/h5diff/testfiles/h5diff_80.txt).
TEST FOR H5REPACK: similar issue as h5diff with the difference that the hyperslab code is run. Added a run to the shell script (with a filter, otherwise the code uses H5Ocopy). 

tested: linux (h5commitest failed , apparently it did not detect the code changes in /tools/lib that fix the bug: the error in an assertion in the hyperslab of 0. I am sure that making h5ccomitest --distclean will detect the new code , but don't want to wait more 3 hours :-) )

2 files changed, 50 insertions, 0 deletions

diff --git a/tools/h5dump/h5dumpgentest.c b/tools/h5dump/h5dumpgentest.c
index fc75681..4d811ee 100644
--- a/tools/h5dump/h5dumpgentest.c
+++ b/tools/h5dump/h5dumpgentest.c
@@ -91,6 +91,7 @@
 #define FILE61  "textlinksrc.h5"
 #define FILE62  "textlinktar.h5"
 #define FILE63  "textlinkfar.h5"
+#define FILE64  "tarray8.h5"
 
 
 
@@ -2812,6 +2813,51 @@ static void gent_array7(void)
     assert(ret >= 0);
 }
 
+static void gent_array8(void)
+{
+    double  *buf;                            /* information to write */
+    hid_t   fid;                             /* HDF5 File IDs  */
+    hid_t   did;                             /* dataset ID   */
+    hid_t   sid;                             /* dataspace ID   */
+    hid_t   tid;                             /* datatype ID   */
+    size_t  size;
+    hsize_t sdims[] = {1};
+    hsize_t tdims[] = {H5TOOLS_BUFSIZE / sizeof(double) + 1};
+    int     i;                               
+    herr_t  ret;                             
+
+    size = ( H5TOOLS_BUFSIZE / sizeof(double) + 1 ) * sizeof(double);
+    buf = malloc( size );
+
+    for( i = 0; i < H5TOOLS_BUFSIZE / sizeof(double) + 1; i++)
+        buf[i] = i;
+
+    /* create file */
+    fid = H5Fcreate(FILE64, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
+
+    /* create a type larger than H5TOOLS_BUFSIZE */
+    tid = H5Tarray_create2(H5T_NATIVE_DOUBLE, 1, tdims);
+    size = H5Tget_size(tid);
+    sid = H5Screate_simple(1, sdims, NULL);
+    did = H5Dcreate2(fid, "dset", tid, sid, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
+#if defined(WRITE_ARRAY)
+    ret = H5Dwrite(did, tid, H5S_ALL, H5S_ALL, H5P_DEFAULT, buf);
+    assert(ret >= 0);
+#endif
+
+    /* close */
+    ret = H5Dclose(did);
+    assert(ret >= 0);
+    ret = H5Tclose(tid);
+    assert(ret >= 0);
+    ret = H5Sclose(sid);
+    assert(ret >= 0);
+
+    ret = H5Fclose(fid);
+    assert(ret >= 0);
+    free( buf );
+}
+
 static void gent_empty(void)
 {
     typedef struct {
@@ -6330,6 +6376,7 @@ int main(void)
     gent_attr_creation_order();
     gent_fpformat();
     gent_extlinks();
+    gent_array8();
 
 
     return 0;
diff --git a/tools/h5dump/testh5dump.sh.in b/tools/h5dump/testh5dump.sh.in
index 69a7a07..c85d34d 100644
--- a/tools/h5dump/testh5dump.sh.in
+++ b/tools/h5dump/testh5dump.sh.in
@@ -511,6 +511,9 @@ TOOLTEST tfpformat.ddl -m %.7f tfpformat.h5
 TOOLTEST textlinksrc.ddl textlinksrc.h5
 TOOLTEST textlinkfar.ddl textlinkfar.h5
 
+# test for datum size > H5TOOLS_BUFSIZE
+TOOLTEST tarray8.ddl -d dset tarray8.h5
+
 
 
 if test $nerrors -eq 0 ; then