Multi Dataset (#2120)

* Fix bug with cross platform compatibility of references within vlens.
No testing yet.

* Merge from multi_rd_wd_coll_io to a more recent branch from develop.
Untested, probably does not work yet.

* Committing clang-format changes

* Committing clang-format changes

* Fix many bugs in multi dataset branch.  Mostly works, some issues in
SWMR tests.

* Committing clang-format changes

* Disable test in swmr.c that was failing due to bug in HDF5 unrelated to
multi dataset.

* Committing clang-format changes

* Fixed fortran multi-dataset tests

* Fixed xlf errors

* Added benchmark code for multi-datasets

* loops over datasets

* added missing error arg.

* Added gnuplot formatting

* Jonathan Kim original MD benchmarking code

* updated MD benchmarking code

* code clean-up

* Only make files in feature test mode

* misc clean-up

* removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option

* Committing clang-format changes

* Change multi dataset API to use arrays of individual parameters instead
of the parameter struct.

* Committing clang-format changes

* Update to new multi dataset Fortran API and tests. (#1724)

* Update to new multi dataset Fortran API and tests.
* Sync Fortran with develop.
* skipping h5pget_mpio_actual_io_mode_f for now

* Fixed issue with dxpl_id, changed to variable size dim. (#1770)

* Remove "is_coll_broken" field from H5D_io_info_t struct

* Committing clang-format changes

* Minor cleanup in multi dataset code.

* Committing clang-format changes

* Clean up in multi dataset code.

* Committing clang-format changes

* Committing clang-format changes

* Fix speeling

* Fix bug in parallel compression. Switch base_maddr in io_info to be a
union.

* Committing clang-format changes

* Implement selection I/O support with multi dataset.  Will be broken in
parallel until PR 1803 is merged to develop then the MDS branch.

* Committing clang-format changes

* Spelling

* Fix bug in multi dataset that could cause errors when only some of the
datasets in the multi dataset I/O used type conversion.

* Committing clang-format changes

* Integrate multi dataset APIs with VOL layer.  Add async versions of
multi dataset APIs.

* Committing clang-format changes

* Spelling fixes

* Fix bug in non-parallel HDF5 compilation.

* Committing clang-format changes

* Fix potential memory/free list error. Minor performance fix. Other minor
changes.

* Committing clang-format changes

* Fix memory leak with memory dataspace for I/O.

* Committing clang-format changes

* Fix stack variables too large.  Rename H5D_dset_info_t to
H5D_dset_io_info_t.

* Committing clang-format changes

* Remove mem_space_alloc field from H5D_dset_io_info_t.  Each function is
now responsible for freeing any spaces it adds to dset_info.

* Committing clang-format changes

* fixed _multi Fortran declaration

* Refactor various things in (mostly) the serial I/O code path to make
things more maintainable.

* Committing clang-format changes

* updated to array based, doxygen, and examples

* Reinstate H5D_chunk_map_t, stored (via pointer) inside
H5D_dset_io_info_t.

* Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t.
Switch temporary dset_infos to be local stack variables.

* Committing clang-format changes

* format cleanup

* format cleanup

* added coll and ind

* Modify all parallel I/O paths to take dset_info instead of assuming
dset_info[0].

* Committing clang-format changes

* fixed output

* Rework parallel I/O code to work properly with multi dataset in more
cases.  Fix bug in parallel compression.

* Committing clang-format changes

* Prevent H5D__multi_chunk_collective_io() from messing up collective opt
property for other datasets in I/O.  Other minor cleanup.  Add new test
case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now
due to failures apparently unrelated to multi dataset code.

* Fix spelling

* Committing clang-format changes

* Replace N log N algorithm for finding chunk in
H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of
io_info->sel_pieces in that function.

* Committing clang-format changes

* Replace sel_pieces skiplist in io_info with flat array of pointers, use
qsort in I/O routine only when necessary.

* Committing clang-format changes

* Add new test case to mdset.c

* Committing clang-format changes

* Fix spelling

* Very minor fix in H5VL__native_dataset_read()

* Fix bug that could affect filtered parallel multi-dataset I/O.

* Add RM entries for H5Dread_multi(), H5Dread_multi_async(),
H5Dwrite_multi(), and H5Dwrite_multi_async()

* Unskip test in swmr.c

* Committing clang-format changes

* Eliminate H5D__pre_read and H5D__pre_write

* Remove examples/ph5mdsettest.c. Will fix and re-add as a test.

* Enable hyperslab combinations in mdset test

* Committing clang-format changes

* Clarify H5Dread/write_multi documentation.

* Fix bugs in multi-dataset I/O.  Expand serial multi dataset test.
Update macro in parallel multi dataset test.

* Committing clang-format changes

* Spelling

* Remove obsolete entry in bin/trace

* Rework type conversion buffer allocation. Only one buffer is shared
between datasets in mdset mode, and it is malloced instead of calloced.

* Committing clang-format changes

* Fix bug in error handling in H5D__read/write

* added multi-dataset fortran check with optional dataset creation id (#2150)

* removed dup. dll entry

* Address comments from code review.

* Remove spurious changes in H5Fmpi.c

* Fix issue with reading unallocated datasets in multi-dataset mode.
Address other comments from code review.

* Committing clang-format changes

* Delay chunk index lookup from io_init to mdio_init so it doesn't add
overhead to single dataset I/O.

* Committing clang-format changes

* Fix inappropriate use of piece_count

* updated copyright on new file, removed benchmark from testing dir.

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>

1 files changed, 62 insertions, 19 deletions

diff --git a/src/H5VLpassthru.c b/src/H5VLpassthru.c
index af79284..d7e730a 100644
--- a/src/H5VLpassthru.c
+++ b/src/H5VLpassthru.c
@@ -121,11 +121,12 @@ static void  *H5VL_pass_through_dataset_create(void *obj, const H5VL_loc_params_
                                                hid_t dcpl_id, hid_t dapl_id, hid_t dxpl_id, void **req);
 static void  *H5VL_pass_through_dataset_open(void *obj, const H5VL_loc_params_t *loc_params, const char *name,
                                              hid_t dapl_id, hid_t dxpl_id, void **req);
-static herr_t H5VL_pass_through_dataset_read(void *dset, hid_t mem_type_id, hid_t mem_space_id,
-                                             hid_t file_space_id, hid_t plist_id, void *buf, void **req);
-static herr_t H5VL_pass_through_dataset_write(void *dset, hid_t mem_type_id, hid_t mem_space_id,
-                                              hid_t file_space_id, hid_t plist_id, const void *buf,
-                                              void **req);
+static herr_t H5VL_pass_through_dataset_read(size_t count, void *dset[], hid_t mem_type_id[],
+                                             hid_t mem_space_id[], hid_t file_space_id[], hid_t plist_id,
+                                             void *buf[], void **req);
+static herr_t H5VL_pass_through_dataset_write(size_t count, void *dset[], hid_t mem_type_id[],
+                                              hid_t mem_space_id[], hid_t file_space_id[], hid_t plist_id,
+                                              const void *buf[], void **req);
 static herr_t H5VL_pass_through_dataset_get(void *dset, H5VL_dataset_get_args_t *args, hid_t dxpl_id,
                                             void **req);
 static herr_t H5VL_pass_through_dataset_specific(void *obj, H5VL_dataset_specific_args_t *args, hid_t dxpl_id,
@@ -1197,22 +1198,43 @@ H5VL_pass_through_dataset_open(void *obj, const H5VL_loc_params_t *loc_params, c
  *-------------------------------------------------------------------------
  */
 static herr_t
-H5VL_pass_through_dataset_read(void *dset, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id,
-                               hid_t plist_id, void *buf, void **req)
+H5VL_pass_through_dataset_read(size_t count, void *dset[], hid_t mem_type_id[], hid_t mem_space_id[],
+                               hid_t file_space_id[], hid_t plist_id, void *buf[], void **req)
 {
-    H5VL_pass_through_t *o = (H5VL_pass_through_t *)dset;
-    herr_t               ret_value;
+    void  *obj_local;        /* Local buffer for obj */
+    void **obj = &obj_local; /* Array of object pointers */
+    size_t i;                /* Local index variable */
+    herr_t ret_value;
 
 #ifdef ENABLE_PASSTHRU_LOGGING
     printf("------- PASS THROUGH VOL DATASET Read\n");
 #endif
 
-    ret_value = H5VLdataset_read(o->under_object, o->under_vol_id, mem_type_id, mem_space_id, file_space_id,
-                                 plist_id, buf, req);
+    /* Allocate obj array if necessary */
+    if (count > 1)
+        if (NULL == (obj = (void **)malloc(count * sizeof(void *))))
+            return -1;
+
+    /* Build obj array */
+    for (i = 0; i < count; i++) {
+        /* Get the object */
+        obj[i] = ((H5VL_pass_through_t *)dset[i])->under_object;
+
+        /* Make sure the class matches */
+        if (((H5VL_pass_through_t *)dset[i])->under_vol_id != ((H5VL_pass_through_t *)dset[0])->under_vol_id)
+            return -1;
+    }
+
+    ret_value = H5VLdataset_read(count, obj, ((H5VL_pass_through_t *)dset[0])->under_vol_id, mem_type_id,
+                                 mem_space_id, file_space_id, plist_id, buf, req);
 
     /* Check for async request */
     if (req && *req)
-        *req = H5VL_pass_through_new_obj(*req, o->under_vol_id);
+        *req = H5VL_pass_through_new_obj(*req, ((H5VL_pass_through_t *)dset[0])->under_vol_id);
+
+    /* Free memory */
+    if (obj != &obj_local)
+        free(obj);
 
     return ret_value;
 } /* end H5VL_pass_through_dataset_read() */
@@ -1228,22 +1250,43 @@ H5VL_pass_through_dataset_read(void *dset, hid_t mem_type_id, hid_t mem_space_id
  *-------------------------------------------------------------------------
  */
 static herr_t
-H5VL_pass_through_dataset_write(void *dset, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id,
-                                hid_t plist_id, const void *buf, void **req)
+H5VL_pass_through_dataset_write(size_t count, void *dset[], hid_t mem_type_id[], hid_t mem_space_id[],
+                                hid_t file_space_id[], hid_t plist_id, const void *buf[], void **req)
 {
-    H5VL_pass_through_t *o = (H5VL_pass_through_t *)dset;
-    herr_t               ret_value;
+    void  *obj_local;        /* Local buffer for obj */
+    void **obj = &obj_local; /* Array of object pointers */
+    size_t i;                /* Local index variable */
+    herr_t ret_value;
 
 #ifdef ENABLE_PASSTHRU_LOGGING
     printf("------- PASS THROUGH VOL DATASET Write\n");
 #endif
 
-    ret_value = H5VLdataset_write(o->under_object, o->under_vol_id, mem_type_id, mem_space_id, file_space_id,
-                                  plist_id, buf, req);
+    /* Allocate obj array if necessary */
+    if (count > 1)
+        if (NULL == (obj = (void **)malloc(count * sizeof(void *))))
+            return -1;
+
+    /* Build obj array */
+    for (i = 0; i < count; i++) {
+        /* Get the object */
+        obj[i] = ((H5VL_pass_through_t *)dset[i])->under_object;
+
+        /* Make sure the class matches */
+        if (((H5VL_pass_through_t *)dset[i])->under_vol_id != ((H5VL_pass_through_t *)dset[0])->under_vol_id)
+            return -1;
+    }
+
+    ret_value = H5VLdataset_write(count, obj, ((H5VL_pass_through_t *)dset[0])->under_vol_id, mem_type_id,
+                                  mem_space_id, file_space_id, plist_id, buf, req);
 
     /* Check for async request */
     if (req && *req)
-        *req = H5VL_pass_through_new_obj(*req, o->under_vol_id);
+        *req = H5VL_pass_through_new_obj(*req, ((H5VL_pass_through_t *)dset[0])->under_vol_id);
+
+    /* Free memory */
+    if (obj != &obj_local)
+        free(obj);
 
     return ret_value;
 } /* end H5VL_pass_through_dataset_write() */