Multi Dataset (#2120)

* Fix bug with cross platform compatibility of references within vlens.
No testing yet.

* Merge from multi_rd_wd_coll_io to a more recent branch from develop.
Untested, probably does not work yet.

* Committing clang-format changes

* Committing clang-format changes

* Fix many bugs in multi dataset branch.  Mostly works, some issues in
SWMR tests.

* Committing clang-format changes

* Disable test in swmr.c that was failing due to bug in HDF5 unrelated to
multi dataset.

* Committing clang-format changes

* Fixed fortran multi-dataset tests

* Fixed xlf errors

* Added benchmark code for multi-datasets

* loops over datasets

* added missing error arg.

* Added gnuplot formatting

* Jonathan Kim original MD benchmarking code

* updated MD benchmarking code

* code clean-up

* Only make files in feature test mode

* misc clean-up

* removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option

* Committing clang-format changes

* Change multi dataset API to use arrays of individual parameters instead
of the parameter struct.

* Committing clang-format changes

* Update to new multi dataset Fortran API and tests. (#1724)

* Update to new multi dataset Fortran API and tests.
* Sync Fortran with develop.
* skipping h5pget_mpio_actual_io_mode_f for now

* Fixed issue with dxpl_id, changed to variable size dim. (#1770)

* Remove "is_coll_broken" field from H5D_io_info_t struct

* Committing clang-format changes

* Minor cleanup in multi dataset code.

* Committing clang-format changes

* Clean up in multi dataset code.

* Committing clang-format changes

* Committing clang-format changes

* Fix speeling

* Fix bug in parallel compression. Switch base_maddr in io_info to be a
union.

* Committing clang-format changes

* Implement selection I/O support with multi dataset.  Will be broken in
parallel until PR 1803 is merged to develop then the MDS branch.

* Committing clang-format changes

* Spelling

* Fix bug in multi dataset that could cause errors when only some of the
datasets in the multi dataset I/O used type conversion.

* Committing clang-format changes

* Integrate multi dataset APIs with VOL layer.  Add async versions of
multi dataset APIs.

* Committing clang-format changes

* Spelling fixes

* Fix bug in non-parallel HDF5 compilation.

* Committing clang-format changes

* Fix potential memory/free list error. Minor performance fix. Other minor
changes.

* Committing clang-format changes

* Fix memory leak with memory dataspace for I/O.

* Committing clang-format changes

* Fix stack variables too large.  Rename H5D_dset_info_t to
H5D_dset_io_info_t.

* Committing clang-format changes

* Remove mem_space_alloc field from H5D_dset_io_info_t.  Each function is
now responsible for freeing any spaces it adds to dset_info.

* Committing clang-format changes

* fixed _multi Fortran declaration

* Refactor various things in (mostly) the serial I/O code path to make
things more maintainable.

* Committing clang-format changes

* updated to array based, doxygen, and examples

* Reinstate H5D_chunk_map_t, stored (via pointer) inside
H5D_dset_io_info_t.

* Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t.
Switch temporary dset_infos to be local stack variables.

* Committing clang-format changes

* format cleanup

* format cleanup

* added coll and ind

* Modify all parallel I/O paths to take dset_info instead of assuming
dset_info[0].

* Committing clang-format changes

* fixed output

* Rework parallel I/O code to work properly with multi dataset in more
cases.  Fix bug in parallel compression.

* Committing clang-format changes

* Prevent H5D__multi_chunk_collective_io() from messing up collective opt
property for other datasets in I/O.  Other minor cleanup.  Add new test
case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now
due to failures apparently unrelated to multi dataset code.

* Fix spelling

* Committing clang-format changes

* Replace N log N algorithm for finding chunk in
H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of
io_info->sel_pieces in that function.

* Committing clang-format changes

* Replace sel_pieces skiplist in io_info with flat array of pointers, use
qsort in I/O routine only when necessary.

* Committing clang-format changes

* Add new test case to mdset.c

* Committing clang-format changes

* Fix spelling

* Very minor fix in H5VL__native_dataset_read()

* Fix bug that could affect filtered parallel multi-dataset I/O.

* Add RM entries for H5Dread_multi(), H5Dread_multi_async(),
H5Dwrite_multi(), and H5Dwrite_multi_async()

* Unskip test in swmr.c

* Committing clang-format changes

* Eliminate H5D__pre_read and H5D__pre_write

* Remove examples/ph5mdsettest.c. Will fix and re-add as a test.

* Enable hyperslab combinations in mdset test

* Committing clang-format changes

* Clarify H5Dread/write_multi documentation.

* Fix bugs in multi-dataset I/O.  Expand serial multi dataset test.
Update macro in parallel multi dataset test.

* Committing clang-format changes

* Spelling

* Remove obsolete entry in bin/trace

* Rework type conversion buffer allocation. Only one buffer is shared
between datasets in mdset mode, and it is malloced instead of calloced.

* Committing clang-format changes

* Fix bug in error handling in H5D__read/write

* added multi-dataset fortran check with optional dataset creation id (#2150)

* removed dup. dll entry

* Address comments from code review.

* Remove spurious changes in H5Fmpi.c

* Fix issue with reading unallocated datasets in multi-dataset mode.
Address other comments from code review.

* Committing clang-format changes

* Delay chunk index lookup from io_init to mdio_init so it doesn't add
overhead to single dataset I/O.

* Committing clang-format changes

* Fix inappropriate use of piece_count

* updated copyright on new file, removed benchmark from testing dir.

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>

1 files changed, 318 insertions, 100 deletions

diff --git a/src/H5Dcontig.c b/src/H5Dcontig.c
index 6408aaf..ce1bef4 100644
--- a/src/H5Dcontig.c
+++ b/src/H5Dcontig.c
@@ -91,19 +91,24 @@ typedef struct H5D_contig_writevv_ud_t {
 /* Layout operation callbacks */
 static herr_t  H5D__contig_construct(H5F_t *f, H5D_t *dset);
 static herr_t  H5D__contig_init(H5F_t *f, const H5D_t *dset, hid_t dapl_id);
-static herr_t  H5D__contig_io_init(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts,
-                                   H5S_t *file_space, H5S_t *mem_space, H5D_chunk_map_t *cm);
-static ssize_t H5D__contig_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq,
-                                  size_t dset_len_arr[], hsize_t dset_offset_arr[], size_t mem_max_nseq,
-                                  size_t *mem_curr_seq, size_t mem_len_arr[], hsize_t mem_offset_arr[]);
-static ssize_t H5D__contig_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq,
-                                   size_t dset_len_arr[], hsize_t dset_offset_arr[], size_t mem_max_nseq,
-                                   size_t *mem_curr_seq, size_t mem_len_arr[], hsize_t mem_offset_arr[]);
+static herr_t  H5D__contig_io_init(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo);
+static herr_t  H5D__contig_mdio_init(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo);
+static ssize_t H5D__contig_readvv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dinfo,
+                                  size_t dset_max_nseq, size_t *dset_curr_seq, size_t dset_len_arr[],
+                                  hsize_t dset_offset_arr[], size_t mem_max_nseq, size_t *mem_curr_seq,
+                                  size_t mem_len_arr[], hsize_t mem_offset_arr[]);
+static ssize_t H5D__contig_writevv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dinfo,
+                                   size_t dset_max_nseq, size_t *dset_curr_seq, size_t dset_len_arr[],
+                                   hsize_t dset_offset_arr[], size_t mem_max_nseq, size_t *mem_curr_seq,
+                                   size_t mem_len_arr[], hsize_t mem_offset_arr[]);
 static herr_t  H5D__contig_flush(H5D_t *dset);
+static herr_t  H5D__contig_io_term(H5D_io_info_t *io_info, H5D_dset_io_info_t *di);
 
 /* Helper routines */
-static herr_t H5D__contig_write_one(H5D_io_info_t *io_info, hsize_t offset, size_t size);
-static htri_t H5D__contig_may_use_select_io(const H5D_io_info_t *io_info, H5D_io_op_type_t op_type);
+static herr_t H5D__contig_write_one(H5D_io_info_t *io_info, H5D_dset_io_info_t *dset_info, hsize_t offset,
+                                    size_t size);
+static htri_t H5D__contig_may_use_select_io(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info,
+                                            H5D_io_op_type_t op_type);
 
 /*********************/
 /* Package Variables */
@@ -116,17 +121,14 @@ const H5D_layout_ops_t H5D_LOPS_CONTIG[1] = {{
     H5D__contig_is_space_alloc, /* is_space_alloc */
     H5D__contig_is_data_cached, /* is_data_cached */
     H5D__contig_io_init,        /* io_init */
+    H5D__contig_mdio_init,      /* mdio_init */
     H5D__contig_read,           /* ser_read */
     H5D__contig_write,          /* ser_write */
-#ifdef H5_HAVE_PARALLEL
-    H5D__contig_collective_read,  /* par_read */
-    H5D__contig_collective_write, /* par_write */
-#endif
-    H5D__contig_readvv,  /* readvv */
-    H5D__contig_writevv, /* writevv */
-    H5D__contig_flush,   /* flush */
-    NULL,                /* io_term */
-    NULL                 /* dest */
+    H5D__contig_readvv,         /* readvv */
+    H5D__contig_writevv,        /* writevv */
+    H5D__contig_flush,          /* flush */
+    H5D__contig_io_term,        /* io_term */
+    NULL                        /* dest */
 }};
 
 /*******************/
@@ -139,6 +141,9 @@ H5FL_BLK_DEFINE(sieve_buf);
 /* Declare extern the free list to manage blocks of type conversion data */
 H5FL_BLK_EXTERN(type_conv);
 
+/* Declare extern the free list to manage the H5D_piece_info_t struct */
+H5FL_EXTERN(H5D_piece_info_t);
+
 /*-------------------------------------------------------------------------
  * Function:	H5D__contig_alloc
  *
@@ -183,15 +188,15 @@ done:
  *-------------------------------------------------------------------------
  */
 herr_t
-H5D__contig_fill(const H5D_io_info_t *io_info)
+H5D__contig_fill(H5D_t *dset)
 {
-    const H5D_t  *dset = io_info->dset; /* the dataset pointer */
-    H5D_io_info_t ioinfo;               /* Dataset I/O info */
-    H5D_storage_t store;                /* Union of storage info for dataset */
-    hssize_t      snpoints;             /* Number of points in space (for error checking) */
-    size_t        npoints;              /* Number of points in space */
-    hsize_t       offset;               /* Offset of dataset */
-    size_t        max_temp_buf;         /* Maximum size of temporary buffer */
+    H5D_io_info_t      ioinfo;       /* Dataset I/O info */
+    H5D_dset_io_info_t dset_info;    /* Dset info */
+    H5D_storage_t      store;        /* Union of storage info for dataset */
+    hssize_t           snpoints;     /* Number of points in space (for error checking) */
+    size_t             npoints;      /* Number of points in space */
+    hsize_t            offset;       /* Offset of dataset */
+    size_t             max_temp_buf; /* Maximum size of temporary buffer */
 #ifdef H5_HAVE_PARALLEL
     MPI_Comm mpi_comm = MPI_COMM_NULL; /* MPI communicator for file */
     int      mpi_rank = (-1);          /* This process's rank  */
@@ -252,7 +257,14 @@ H5D__contig_fill(const H5D_io_info_t *io_info)
     offset = 0;
 
     /* Simple setup for dataset I/O info struct */
-    H5D_BUILD_IO_INFO_WRT(&ioinfo, dset, &store, fb_info.fill_buf);
+    ioinfo.op_type = H5D_IO_OP_WRITE;
+
+    dset_info.dset      = (H5D_t *)dset;
+    dset_info.store     = &store;
+    dset_info.buf.cvp   = fb_info.fill_buf;
+    dset_info.mem_space = NULL;
+    ioinfo.dsets_info   = &dset_info;
+    ioinfo.f_sh         = H5F_SHARED(dset->oloc.file);
 
     /*
      * Fill the entire current extent with the fill value.  We can do
@@ -281,7 +293,7 @@ H5D__contig_fill(const H5D_io_info_t *io_info)
             /* Write the chunks out from only one process */
             /* !! Use the internal "independent" DXPL!! -QAK */
             if (H5_PAR_META_WRITE == mpi_rank) {
-                if (H5D__contig_write_one(&ioinfo, offset, size) < 0) {
+                if (H5D__contig_write_one(&ioinfo, &dset_info, offset, size) < 0) {
                     /* If writing fails, push an error and stop writing, but
                      * still participate in following MPI_Barrier.
                      */
@@ -297,7 +309,7 @@ H5D__contig_fill(const H5D_io_info_t *io_info)
         else {
 #endif /* H5_HAVE_PARALLEL */
             H5_CHECK_OVERFLOW(size, size_t, hsize_t);
-            if (H5D__contig_write_one(&ioinfo, offset, size) < 0)
+            if (H5D__contig_write_one(&ioinfo, &dset_info, offset, size) < 0)
                 HGOTO_ERROR(H5E_DATASET, H5E_CANTINIT, FAIL, "unable to write fill value to dataset")
 #ifdef H5_HAVE_PARALLEL
         } /* end else */
@@ -562,34 +574,161 @@ H5D__contig_is_data_cached(const H5D_shared_t *shared_dset)
  *
  * Return:	Non-negative on success/Negative on failure
  *
- * Programmer:	Quincey Koziol
- *              Thursday, March 20, 2008
- *
+ * Programmer:	Jonathan Kim
  *-------------------------------------------------------------------------
  */
 static herr_t
-H5D__contig_io_init(H5D_io_info_t *io_info, const H5D_type_info_t H5_ATTR_UNUSED *type_info,
-                    hsize_t H5_ATTR_UNUSED nelmts, H5S_t H5_ATTR_UNUSED *file_space,
-                    H5S_t H5_ATTR_UNUSED *mem_space, H5D_chunk_map_t H5_ATTR_UNUSED *cm)
+H5D__contig_io_init(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo)
 {
+    H5D_t *dataset = dinfo->dset; /* Local pointer to dataset info */
+
+    hssize_t old_offset[H5O_LAYOUT_NDIMS];  /* Old selection offset */
+    htri_t   file_space_normalized = FALSE; /* File dataspace was normalized */
+
+    int sf_ndims; /* The number of dimensions of the file dataspace (signed) */
+
     htri_t use_selection_io = FALSE;   /* Whether to use selection I/O */
-    htri_t ret_value        = SUCCEED; /* Return value */
+    herr_t ret_value        = SUCCEED; /* Return value */
 
     FUNC_ENTER_PACKAGE
 
-    io_info->store->contig.dset_addr = io_info->dset->shared->layout.storage.u.contig.addr;
-    io_info->store->contig.dset_size = io_info->dset->shared->layout.storage.u.contig.size;
+    dinfo->store->contig.dset_addr = dataset->shared->layout.storage.u.contig.addr;
+    dinfo->store->contig.dset_size = dataset->shared->layout.storage.u.contig.size;
+
+    /* Initialize piece info */
+    dinfo->layout_io_info.contig_piece_info = NULL;
+
+    /* Get layout for dataset */
+    dinfo->layout = &(dataset->shared->layout);
+
+    /* Get dim number and dimensionality for each dataspace */
+    if ((sf_ndims = H5S_GET_EXTENT_NDIMS(dinfo->file_space)) < 0)
+        HGOTO_ERROR(H5E_DATASPACE, H5E_CANTGET, FAIL, "unable to get dimension number")
+
+    /* Normalize hyperslab selections by adjusting them by the offset */
+    /* (It might be worthwhile to normalize both the file and memory dataspaces
+     * before any (contiguous, chunked, etc) file I/O operation, in order to
+     * speed up hyperslab calculations by removing the extra checks and/or
+     * additions involving the offset and the hyperslab selection -QAK)
+     */
+    if ((file_space_normalized = H5S_hyper_normalize_offset(dinfo->file_space, old_offset)) < 0)
+        HGOTO_ERROR(H5E_DATASET, H5E_BADSELECT, FAIL, "unable to normalize dataspace by offset")
+
+    /* if selected elements exist */
+    if (dinfo->nelmts) {
+        int               u;
+        H5D_piece_info_t *new_piece_info; /* piece information to insert into skip list */
+
+        /* Get copy of dset file_space, so it can be changed temporarily
+         * purpose
+         * This tmp_fspace allows multiple write before close dset */
+        H5S_t *tmp_fspace; /* Temporary file dataspace */
+
+        /* Create "temporary" chunk for selection operations (copy file space) */
+        if (NULL == (tmp_fspace = H5S_copy(dinfo->file_space, TRUE, FALSE)))
+            HGOTO_ERROR(H5E_DATASPACE, H5E_CANTCOPY, FAIL, "unable to copy memory space")
+
+        /* Add temporary chunk to the list of pieces */
+        /* collect piece_info into Skip List */
+        /* Allocate the file & memory chunk information */
+        if (NULL == (new_piece_info = H5FL_MALLOC(H5D_piece_info_t))) {
+            (void)H5S_close(tmp_fspace);
+            HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "can't allocate chunk info")
+        } /* end if */
+
+        /* Set the piece index */
+        new_piece_info->index = 0;
+
+        /* Set the file chunk dataspace */
+        new_piece_info->fspace        = tmp_fspace;
+        new_piece_info->fspace_shared = FALSE;
+
+        /* Set the memory chunk dataspace */
+        /* same as one chunk, just use dset mem space */
+        new_piece_info->mspace = dinfo->mem_space;
+
+        /* set true for sharing mem space with dset, which means
+         * fspace gets free by application H5Sclose(), and
+         * doesn't require providing layout_ops.io_term() for H5D_LOPS_CONTIG.
+         */
+        new_piece_info->mspace_shared = TRUE;
+
+        /* Set the number of points */
+        new_piece_info->piece_points = dinfo->nelmts;
+
+        /* Copy the piece's coordinates */
+        for (u = 0; u < sf_ndims; u++)
+            new_piece_info->scaled[u] = 0;
+        new_piece_info->scaled[sf_ndims] = 0;
+
+        /* make connection to related dset info from this piece_info */
+        new_piece_info->dset_info = dinfo;
+
+        /* get dset file address for piece */
+        new_piece_info->faddr = dinfo->dset->shared->layout.storage.u.contig.addr;
 
-    /* Check if we're performing selection I/O */
-    if ((use_selection_io = H5D__contig_may_use_select_io(io_info, H5D_IO_OP_READ)) < 0)
-        HGOTO_ERROR(H5E_DATASET, H5E_CANTGET, FAIL, "can't check if selection I/O is possible")
-    io_info->use_select_io = (hbool_t)use_selection_io;
+        /* Save piece to dataset info struct so it is freed at the end of the
+         * operation */
+        dinfo->layout_io_info.contig_piece_info = new_piece_info;
+
+        /* Add piece to piece_count */
+        io_info->piece_count++;
+    } /* end if */
+
+    /* Check if we're performing selection I/O if it hasn't been disabled
+     * already */
+    if (io_info->use_select_io) {
+        if ((use_selection_io = H5D__contig_may_use_select_io(io_info, dinfo, H5D_IO_OP_READ)) < 0)
+            HGOTO_ERROR(H5E_DATASET, H5E_CANTGET, FAIL, "can't check if selection I/O is possible")
+        io_info->use_select_io = (hbool_t)use_selection_io;
+    }
 
 done:
+    if (ret_value < 0) {
+        if (H5D__contig_io_term(io_info, dinfo) < 0)
+            HDONE_ERROR(H5E_DATASPACE, H5E_CANTRELEASE, FAIL, "unable to release dataset I/O info")
+    } /* end if */
+
+    if (file_space_normalized) {
+        /* (Casting away const OK -QAK) */
+        if (H5S_hyper_denormalize_offset(dinfo->file_space, old_offset) < 0)
+            HDONE_ERROR(H5E_DATASET, H5E_BADSELECT, FAIL, "unable to normalize dataspace by offset")
+    } /* end if */
+
     FUNC_LEAVE_NOAPI(ret_value)
 } /* end H5D__contig_io_init() */
 
 /*-------------------------------------------------------------------------
+ * Function:   H5D__contig_mdio_init
+ *
+ * Purpose:    Performs second phase of initialization for multi-dataset
+ *             I/O.  Currently just adds data block to sel_pieces.
+ *
+ * Return:     Non-negative on success/Negative on failure
+ *
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5D__contig_mdio_init(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo)
+{
+    FUNC_ENTER_PACKAGE_NOERR
+
+    /* Add piece if it exists */
+    if (dinfo->layout_io_info.contig_piece_info) {
+        HDassert(io_info->sel_pieces);
+        HDassert(io_info->pieces_added < io_info->piece_count);
+
+        /* Add contiguous data block to sel_pieces */
+        io_info->sel_pieces[io_info->pieces_added] = dinfo->layout_io_info.contig_piece_info;
+
+        /* Update pieces_added */
+        io_info->pieces_added++;
+    }
+
+    FUNC_LEAVE_NOAPI(SUCCEED)
+} /* end H5D__contig_mdio_init() */
+
+/*-------------------------------------------------------------------------
  * Function:    H5D__contig_may_use_select_io
  *
  * Purpose:    A small internal function to if it may be possible to use
@@ -603,7 +742,8 @@ done:
  *-------------------------------------------------------------------------
  */
 static htri_t
-H5D__contig_may_use_select_io(const H5D_io_info_t *io_info, H5D_io_op_type_t op_type)
+H5D__contig_may_use_select_io(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info,
+                              H5D_io_op_type_t op_type)
 {
     const H5D_t *dataset   = NULL; /* Local pointer to dataset info */
     htri_t       ret_value = FAIL; /* Return value */
@@ -612,24 +752,25 @@ H5D__contig_may_use_select_io(const H5D_io_info_t *io_info, H5D_io_op_type_t op_
 
     /* Sanity check */
     HDassert(io_info);
-    HDassert(io_info->dset);
+    HDassert(dset_info);
+    HDassert(dset_info->dset);
     HDassert(op_type == H5D_IO_OP_READ || op_type == H5D_IO_OP_WRITE);
 
-    dataset = io_info->dset;
+    dataset = dset_info->dset;
 
     /* Don't use selection I/O if it's globally disabled, if there is a type
      * conversion, or if it's not a contiguous dataset, or if the sieve buffer
      * exists (write) or is dirty (read) */
-    if (!H5_use_selection_io_g || io_info->io_ops.single_read != H5D__select_read ||
-        io_info->layout_ops.readvv != H5D__contig_readvv ||
+    if (dset_info->io_ops.single_read != H5D__select_read ||
+        dset_info->layout_ops.readvv != H5D__contig_readvv ||
         (op_type == H5D_IO_OP_READ && dataset->shared->cache.contig.sieve_dirty) ||
         (op_type == H5D_IO_OP_WRITE && dataset->shared->cache.contig.sieve_buf))
         ret_value = FALSE;
     else {
         hbool_t page_buf_enabled;
 
-        HDassert(io_info->io_ops.single_write == H5D__select_write);
-        HDassert(io_info->layout_ops.writevv == H5D__contig_writevv);
+        HDassert(dset_info->io_ops.single_write == H5D__select_write);
+        HDassert(dset_info->layout_ops.writevv == H5D__contig_writevv);
 
         /* Check if the page buffer is enabled */
         if (H5PB_enabled(io_info->f_sh, H5FD_MEM_DRAW, &page_buf_enabled) < 0)
@@ -657,8 +798,7 @@ done:
  *-------------------------------------------------------------------------
  */
 herr_t
-H5D__contig_read(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts, H5S_t *file_space,
-                 H5S_t *mem_space, H5D_chunk_map_t H5_ATTR_UNUSED *fm)
+H5D__contig_read(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo)
 {
     herr_t ret_value = SUCCEED; /* Return value */
 
@@ -666,25 +806,49 @@ H5D__contig_read(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize
 
     /* Sanity check */
     HDassert(io_info);
-    HDassert(io_info->u.rbuf);
-    HDassert(type_info);
-    HDassert(mem_space);
-    HDassert(file_space);
+    HDassert(dinfo);
+    HDassert(dinfo->buf.vp);
+    HDassert(dinfo->mem_space);
+    HDassert(dinfo->file_space);
 
     if (io_info->use_select_io) {
-        size_t dst_type_size = type_info->dst_type_size;
-
-        /* Issue selection I/O call (we can skip the page buffer because we've
-         * already verified it won't be used, and the metadata accumulator
-         * because this is raw data) */
-        if (H5F_shared_select_read(H5F_SHARED(io_info->dset->oloc.file), H5FD_MEM_DRAW, nelmts > 0 ? 1 : 0,
-                                   &mem_space, &file_space, &(io_info->store->contig.dset_addr),
-                                   &dst_type_size, &(io_info->u.rbuf)) < 0)
-            HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "contiguous selection read failed")
+        /* Only perform I/O if not performing multi dataset I/O with selection
+         * I/O, otherwise the higher level will handle it after all datasets
+         * have been processed */
+        if (H5D_LAYOUT_CB_PERFORM_IO(io_info)) {
+            size_t dst_type_size = dinfo->type_info.dst_type_size;
+
+            /* Issue selection I/O call (we can skip the page buffer because we've
+             * already verified it won't be used, and the metadata accumulator
+             * because this is raw data) */
+            if (H5F_shared_select_read(H5F_SHARED(dinfo->dset->oloc.file), H5FD_MEM_DRAW,
+                                       dinfo->nelmts > 0 ? 1 : 0, &dinfo->mem_space, &dinfo->file_space,
+                                       &(dinfo->store->contig.dset_addr), &dst_type_size,
+                                       &(dinfo->buf.vp)) < 0)
+                HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "contiguous selection read failed")
+        }
+        else {
+            if (dinfo->layout_io_info.contig_piece_info) {
+                /* Add to mdset selection I/O arrays */
+                HDassert(io_info->mem_spaces);
+                HDassert(io_info->file_spaces);
+                HDassert(io_info->addrs);
+                HDassert(io_info->element_sizes);
+                HDassert(io_info->rbufs);
+                HDassert(io_info->pieces_added < io_info->piece_count);
+
+                io_info->mem_spaces[io_info->pieces_added]    = dinfo->mem_space;
+                io_info->file_spaces[io_info->pieces_added]   = dinfo->file_space;
+                io_info->addrs[io_info->pieces_added]         = dinfo->store->contig.dset_addr;
+                io_info->element_sizes[io_info->pieces_added] = dinfo->type_info.src_type_size;
+                io_info->rbufs[io_info->pieces_added]         = dinfo->buf.vp;
+                io_info->pieces_added++;
+            }
+        }
     } /* end if */
     else
         /* Read data through legacy (non-selection I/O) pathway */
-        if ((io_info->io_ops.single_read)(io_info, type_info, nelmts, file_space, mem_space) < 0)
+        if ((dinfo->io_ops.single_read)(io_info, dinfo) < 0)
             HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "contiguous read failed")
 
 done:
@@ -704,8 +868,7 @@ done:
  *-------------------------------------------------------------------------
  */
 herr_t
-H5D__contig_write(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts, H5S_t *file_space,
-                  H5S_t *mem_space, H5D_chunk_map_t H5_ATTR_UNUSED *fm)
+H5D__contig_write(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo)
 {
     herr_t ret_value = SUCCEED; /* Return value */
 
@@ -713,25 +876,49 @@ H5D__contig_write(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsiz
 
     /* Sanity check */
     HDassert(io_info);
-    HDassert(io_info->u.wbuf);
-    HDassert(type_info);
-    HDassert(mem_space);
-    HDassert(file_space);
+    HDassert(dinfo);
+    HDassert(dinfo->buf.cvp);
+    HDassert(dinfo->mem_space);
+    HDassert(dinfo->file_space);
 
     if (io_info->use_select_io) {
-        size_t dst_type_size = type_info->dst_type_size;
-
-        /* Issue selection I/O call (we can skip the page buffer because we've
-         * already verified it won't be used, and the metadata accumulator
-         * because this is raw data) */
-        if (H5F_shared_select_write(H5F_SHARED(io_info->dset->oloc.file), H5FD_MEM_DRAW, nelmts > 0 ? 1 : 0,
-                                    &mem_space, &file_space, &(io_info->store->contig.dset_addr),
-                                    &dst_type_size, &(io_info->u.wbuf)) < 0)
-            HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "contiguous selection write failed")
+        /* Only perform I/O if not performing multi dataset I/O with selection
+         * I/O, otherwise the higher level will handle it after all datasets
+         * have been processed */
+        if (H5D_LAYOUT_CB_PERFORM_IO(io_info)) {
+            size_t dst_type_size = dinfo->type_info.dst_type_size;
+
+            /* Issue selection I/O call (we can skip the page buffer because we've
+             * already verified it won't be used, and the metadata accumulator
+             * because this is raw data) */
+            if (H5F_shared_select_write(H5F_SHARED(dinfo->dset->oloc.file), H5FD_MEM_DRAW,
+                                        dinfo->nelmts > 0 ? 1 : 0, &dinfo->mem_space, &dinfo->file_space,
+                                        &(dinfo->store->contig.dset_addr), &dst_type_size,
+                                        &(dinfo->buf.cvp)) < 0)
+                HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "contiguous selection write failed")
+        }
+        else {
+            if (dinfo->layout_io_info.contig_piece_info) {
+                /* Add to mdset selection I/O arrays */
+                HDassert(io_info->mem_spaces);
+                HDassert(io_info->file_spaces);
+                HDassert(io_info->addrs);
+                HDassert(io_info->element_sizes);
+                HDassert(io_info->wbufs);
+                HDassert(io_info->pieces_added < io_info->piece_count);
+
+                io_info->mem_spaces[io_info->pieces_added]    = dinfo->mem_space;
+                io_info->file_spaces[io_info->pieces_added]   = dinfo->file_space;
+                io_info->addrs[io_info->pieces_added]         = dinfo->store->contig.dset_addr;
+                io_info->element_sizes[io_info->pieces_added] = dinfo->type_info.dst_type_size;
+                io_info->wbufs[io_info->pieces_added]         = dinfo->buf.cvp;
+                io_info->pieces_added++;
+            }
+        }
     } /* end if */
     else
         /* Write data through legacy (non-selection I/O) pathway */
-        if ((io_info->io_ops.single_write)(io_info, type_info, nelmts, file_space, mem_space) < 0)
+        if ((dinfo->io_ops.single_write)(io_info, dinfo) < 0)
             HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "contiguous write failed")
 
 done:
@@ -753,7 +940,7 @@ done:
  *-------------------------------------------------------------------------
  */
 static herr_t
-H5D__contig_write_one(H5D_io_info_t *io_info, hsize_t offset, size_t size)
+H5D__contig_write_one(H5D_io_info_t *io_info, H5D_dset_io_info_t *dset_info, hsize_t offset, size_t size)
 {
     hsize_t dset_off      = offset;  /* Offset in dataset */
     size_t  dset_len      = size;    /* Length in dataset */
@@ -767,7 +954,7 @@ H5D__contig_write_one(H5D_io_info_t *io_info, hsize_t offset, size_t size)
 
     HDassert(io_info);
 
-    if (H5D__contig_writevv(io_info, (size_t)1, &dset_curr_seq, &dset_len, &dset_off, (size_t)1,
+    if (H5D__contig_writevv(io_info, dset_info, (size_t)1, &dset_curr_seq, &dset_len, &dset_off, (size_t)1,
                             &mem_curr_seq, &mem_len, &mem_off) < 0)
         HGOTO_ERROR(H5E_IO, H5E_WRITEERROR, FAIL, "vector write failed")
 
@@ -990,9 +1177,9 @@ done:
  *-------------------------------------------------------------------------
  */
 static ssize_t
-H5D__contig_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq,
-                   size_t dset_len_arr[], hsize_t dset_off_arr[], size_t mem_max_nseq, size_t *mem_curr_seq,
-                   size_t mem_len_arr[], hsize_t mem_off_arr[])
+H5D__contig_readvv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info, size_t dset_max_nseq,
+                   size_t *dset_curr_seq, size_t dset_len_arr[], hsize_t dset_off_arr[], size_t mem_max_nseq,
+                   size_t *mem_curr_seq, size_t mem_len_arr[], hsize_t mem_off_arr[])
 {
     ssize_t ret_value = -1; /* Return value */
 
@@ -1000,6 +1187,7 @@ H5D__contig_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *d
 
     /* Check args */
     HDassert(io_info);
+    HDassert(dset_info);
     HDassert(dset_curr_seq);
     HDassert(dset_len_arr);
     HDassert(dset_off_arr);
@@ -1013,9 +1201,9 @@ H5D__contig_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *d
 
         /* Set up user data for H5VM_opvv() */
         udata.f_sh         = io_info->f_sh;
-        udata.dset_contig  = &(io_info->dset->shared->cache.contig);
-        udata.store_contig = &(io_info->store->contig);
-        udata.rbuf         = (unsigned char *)io_info->u.rbuf;
+        udata.dset_contig  = &(dset_info->dset->shared->cache.contig);
+        udata.store_contig = &(dset_info->store->contig);
+        udata.rbuf         = (unsigned char *)dset_info->buf.vp;
 
         /* Call generic sequence operation routine */
         if ((ret_value =
@@ -1028,8 +1216,8 @@ H5D__contig_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *d
 
         /* Set up user data for H5VM_opvv() */
         udata.f_sh      = io_info->f_sh;
-        udata.dset_addr = io_info->store->contig.dset_addr;
-        udata.rbuf      = (unsigned char *)io_info->u.rbuf;
+        udata.dset_addr = dset_info->store->contig.dset_addr;
+        udata.rbuf      = (unsigned char *)dset_info->buf.vp;
 
         /* Call generic sequence operation routine */
         if ((ret_value = H5VM_opvv(dset_max_nseq, dset_curr_seq, dset_len_arr, dset_off_arr, mem_max_nseq,
@@ -1308,9 +1496,9 @@ done:
  *-------------------------------------------------------------------------
  */
 static ssize_t
-H5D__contig_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq,
-                    size_t dset_len_arr[], hsize_t dset_off_arr[], size_t mem_max_nseq, size_t *mem_curr_seq,
-                    size_t mem_len_arr[], hsize_t mem_off_arr[])
+H5D__contig_writevv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info, size_t dset_max_nseq,
+                    size_t *dset_curr_seq, size_t dset_len_arr[], hsize_t dset_off_arr[], size_t mem_max_nseq,
+                    size_t *mem_curr_seq, size_t mem_len_arr[], hsize_t mem_off_arr[])
 {
     ssize_t ret_value = -1; /* Return value (Size of sequence in bytes) */
 
@@ -1318,6 +1506,7 @@ H5D__contig_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *
 
     /* Check args */
     HDassert(io_info);
+    HDassert(dset_info);
     HDassert(dset_curr_seq);
     HDassert(dset_len_arr);
     HDassert(dset_off_arr);
@@ -1331,9 +1520,9 @@ H5D__contig_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *
 
         /* Set up user data for H5VM_opvv() */
         udata.f_sh         = io_info->f_sh;
-        udata.dset_contig  = &(io_info->dset->shared->cache.contig);
-        udata.store_contig = &(io_info->store->contig);
-        udata.wbuf         = (const unsigned char *)io_info->u.wbuf;
+        udata.dset_contig  = &(dset_info->dset->shared->cache.contig);
+        udata.store_contig = &(dset_info->store->contig);
+        udata.wbuf         = (const unsigned char *)dset_info->buf.cvp;
 
         /* Call generic sequence operation routine */
         if ((ret_value =
@@ -1346,8 +1535,8 @@ H5D__contig_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *
 
         /* Set up user data for H5VM_opvv() */
         udata.f_sh      = io_info->f_sh;
-        udata.dset_addr = io_info->store->contig.dset_addr;
-        udata.wbuf      = (const unsigned char *)io_info->u.wbuf;
+        udata.dset_addr = dset_info->store->contig.dset_addr;
+        udata.wbuf      = (const unsigned char *)dset_info->buf.cvp;
 
         /* Call generic sequence operation routine */
         if ((ret_value = H5VM_opvv(dset_max_nseq, dset_curr_seq, dset_len_arr, dset_off_arr, mem_max_nseq,
@@ -1391,6 +1580,35 @@ done:
 } /* end H5D__contig_flush() */
 
 /*-------------------------------------------------------------------------
+ * Function:    H5D__contig_io_term
+ *
+ * Purpose:    Destroy I/O operation information.
+ *
+ * Return:    Non-negative on success/Negative on failure
+ *
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5D__contig_io_term(H5D_io_info_t H5_ATTR_UNUSED *io_info, H5D_dset_io_info_t *di)
+{
+    herr_t ret_value = SUCCEED; /*return value        */
+
+    FUNC_ENTER_PACKAGE
+
+    HDassert(di);
+
+    /* Free piece info */
+    if (di->layout_io_info.contig_piece_info) {
+        if (H5D__free_piece_info(di->layout_io_info.contig_piece_info, NULL, NULL) < 0)
+            HGOTO_ERROR(H5E_DATASET, H5E_CANTFREE, FAIL, "can't free piece info")
+        di->layout_io_info.contig_piece_info = NULL;
+    }
+
+done:
+    FUNC_LEAVE_NOAPI(ret_value)
+} /* end H5D__contig_io_term() */
+
+/*-------------------------------------------------------------------------
  * Function:	H5D__contig_copy
  *
  * Purpose:	Copy contiguous storage raw data from SRC file to DST file.