Multi Dataset (#2120)

* Fix bug with cross platform compatibility of references within vlens.
No testing yet.

* Merge from multi_rd_wd_coll_io to a more recent branch from develop.
Untested, probably does not work yet.

* Committing clang-format changes

* Committing clang-format changes

* Fix many bugs in multi dataset branch.  Mostly works, some issues in
SWMR tests.

* Committing clang-format changes

* Disable test in swmr.c that was failing due to bug in HDF5 unrelated to
multi dataset.

* Committing clang-format changes

* Fixed fortran multi-dataset tests

* Fixed xlf errors

* Added benchmark code for multi-datasets

* loops over datasets

* added missing error arg.

* Added gnuplot formatting

* Jonathan Kim original MD benchmarking code

* updated MD benchmarking code

* code clean-up

* Only make files in feature test mode

* misc clean-up

* removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option

* Committing clang-format changes

* Change multi dataset API to use arrays of individual parameters instead
of the parameter struct.

* Committing clang-format changes

* Update to new multi dataset Fortran API and tests. (#1724)

* Update to new multi dataset Fortran API and tests.
* Sync Fortran with develop.
* skipping h5pget_mpio_actual_io_mode_f for now

* Fixed issue with dxpl_id, changed to variable size dim. (#1770)

* Remove "is_coll_broken" field from H5D_io_info_t struct

* Committing clang-format changes

* Minor cleanup in multi dataset code.

* Committing clang-format changes

* Clean up in multi dataset code.

* Committing clang-format changes

* Committing clang-format changes

* Fix speeling

* Fix bug in parallel compression. Switch base_maddr in io_info to be a
union.

* Committing clang-format changes

* Implement selection I/O support with multi dataset.  Will be broken in
parallel until PR 1803 is merged to develop then the MDS branch.

* Committing clang-format changes

* Spelling

* Fix bug in multi dataset that could cause errors when only some of the
datasets in the multi dataset I/O used type conversion.

* Committing clang-format changes

* Integrate multi dataset APIs with VOL layer.  Add async versions of
multi dataset APIs.

* Committing clang-format changes

* Spelling fixes

* Fix bug in non-parallel HDF5 compilation.

* Committing clang-format changes

* Fix potential memory/free list error. Minor performance fix. Other minor
changes.

* Committing clang-format changes

* Fix memory leak with memory dataspace for I/O.

* Committing clang-format changes

* Fix stack variables too large.  Rename H5D_dset_info_t to
H5D_dset_io_info_t.

* Committing clang-format changes

* Remove mem_space_alloc field from H5D_dset_io_info_t.  Each function is
now responsible for freeing any spaces it adds to dset_info.

* Committing clang-format changes

* fixed _multi Fortran declaration

* Refactor various things in (mostly) the serial I/O code path to make
things more maintainable.

* Committing clang-format changes

* updated to array based, doxygen, and examples

* Reinstate H5D_chunk_map_t, stored (via pointer) inside
H5D_dset_io_info_t.

* Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t.
Switch temporary dset_infos to be local stack variables.

* Committing clang-format changes

* format cleanup

* format cleanup

* added coll and ind

* Modify all parallel I/O paths to take dset_info instead of assuming
dset_info[0].

* Committing clang-format changes

* fixed output

* Rework parallel I/O code to work properly with multi dataset in more
cases.  Fix bug in parallel compression.

* Committing clang-format changes

* Prevent H5D__multi_chunk_collective_io() from messing up collective opt
property for other datasets in I/O.  Other minor cleanup.  Add new test
case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now
due to failures apparently unrelated to multi dataset code.

* Fix spelling

* Committing clang-format changes

* Replace N log N algorithm for finding chunk in
H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of
io_info->sel_pieces in that function.

* Committing clang-format changes

* Replace sel_pieces skiplist in io_info with flat array of pointers, use
qsort in I/O routine only when necessary.

* Committing clang-format changes

* Add new test case to mdset.c

* Committing clang-format changes

* Fix spelling

* Very minor fix in H5VL__native_dataset_read()

* Fix bug that could affect filtered parallel multi-dataset I/O.

* Add RM entries for H5Dread_multi(), H5Dread_multi_async(),
H5Dwrite_multi(), and H5Dwrite_multi_async()

* Unskip test in swmr.c

* Committing clang-format changes

* Eliminate H5D__pre_read and H5D__pre_write

* Remove examples/ph5mdsettest.c. Will fix and re-add as a test.

* Enable hyperslab combinations in mdset test

* Committing clang-format changes

* Clarify H5Dread/write_multi documentation.

* Fix bugs in multi-dataset I/O.  Expand serial multi dataset test.
Update macro in parallel multi dataset test.

* Committing clang-format changes

* Spelling

* Remove obsolete entry in bin/trace

* Rework type conversion buffer allocation. Only one buffer is shared
between datasets in mdset mode, and it is malloced instead of calloced.

* Committing clang-format changes

* Fix bug in error handling in H5D__read/write

* added multi-dataset fortran check with optional dataset creation id (#2150)

* removed dup. dll entry

* Address comments from code review.

* Remove spurious changes in H5Fmpi.c

* Fix issue with reading unallocated datasets in multi-dataset mode.
Address other comments from code review.

* Committing clang-format changes

* Delay chunk index lookup from io_init to mdio_init so it doesn't add
overhead to single dataset I/O.

* Committing clang-format changes

* Fix inappropriate use of piece_count

* updated copyright on new file, removed benchmark from testing dir.

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>

1 files changed, 275 insertions, 33 deletions

diff --git a/src/H5D.c b/src/H5D.c
index d9c75b2..66240da 100644
--- a/src/H5D.c
+++ b/src/H5D.c
@@ -50,10 +50,11 @@ static hid_t  H5D__create_api_common(hid_t loc_id, const char *name, hid_t type_
 static hid_t  H5D__open_api_common(hid_t loc_id, const char *name, hid_t dapl_id, void **token_ptr,
                                    H5VL_object_t **_vol_obj_ptr);
 static hid_t  H5D__get_space_api_common(hid_t dset_id, void **token_ptr, H5VL_object_t **_vol_obj_ptr);
-static herr_t H5D__read_api_common(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id,
-                                   hid_t dxpl_id, void *buf, void **token_ptr, H5VL_object_t **_vol_obj_ptr);
-static herr_t H5D__write_api_common(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id,
-                                    hid_t dxpl_id, const void *buf, void **token_ptr,
+static herr_t H5D__read_api_common(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[],
+                                   hid_t file_space_id[], hid_t dxpl_id, void *buf[], void **token_ptr,
+                                   H5VL_object_t **_vol_obj_ptr);
+static herr_t H5D__write_api_common(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[],
+                                    hid_t file_space_id[], hid_t dxpl_id, const void *buf[], void **token_ptr,
                                     H5VL_object_t **_vol_obj_ptr);
 static herr_t H5D__set_extent_api_common(hid_t dset_id, const hsize_t size[], void **token_ptr,
                                          H5VL_object_t **_vol_obj_ptr);
@@ -943,26 +944,64 @@ done:
  *-------------------------------------------------------------------------
  */
 static herr_t
-H5D__read_api_common(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id, hid_t dxpl_id,
-                     void *buf, void **token_ptr, H5VL_object_t **_vol_obj_ptr)
+H5D__read_api_common(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[],
+                     hid_t file_space_id[], hid_t dxpl_id, void *buf[], void **token_ptr,
+                     H5VL_object_t **_vol_obj_ptr)
 {
     H5VL_object_t  *tmp_vol_obj = NULL; /* Object for loc_id */
     H5VL_object_t **vol_obj_ptr =
         (_vol_obj_ptr ? _vol_obj_ptr : &tmp_vol_obj); /* Ptr to object ptr for loc_id */
-    herr_t ret_value = SUCCEED;                       /* Return value */
+    void   *obj_local;                                /* Local buffer for obj */
+    void  **obj = &obj_local;                         /* Array of object pointers */
+    H5VL_t *connector;                                /* VOL connector pointer */
+    size_t  i;                                        /* Local index variable */
+    herr_t  ret_value = SUCCEED;                      /* Return value */
 
     FUNC_ENTER_PACKAGE
 
     /* Check arguments */
-    if (mem_space_id < 0)
-        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "invalid memory dataspace ID")
-    if (file_space_id < 0)
-        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "invalid file dataspace ID")
+    if (count == 0)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "count must be greater than 0")
+    if (!dset_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "dset_id array not provided")
+    if (!mem_type_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "mem_type_id array not provided")
+    if (!mem_space_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "mem_space_id array not provided")
+    if (!file_space_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "file_space_id array not provided")
+    if (!buf)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "buf array not provided")
 
-    /* Get dataset pointer */
-    if (NULL == (*vol_obj_ptr = (H5VL_object_t *)H5I_object_verify(dset_id, H5I_DATASET)))
+    /* Allocate obj array if necessary */
+    if (count > 1)
+        if (NULL == (obj = (void **)H5MM_malloc(count * sizeof(void *))))
+            HGOTO_ERROR(H5E_VOL, H5E_CANTALLOC, FAIL, "can't allocate space for object array")
+
+    /* Get vol_obj_ptr (return just the first dataset to caller if requested) */
+    if (NULL == (*vol_obj_ptr = (H5VL_object_t *)H5I_object_verify(dset_id[0], H5I_DATASET)))
         HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "dset_id is not a dataset ID")
 
+    /* Save the connector of the first dataset.  Unpack the connector and call
+     * the "direct" read function here to avoid allocating an array of count
+     * H5VL_object_ts. */
+    connector = (*vol_obj_ptr)->connector;
+
+    /* Build obj array */
+    obj[0] = (*vol_obj_ptr)->data;
+    for (i = 1; i < count; i++) {
+        /* Get the object */
+        if (NULL == (tmp_vol_obj = (H5VL_object_t *)H5I_object_verify(dset_id[i], H5I_DATASET)))
+            HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "dset_id is not a dataset ID")
+        obj[i] = tmp_vol_obj->data;
+
+        /* Make sure the class matches */
+        if (tmp_vol_obj->connector->cls->value != connector->cls->value)
+            HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL,
+                        "datasets are accessed through different VOL connectors and can't be used in the "
+                        "same I/O call")
+    }
+
     /* Get the default dataset transfer property list if the user didn't provide one */
     if (H5P_DEFAULT == dxpl_id)
         dxpl_id = H5P_DATASET_XFER_DEFAULT;
@@ -970,11 +1009,15 @@ H5D__read_api_common(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t
         HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "not xfer parms")
 
     /* Read the data */
-    if (H5VL_dataset_read(*vol_obj_ptr, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, token_ptr) <
-        0)
+    if (H5VL_dataset_read_direct(count, obj, connector, mem_type_id, mem_space_id, file_space_id, dxpl_id,
+                                 buf, token_ptr) < 0)
         HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "can't read data")
 
 done:
+    /* Free memory */
+    if (obj != &obj_local)
+        H5MM_free(obj);
+
     FUNC_LEAVE_NOAPI(ret_value)
 } /* end H5D__read_api_common() */
 
@@ -1022,7 +1065,8 @@ H5Dread(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_i
     H5TRACE6("e", "iiiiix", dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf);
 
     /* Read the data */
-    if (H5D__read_api_common(dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, NULL, NULL) < 0)
+    if (H5D__read_api_common(1, &dset_id, &mem_type_id, &mem_space_id, &file_space_id, dxpl_id, &buf, NULL,
+                             NULL) < 0)
         HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "can't synchronously read data")
 
 done:
@@ -1059,8 +1103,8 @@ H5Dread_async(const char *app_file, const char *app_func, unsigned app_line, hid
         token_ptr = &token; /* Point at token for VOL connector to set up */
 
     /* Read the data */
-    if (H5D__read_api_common(dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, token_ptr,
-                             &vol_obj) < 0)
+    if (H5D__read_api_common(1, &dset_id, &mem_type_id, &mem_space_id, &file_space_id, dxpl_id, &buf,
+                             token_ptr, &vol_obj) < 0)
         HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "can't asynchronously read data")
 
     /* If a token was created, add the token to the event set */
@@ -1076,6 +1120,84 @@ done:
 } /* end H5Dread_async() */
 
 /*-------------------------------------------------------------------------
+ * Function:	H5Dread_multi
+ *
+ * Purpose:	Multi-version of H5Dread(), which reads selections from
+ *              multiple datasets from a file into application memory BUFS.
+ *
+ * Return:	Non-negative on success/Negative on failure
+ *
+ * Programmer:	Jonathan Kim Nov, 2013
+ *
+ *-------------------------------------------------------------------------
+ */
+herr_t
+H5Dread_multi(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], hid_t file_space_id[],
+              hid_t dxpl_id, void *buf[] /*out*/)
+{
+    herr_t ret_value = SUCCEED; /* Return value */
+
+    FUNC_ENTER_API(FAIL)
+    H5TRACE7("e", "z*i*i*i*iix", count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf);
+
+    if (count == 0)
+        HGOTO_DONE(SUCCEED)
+
+    /* Read the data */
+    if (H5D__read_api_common(count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, NULL,
+                             NULL) < 0)
+        HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "can't synchronously read data")
+
+done:
+    FUNC_LEAVE_API(ret_value)
+} /* end H5Dread_multi() */
+
+/*-------------------------------------------------------------------------
+ * Function:    H5Dread_multi_async
+ *
+ * Purpose:     Asynchronously read dataset elements from multiple
+ *              datasets.
+ *
+ * Return:      Non-negative on success/Negative on failure
+ *
+ *-------------------------------------------------------------------------
+ */
+herr_t
+H5Dread_multi_async(const char *app_file, const char *app_func, unsigned app_line, size_t count,
+                    hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], hid_t file_space_id[],
+                    hid_t dxpl_id, void *buf[] /*out*/, hid_t es_id)
+{
+    H5VL_object_t *vol_obj   = NULL;            /* Dataset VOL object */
+    void          *token     = NULL;            /* Request token for async operation        */
+    void         **token_ptr = H5_REQUEST_NULL; /* Pointer to request token for async operation        */
+    herr_t         ret_value = SUCCEED;         /* Return value */
+
+    FUNC_ENTER_API(FAIL)
+    H5TRACE11("e", "*s*sIuz*i*i*i*iixi", app_file, app_func, app_line, count, dset_id, mem_type_id,
+              mem_space_id, file_space_id, dxpl_id, buf, es_id);
+
+    /* Set up request token pointer for asynchronous operation */
+    if (H5ES_NONE != es_id)
+        token_ptr = &token; /* Point at token for VOL connector to set up */
+
+    /* Read the data */
+    if (H5D__read_api_common(count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf,
+                             token_ptr, &vol_obj) < 0)
+        HGOTO_ERROR(H5E_DATASET, H5E_READERROR, FAIL, "can't asynchronously read data")
+
+    /* If a token was created, add the token to the event set */
+    if (NULL != token)
+        /* clang-format off */
+        if (H5ES_insert(es_id, vol_obj->connector, token,
+                        H5ARG_TRACE11(__func__, "*s*sIuz*i*i*i*iixi", app_file, app_func, app_line, count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, es_id)) < 0)
+            /* clang-format on */
+            HGOTO_ERROR(H5E_DATASET, H5E_CANTINSERT, FAIL, "can't insert token into event set")
+
+done:
+    FUNC_LEAVE_API(ret_value)
+} /* end H5Dread_multi_async() */
+
+/*-------------------------------------------------------------------------
  * Function:    H5Dread_chunk
  *
  * Purpose:     Reads an entire chunk from the file directly.
@@ -1142,26 +1264,64 @@ done:
  *-------------------------------------------------------------------------
  */
 static herr_t
-H5D__write_api_common(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id,
-                      hid_t dxpl_id, const void *buf, void **token_ptr, H5VL_object_t **_vol_obj_ptr)
+H5D__write_api_common(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[],
+                      hid_t file_space_id[], hid_t dxpl_id, const void *buf[], void **token_ptr,
+                      H5VL_object_t **_vol_obj_ptr)
 {
     H5VL_object_t  *tmp_vol_obj = NULL; /* Object for loc_id */
     H5VL_object_t **vol_obj_ptr =
         (_vol_obj_ptr ? _vol_obj_ptr : &tmp_vol_obj); /* Ptr to object ptr for loc_id */
-    herr_t ret_value = SUCCEED;                       /* Return value */
+    void   *obj_local;                                /* Local buffer for obj */
+    void  **obj = &obj_local;                         /* Array of object pointers */
+    H5VL_t *connector;                                /* VOL connector pointer */
+    size_t  i;                                        /* Local index variable */
+    herr_t  ret_value = SUCCEED;                      /* Return value */
 
     FUNC_ENTER_PACKAGE
 
     /* Check arguments */
-    if (mem_space_id < 0)
-        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "invalid memory dataspace ID")
-    if (file_space_id < 0)
-        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "invalid file dataspace ID")
+    if (count == 0)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "count must be greater than 0")
+    if (!dset_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "dset_id array not provided")
+    if (!mem_type_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "mem_type_id array not provided")
+    if (!mem_space_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "mem_space_id array not provided")
+    if (!file_space_id)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "file_space_id array not provided")
+    if (!buf)
+        HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "buf array not provided")
 
-    /* Get dataset pointer */
-    if (NULL == (*vol_obj_ptr = (H5VL_object_t *)H5I_object_verify(dset_id, H5I_DATASET)))
+    /* Allocate obj array if necessary */
+    if (count > 1)
+        if (NULL == (obj = (void **)H5MM_malloc(count * sizeof(void *))))
+            HGOTO_ERROR(H5E_VOL, H5E_CANTALLOC, FAIL, "can't allocate space for object array")
+
+    /* Get vol_obj_ptr (return just the first dataset to caller if requested) */
+    if (NULL == (*vol_obj_ptr = (H5VL_object_t *)H5I_object_verify(dset_id[0], H5I_DATASET)))
         HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "dset_id is not a dataset ID")
 
+    /* Save the connector of the first dataset.  Unpack the connector and call
+     * the "direct" write function here to avoid allocating an array of count
+     * H5VL_object_ts. */
+    connector = (*vol_obj_ptr)->connector;
+
+    /* Build obj array */
+    obj[0] = (*vol_obj_ptr)->data;
+    for (i = 1; i < count; i++) {
+        /* Get the object */
+        if (NULL == (tmp_vol_obj = (H5VL_object_t *)H5I_object_verify(dset_id[i], H5I_DATASET)))
+            HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "dset_id is not a dataset ID")
+        obj[i] = tmp_vol_obj->data;
+
+        /* Make sure the class matches */
+        if (tmp_vol_obj->connector->cls->value != connector->cls->value)
+            HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL,
+                        "datasets are accessed through different VOL connectors and can't be used in the "
+                        "same I/O call")
+    }
+
     /* Get the default dataset transfer property list if the user didn't provide one */
     if (H5P_DEFAULT == dxpl_id)
         dxpl_id = H5P_DATASET_XFER_DEFAULT;
@@ -1169,11 +1329,15 @@ H5D__write_api_common(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_
         HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "not xfer parms")
 
     /* Write the data */
-    if (H5VL_dataset_write(*vol_obj_ptr, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, token_ptr) <
-        0)
+    if (H5VL_dataset_write_direct(count, obj, connector, mem_type_id, mem_space_id, file_space_id, dxpl_id,
+                                  buf, token_ptr) < 0)
         HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "can't write data")
 
 done:
+    /* Free memory */
+    if (obj != &obj_local)
+        H5MM_free(obj);
+
     FUNC_LEAVE_NOAPI(ret_value)
 } /* end H5D__write_api_common() */
 
@@ -1222,8 +1386,8 @@ H5Dwrite(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_
     H5TRACE6("e", "iiiii*x", dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf);
 
     /* Write the data */
-    if (H5D__write_api_common(dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, NULL, NULL) <
-        0)
+    if (H5D__write_api_common(1, &dset_id, &mem_type_id, &mem_space_id, &file_space_id, dxpl_id, &buf, NULL,
+                              NULL) < 0)
         HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "can't synchronously write data")
 
 done:
@@ -1261,8 +1425,8 @@ H5Dwrite_async(const char *app_file, const char *app_func, unsigned app_line, hi
         token_ptr = &token; /* Point at token for VOL connector to set up */
 
     /* Write the data */
-    if (H5D__write_api_common(dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, token_ptr,
-                              &vol_obj) < 0)
+    if (H5D__write_api_common(1, &dset_id, &mem_type_id, &mem_space_id, &file_space_id, dxpl_id, &buf,
+                              token_ptr, &vol_obj) < 0)
         HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "can't asynchronously write data")
 
     /* If a token was created, add the token to the event set */
@@ -1278,6 +1442,84 @@ done:
 } /* end H5Dwrite_async() */
 
 /*-------------------------------------------------------------------------
+ * Function:	H5Dwrite_multi
+ *
+ * Purpose:	Multi-version of H5Dwrite(), which writes selections from
+ *              application memory BUFs into multiple datasets in a file.
+ *
+ * Return:	Non-negative on success/Negative on failure
+ *
+ * Programmer:	Jonathan Kim  Nov, 2013
+ *
+ *-------------------------------------------------------------------------
+ */
+herr_t
+H5Dwrite_multi(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[],
+               hid_t file_space_id[], hid_t dxpl_id, const void *buf[])
+{
+    herr_t ret_value = SUCCEED; /* Return value */
+
+    FUNC_ENTER_API(FAIL)
+    H5TRACE7("e", "z*i*i*i*ii**x", count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf);
+
+    if (count == 0)
+        HGOTO_DONE(SUCCEED)
+
+    /* Write the data */
+    if (H5D__write_api_common(count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, NULL,
+                              NULL) < 0)
+        HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "can't synchronously write data")
+
+done:
+    FUNC_LEAVE_API(ret_value)
+} /* end H5Dwrite_multi() */
+
+/*-------------------------------------------------------------------------
+ * Function:    H5Dwrite_multi_async
+ *
+ * Purpose:     Asynchronously write dataset elements to multiple
+ *              datasets.
+ *
+ * Return:      Non-negative on success/Negative on failure
+ *
+ *-------------------------------------------------------------------------
+ */
+herr_t
+H5Dwrite_multi_async(const char *app_file, const char *app_func, unsigned app_line, size_t count,
+                     hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], hid_t file_space_id[],
+                     hid_t dxpl_id, const void *buf[], hid_t es_id)
+{
+    H5VL_object_t *vol_obj   = NULL;            /* Dataset VOL object */
+    void          *token     = NULL;            /* Request token for async operation        */
+    void         **token_ptr = H5_REQUEST_NULL; /* Pointer to request token for async operation        */
+    herr_t         ret_value = SUCCEED;         /* Return value */
+
+    FUNC_ENTER_API(FAIL)
+    H5TRACE11("e", "*s*sIuz*i*i*i*ii**xi", app_file, app_func, app_line, count, dset_id, mem_type_id,
+              mem_space_id, file_space_id, dxpl_id, buf, es_id);
+
+    /* Set up request token pointer for asynchronous operation */
+    if (H5ES_NONE != es_id)
+        token_ptr = &token; /* Point at token for VOL connector to set up */
+
+    /* Write the data */
+    if (H5D__write_api_common(count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf,
+                              token_ptr, &vol_obj) < 0)
+        HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "can't asynchronously write data")
+
+    /* If a token was created, add the token to the event set */
+    if (NULL != token)
+        /* clang-format off */
+        if (H5ES_insert(es_id, vol_obj->connector, token,
+                        H5ARG_TRACE11(__func__, "*s*sIuz*i*i*i*ii**xi", app_file, app_func, app_line, count, dset_id, mem_type_id, mem_space_id, file_space_id, dxpl_id, buf, es_id)) < 0)
+            /* clang-format on */
+            HGOTO_ERROR(H5E_DATASET, H5E_CANTINSERT, FAIL, "can't insert token into event set")
+
+done:
+    FUNC_LEAVE_API(ret_value)
+} /* end H5Dwrite_multi_async() */
+
+/*-------------------------------------------------------------------------
  * Function:    H5Dwrite_chunk
  *
  * Purpose:     Writes an entire chunk to the file directly.