diff options
author | Neil Fortner <fortnern@gmail.com> | 2022-10-19 16:13:15 (GMT) |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-10-19 16:13:15 (GMT) |
commit | 93754cae33d4ed45850745664ce5e59f270f38f8 (patch) | |
tree | ac3154f44fe7969dacb452474c32fcaeb84e620d /src/H5Dpublic.h | |
parent | a898cef6c2e277f74cc99302a3c49c39a7428b5b (diff) | |
download | hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.zip hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.tar.gz hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.tar.bz2 |
Multi Dataset (#2120)
* Fix bug with cross platform compatibility of references within vlens.
No testing yet.
* Merge from multi_rd_wd_coll_io to a more recent branch from develop.
Untested, probably does not work yet.
* Committing clang-format changes
* Committing clang-format changes
* Fix many bugs in multi dataset branch. Mostly works, some issues in
SWMR tests.
* Committing clang-format changes
* Disable test in swmr.c that was failing due to bug in HDF5 unrelated to
multi dataset.
* Committing clang-format changes
* Fixed fortran multi-dataset tests
* Fixed xlf errors
* Added benchmark code for multi-datasets
* loops over datasets
* added missing error arg.
* Added gnuplot formatting
* Jonathan Kim original MD benchmarking code
* updated MD benchmarking code
* code clean-up
* Only make files in feature test mode
* misc clean-up
* removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option
* Committing clang-format changes
* Change multi dataset API to use arrays of individual parameters instead
of the parameter struct.
* Committing clang-format changes
* Update to new multi dataset Fortran API and tests. (#1724)
* Update to new multi dataset Fortran API and tests.
* Sync Fortran with develop.
* skipping h5pget_mpio_actual_io_mode_f for now
* Fixed issue with dxpl_id, changed to variable size dim. (#1770)
* Remove "is_coll_broken" field from H5D_io_info_t struct
* Committing clang-format changes
* Minor cleanup in multi dataset code.
* Committing clang-format changes
* Clean up in multi dataset code.
* Committing clang-format changes
* Committing clang-format changes
* Fix speeling
* Fix bug in parallel compression. Switch base_maddr in io_info to be a
union.
* Committing clang-format changes
* Implement selection I/O support with multi dataset. Will be broken in
parallel until PR 1803 is merged to develop then the MDS branch.
* Committing clang-format changes
* Spelling
* Fix bug in multi dataset that could cause errors when only some of the
datasets in the multi dataset I/O used type conversion.
* Committing clang-format changes
* Integrate multi dataset APIs with VOL layer. Add async versions of
multi dataset APIs.
* Committing clang-format changes
* Spelling fixes
* Fix bug in non-parallel HDF5 compilation.
* Committing clang-format changes
* Fix potential memory/free list error. Minor performance fix. Other minor
changes.
* Committing clang-format changes
* Fix memory leak with memory dataspace for I/O.
* Committing clang-format changes
* Fix stack variables too large. Rename H5D_dset_info_t to
H5D_dset_io_info_t.
* Committing clang-format changes
* Remove mem_space_alloc field from H5D_dset_io_info_t. Each function is
now responsible for freeing any spaces it adds to dset_info.
* Committing clang-format changes
* fixed _multi Fortran declaration
* Refactor various things in (mostly) the serial I/O code path to make
things more maintainable.
* Committing clang-format changes
* updated to array based, doxygen, and examples
* Reinstate H5D_chunk_map_t, stored (via pointer) inside
H5D_dset_io_info_t.
* Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t.
Switch temporary dset_infos to be local stack variables.
* Committing clang-format changes
* format cleanup
* format cleanup
* added coll and ind
* Modify all parallel I/O paths to take dset_info instead of assuming
dset_info[0].
* Committing clang-format changes
* fixed output
* Rework parallel I/O code to work properly with multi dataset in more
cases. Fix bug in parallel compression.
* Committing clang-format changes
* Prevent H5D__multi_chunk_collective_io() from messing up collective opt
property for other datasets in I/O. Other minor cleanup. Add new test
case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now
due to failures apparently unrelated to multi dataset code.
* Fix spelling
* Committing clang-format changes
* Replace N log N algorithm for finding chunk in
H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of
io_info->sel_pieces in that function.
* Committing clang-format changes
* Replace sel_pieces skiplist in io_info with flat array of pointers, use
qsort in I/O routine only when necessary.
* Committing clang-format changes
* Add new test case to mdset.c
* Committing clang-format changes
* Fix spelling
* Very minor fix in H5VL__native_dataset_read()
* Fix bug that could affect filtered parallel multi-dataset I/O.
* Add RM entries for H5Dread_multi(), H5Dread_multi_async(),
H5Dwrite_multi(), and H5Dwrite_multi_async()
* Unskip test in swmr.c
* Committing clang-format changes
* Eliminate H5D__pre_read and H5D__pre_write
* Remove examples/ph5mdsettest.c. Will fix and re-add as a test.
* Enable hyperslab combinations in mdset test
* Committing clang-format changes
* Clarify H5Dread/write_multi documentation.
* Fix bugs in multi-dataset I/O. Expand serial multi dataset test.
Update macro in parallel multi dataset test.
* Committing clang-format changes
* Spelling
* Remove obsolete entry in bin/trace
* Rework type conversion buffer allocation. Only one buffer is shared
between datasets in mdset mode, and it is malloced instead of calloced.
* Committing clang-format changes
* Fix bug in error handling in H5D__read/write
* added multi-dataset fortran check with optional dataset creation id (#2150)
* removed dup. dll entry
* Address comments from code review.
* Remove spurious changes in H5Fmpi.c
* Fix issue with reading unallocated datasets in multi-dataset mode.
Address other comments from code review.
* Committing clang-format changes
* Delay chunk index lookup from io_init to mdio_init so it doesn't add
overhead to single dataset I/O.
* Committing clang-format changes
* Fix inappropriate use of piece_count
* updated copyright on new file, removed benchmark from testing dir.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>
Diffstat (limited to 'src/H5Dpublic.h')
-rw-r--r-- | src/H5Dpublic.h | 111 |
1 files changed, 111 insertions, 0 deletions
diff --git a/src/H5Dpublic.h b/src/H5Dpublic.h index 6fad138..8040272 100644 --- a/src/H5Dpublic.h +++ b/src/H5Dpublic.h @@ -852,6 +852,52 @@ H5_DLL herr_t H5Dread(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_ /** * -------------------------------------------------------------------------- + * \ingroup H5D + * + * \brief Reads raw data from a set of datasets into the provided buffers + * + * \param[in] count Number of datasets to read from + * \param[in] dset_id Identifiers of the datasets to read from + * \param[in] mem_type_id Identifiers of the memory datatypes + * \param[in] mem_space_id Identifiers of the memory dataspaces + * \param[in] file_space_id Identifiers of the datasets' dataspaces in the file + * \param[in] dxpl_id Identifier of a transfer property list + * \param[out] buf Buffers to receive data read from file + * + * \return \herr_t + * + * \details H5Dread_multi() reads data from \p count datasets, whose identifiers + * are listed in the \p dset_id array, from the file into multiple + * application memory buffers listed in the \p buf array. Data transfer + * properties are defined by the argument \p dxpl_id. The memory + * datatypes of each dataset are listed by identifier in the \p + * mem_type_id array. The parts of each dataset to read are listed by + * identifier in the \p file_space_id array, and the parts of each + * application memory buffer to read to are listed by identifier in the + * \p mem_space_id array. All array parameters have length \p count. + * + * This function will produce the same results as \p count calls to + * H5Dread(). Information listed in that function about the specifics + * of its behaviour also apply to H5Dread_multi(). By calling + * H5Dread_multi() instead of multiple calls to H5Dread(), however, the + * library can in some cases pass information about the entire I/O + * operation to the file driver, which can improve performance. + * + * All datasets must be in the same HDF5 file, and each unique dataset + * may only be listed once. If this function is called collectively in + * parallel, each rank must pass exactly the same list of datasets in + * \p dset_id , though the other parameters may differ. + * + * \since 1.13.3 + * + * \see H5Dread() + * + */ +H5_DLL herr_t H5Dread_multi(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], + hid_t file_space_id[], hid_t dxpl_id, void *buf[] /*out*/); + +/** + * -------------------------------------------------------------------------- * \ingroup ASYNC * \async_variant_of{H5Dread} */ @@ -861,6 +907,15 @@ H5_DLL herr_t H5Dread_async(const char *app_file, const char *app_func, unsigned /** * -------------------------------------------------------------------------- + * \ingroup ASYNC + * \async_variant_of{H5Dread_multi} + */ +H5_DLL herr_t H5Dread_multi_async(const char *app_file, const char *app_func, unsigned app_line, size_t count, + hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], + hid_t file_space_id[], hid_t dxpl_id, void *buf[] /*out*/, hid_t es_id); + +/** + * -------------------------------------------------------------------------- * \ingroup H5D * * \brief Writes raw data from a buffer to a dataset @@ -977,6 +1032,53 @@ H5_DLL herr_t H5Dwrite(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid /** * -------------------------------------------------------------------------- + * \ingroup H5D + * + * \brief Writes raw data from a set buffers to a set of datasets + * + * \param[in] count Number of datasets to write to + * \param[in] dset_id Identifiers of the datasets to write to + * \param[in] mem_type_id Identifiers of the memory datatypes + * \param[in] mem_space_id Identifiers of the memory dataspaces + * \param[in] file_space_id Identifiers of the datasets' dataspaces in the file + * \param[in] dxpl_id Identifier of a transfer property list + * \param[in] buf Buffers with data to be written to the file + * + * \return \herr_t + * + * \details H5Dwrite_multi() writes data to \p count datasets, whose identifiers + * are listed in the \p dset_id array, from multiple application memory + * buffers listed in the \p buf array. Data transfer properties are + * defined by the argument \p dxpl_id. The memory datatypes of each + * dataset are listed by identifier in the \p mem_type_id array. The + * parts of each dataset to write are listed by identifier in the \p + * file_space_id array, and the parts of each application memory buffer + * to write from are listed by identifier in the \p mem_space_id array. + * All array parameters have length \p count. + * + * This function will produce the same results as \p count calls to + * H5Dwrite(). Information listed in that function's documentation + * about the specifics of its behaviour also apply to H5Dwrite_multi(). + * By calling H5Dwrite_multi() instead of multiple calls to H5Dwrite(), + * however, the library can in some cases pass information about the + * entire I/O operation to the file driver, which can improve + * performance. + * + * All datasets must be in the same HDF5 file, and each unique dataset + * may only be listed once. If this function is called collectively in + * parallel, each rank must pass exactly the same list of datasets in + * \p dset_id , though the other parameters may differ. + * + * \since 1.13.3 + * + * \see H5Dwrite() + * + */ +H5_DLL herr_t H5Dwrite_multi(size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], + hid_t file_space_id[], hid_t dxpl_id, const void *buf[]); + +/** + * -------------------------------------------------------------------------- * \ingroup ASYNC * \async_variant_of{H5Dwrite} */ @@ -986,6 +1088,15 @@ H5_DLL herr_t H5Dwrite_async(const char *app_file, const char *app_func, unsigne /** * -------------------------------------------------------------------------- + * \ingroup ASYNC + * \async_variant_of{H5Dwrite_multi} + */ +H5_DLL herr_t H5Dwrite_multi_async(const char *app_file, const char *app_func, unsigned app_line, + size_t count, hid_t dset_id[], hid_t mem_type_id[], hid_t mem_space_id[], + hid_t file_space_id[], hid_t dxpl_id, const void *buf[], hid_t es_id); + +/** + * -------------------------------------------------------------------------- * \ingroup H5D * * \brief Writes a raw data chunk from a buffer directly to a dataset in a file |