diff options
author | Neil Fortner <fortnern@gmail.com> | 2022-10-19 16:13:15 (GMT) |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-10-19 16:13:15 (GMT) |
commit | 93754cae33d4ed45850745664ce5e59f270f38f8 (patch) | |
tree | ac3154f44fe7969dacb452474c32fcaeb84e620d /src/H5Dcompact.c | |
parent | a898cef6c2e277f74cc99302a3c49c39a7428b5b (diff) | |
download | hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.zip hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.tar.gz hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.tar.bz2 |
Multi Dataset (#2120)
* Fix bug with cross platform compatibility of references within vlens.
No testing yet.
* Merge from multi_rd_wd_coll_io to a more recent branch from develop.
Untested, probably does not work yet.
* Committing clang-format changes
* Committing clang-format changes
* Fix many bugs in multi dataset branch. Mostly works, some issues in
SWMR tests.
* Committing clang-format changes
* Disable test in swmr.c that was failing due to bug in HDF5 unrelated to
multi dataset.
* Committing clang-format changes
* Fixed fortran multi-dataset tests
* Fixed xlf errors
* Added benchmark code for multi-datasets
* loops over datasets
* added missing error arg.
* Added gnuplot formatting
* Jonathan Kim original MD benchmarking code
* updated MD benchmarking code
* code clean-up
* Only make files in feature test mode
* misc clean-up
* removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option
* Committing clang-format changes
* Change multi dataset API to use arrays of individual parameters instead
of the parameter struct.
* Committing clang-format changes
* Update to new multi dataset Fortran API and tests. (#1724)
* Update to new multi dataset Fortran API and tests.
* Sync Fortran with develop.
* skipping h5pget_mpio_actual_io_mode_f for now
* Fixed issue with dxpl_id, changed to variable size dim. (#1770)
* Remove "is_coll_broken" field from H5D_io_info_t struct
* Committing clang-format changes
* Minor cleanup in multi dataset code.
* Committing clang-format changes
* Clean up in multi dataset code.
* Committing clang-format changes
* Committing clang-format changes
* Fix speeling
* Fix bug in parallel compression. Switch base_maddr in io_info to be a
union.
* Committing clang-format changes
* Implement selection I/O support with multi dataset. Will be broken in
parallel until PR 1803 is merged to develop then the MDS branch.
* Committing clang-format changes
* Spelling
* Fix bug in multi dataset that could cause errors when only some of the
datasets in the multi dataset I/O used type conversion.
* Committing clang-format changes
* Integrate multi dataset APIs with VOL layer. Add async versions of
multi dataset APIs.
* Committing clang-format changes
* Spelling fixes
* Fix bug in non-parallel HDF5 compilation.
* Committing clang-format changes
* Fix potential memory/free list error. Minor performance fix. Other minor
changes.
* Committing clang-format changes
* Fix memory leak with memory dataspace for I/O.
* Committing clang-format changes
* Fix stack variables too large. Rename H5D_dset_info_t to
H5D_dset_io_info_t.
* Committing clang-format changes
* Remove mem_space_alloc field from H5D_dset_io_info_t. Each function is
now responsible for freeing any spaces it adds to dset_info.
* Committing clang-format changes
* fixed _multi Fortran declaration
* Refactor various things in (mostly) the serial I/O code path to make
things more maintainable.
* Committing clang-format changes
* updated to array based, doxygen, and examples
* Reinstate H5D_chunk_map_t, stored (via pointer) inside
H5D_dset_io_info_t.
* Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t.
Switch temporary dset_infos to be local stack variables.
* Committing clang-format changes
* format cleanup
* format cleanup
* added coll and ind
* Modify all parallel I/O paths to take dset_info instead of assuming
dset_info[0].
* Committing clang-format changes
* fixed output
* Rework parallel I/O code to work properly with multi dataset in more
cases. Fix bug in parallel compression.
* Committing clang-format changes
* Prevent H5D__multi_chunk_collective_io() from messing up collective opt
property for other datasets in I/O. Other minor cleanup. Add new test
case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now
due to failures apparently unrelated to multi dataset code.
* Fix spelling
* Committing clang-format changes
* Replace N log N algorithm for finding chunk in
H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of
io_info->sel_pieces in that function.
* Committing clang-format changes
* Replace sel_pieces skiplist in io_info with flat array of pointers, use
qsort in I/O routine only when necessary.
* Committing clang-format changes
* Add new test case to mdset.c
* Committing clang-format changes
* Fix spelling
* Very minor fix in H5VL__native_dataset_read()
* Fix bug that could affect filtered parallel multi-dataset I/O.
* Add RM entries for H5Dread_multi(), H5Dread_multi_async(),
H5Dwrite_multi(), and H5Dwrite_multi_async()
* Unskip test in swmr.c
* Committing clang-format changes
* Eliminate H5D__pre_read and H5D__pre_write
* Remove examples/ph5mdsettest.c. Will fix and re-add as a test.
* Enable hyperslab combinations in mdset test
* Committing clang-format changes
* Clarify H5Dread/write_multi documentation.
* Fix bugs in multi-dataset I/O. Expand serial multi dataset test.
Update macro in parallel multi dataset test.
* Committing clang-format changes
* Spelling
* Remove obsolete entry in bin/trace
* Rework type conversion buffer allocation. Only one buffer is shared
between datasets in mdset mode, and it is malloced instead of calloced.
* Committing clang-format changes
* Fix bug in error handling in H5D__read/write
* added multi-dataset fortran check with optional dataset creation id (#2150)
* removed dup. dll entry
* Address comments from code review.
* Remove spurious changes in H5Fmpi.c
* Fix issue with reading unallocated datasets in multi-dataset mode.
Address other comments from code review.
* Committing clang-format changes
* Delay chunk index lookup from io_init to mdio_init so it doesn't add
overhead to single dataset I/O.
* Committing clang-format changes
* Fix inappropriate use of piece_count
* updated copyright on new file, removed benchmark from testing dir.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>
Diffstat (limited to 'src/H5Dcompact.c')
-rw-r--r-- | src/H5Dcompact.c | 77 |
1 files changed, 40 insertions, 37 deletions
diff --git a/src/H5Dcompact.c b/src/H5Dcompact.c index 8ac1f0a..a9e937f 100644 --- a/src/H5Dcompact.c +++ b/src/H5Dcompact.c @@ -63,15 +63,16 @@ typedef struct H5D_compact_iovv_memmanage_ud_t { /* Layout operation callbacks */ static herr_t H5D__compact_construct(H5F_t *f, H5D_t *dset); static hbool_t H5D__compact_is_space_alloc(const H5O_storage_t *storage); -static herr_t H5D__compact_io_init(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts, - H5S_t *file_space, H5S_t *mem_space, H5D_chunk_map_t *cm); +static herr_t H5D__compact_io_init(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo); static herr_t H5D__compact_iovv_memmanage_cb(hsize_t dst_off, hsize_t src_off, size_t len, void *_udata); -static ssize_t H5D__compact_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq, - size_t dset_size_arr[], hsize_t dset_offset_arr[], size_t mem_max_nseq, - size_t *mem_curr_seq, size_t mem_size_arr[], hsize_t mem_offset_arr[]); -static ssize_t H5D__compact_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq, - size_t dset_size_arr[], hsize_t dset_offset_arr[], size_t mem_max_nseq, - size_t *mem_curr_seq, size_t mem_size_arr[], hsize_t mem_offset_arr[]); +static ssize_t H5D__compact_readvv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info, + size_t dset_max_nseq, size_t *dset_curr_seq, size_t dset_size_arr[], + hsize_t dset_offset_arr[], size_t mem_max_nseq, size_t *mem_curr_seq, + size_t mem_size_arr[], hsize_t mem_offset_arr[]); +static ssize_t H5D__compact_writevv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info, + size_t dset_max_nseq, size_t *dset_curr_seq, size_t dset_size_arr[], + hsize_t dset_offset_arr[], size_t mem_max_nseq, size_t *mem_curr_seq, + size_t mem_size_arr[], hsize_t mem_offset_arr[]); static herr_t H5D__compact_flush(H5D_t *dset); static herr_t H5D__compact_dest(H5D_t *dset); @@ -86,17 +87,14 @@ const H5D_layout_ops_t H5D_LOPS_COMPACT[1] = {{ H5D__compact_is_space_alloc, /* is_space_alloc */ NULL, /* is_data_cached */ H5D__compact_io_init, /* io_init */ + NULL, /* mdio_init */ H5D__contig_read, /* ser_read */ H5D__contig_write, /* ser_write */ -#ifdef H5_HAVE_PARALLEL - NULL, /* par_read */ - NULL, /* par_write */ -#endif - H5D__compact_readvv, /* readvv */ - H5D__compact_writevv, /* writevv */ - H5D__compact_flush, /* flush */ - NULL, /* io_term */ - H5D__compact_dest /* dest */ + H5D__compact_readvv, /* readvv */ + H5D__compact_writevv, /* writevv */ + H5D__compact_flush, /* flush */ + NULL, /* io_term */ + H5D__compact_dest /* dest */ }}; /*******************/ @@ -247,14 +245,15 @@ H5D__compact_is_space_alloc(const H5O_storage_t H5_ATTR_UNUSED *storage) *------------------------------------------------------------------------- */ static herr_t -H5D__compact_io_init(H5D_io_info_t *io_info, const H5D_type_info_t H5_ATTR_UNUSED *type_info, - hsize_t H5_ATTR_UNUSED nelmts, H5S_t H5_ATTR_UNUSED *file_space, - H5S_t H5_ATTR_UNUSED *mem_space, H5D_chunk_map_t H5_ATTR_UNUSED *cm) +H5D__compact_io_init(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo) { FUNC_ENTER_PACKAGE_NOERR - io_info->store->compact.buf = io_info->dset->shared->layout.storage.u.compact.buf; - io_info->store->compact.dirty = &io_info->dset->shared->layout.storage.u.compact.dirty; + dinfo->store->compact.buf = dinfo->dset->shared->layout.storage.u.compact.buf; + dinfo->store->compact.dirty = &dinfo->dset->shared->layout.storage.u.compact.dirty; + + /* Disable selection I/O */ + io_info->use_select_io = FALSE; FUNC_LEAVE_NOAPI(SUCCEED) } /* end H5D__compact_io_init() */ @@ -320,15 +319,17 @@ done: *------------------------------------------------------------------------- */ static ssize_t -H5D__compact_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq, - size_t dset_size_arr[], hsize_t dset_offset_arr[], size_t mem_max_nseq, - size_t *mem_curr_seq, size_t mem_size_arr[], hsize_t mem_offset_arr[]) +H5D__compact_readvv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info, size_t dset_max_nseq, + size_t *dset_curr_seq, size_t dset_size_arr[], hsize_t dset_offset_arr[], + size_t mem_max_nseq, size_t *mem_curr_seq, size_t mem_size_arr[], + hsize_t mem_offset_arr[]) { ssize_t ret_value = -1; /* Return value */ FUNC_ENTER_PACKAGE HDassert(io_info); + HDassert(dset_info); /* Check if file driver wishes to do its own memory management */ if (H5F_SHARED_HAS_FEATURE(io_info->f_sh, H5FD_FEAT_MEMMANAGE)) { @@ -336,8 +337,8 @@ H5D__compact_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t * /* Set up udata for memory copy operation */ udata.f_sh = io_info->f_sh; - udata.dstbuf = io_info->u.rbuf; - udata.srcbuf = io_info->store->compact.buf; + udata.dstbuf = dset_info->buf.vp; + udata.srcbuf = dset_info->store->compact.buf; /* Request that file driver does the memory copy */ if ((ret_value = H5VM_opvv(mem_max_nseq, mem_curr_seq, mem_size_arr, mem_offset_arr, dset_max_nseq, @@ -347,8 +348,8 @@ H5D__compact_readvv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t * } else { /* Use the vectorized memory copy routine to do actual work */ - if ((ret_value = H5VM_memcpyvv(io_info->u.rbuf, mem_max_nseq, mem_curr_seq, mem_size_arr, - mem_offset_arr, io_info->store->compact.buf, dset_max_nseq, + if ((ret_value = H5VM_memcpyvv(dset_info->buf.vp, mem_max_nseq, mem_curr_seq, mem_size_arr, + mem_offset_arr, dset_info->store->compact.buf, dset_max_nseq, dset_curr_seq, dset_size_arr, dset_offset_arr)) < 0) HGOTO_ERROR(H5E_IO, H5E_WRITEERROR, FAIL, "vectorized memcpy failed") } @@ -379,15 +380,17 @@ done: *------------------------------------------------------------------------- */ static ssize_t -H5D__compact_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t *dset_curr_seq, - size_t dset_size_arr[], hsize_t dset_offset_arr[], size_t mem_max_nseq, - size_t *mem_curr_seq, size_t mem_size_arr[], hsize_t mem_offset_arr[]) +H5D__compact_writevv(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info, size_t dset_max_nseq, + size_t *dset_curr_seq, size_t dset_size_arr[], hsize_t dset_offset_arr[], + size_t mem_max_nseq, size_t *mem_curr_seq, size_t mem_size_arr[], + hsize_t mem_offset_arr[]) { ssize_t ret_value = -1; /* Return value */ FUNC_ENTER_PACKAGE HDassert(io_info); + HDassert(dset_info); /* Check if file driver wishes to do its own memory management */ if (H5F_SHARED_HAS_FEATURE(io_info->f_sh, H5FD_FEAT_MEMMANAGE)) { @@ -395,8 +398,8 @@ H5D__compact_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t /* Set up udata for memory copy operation */ udata.f_sh = io_info->f_sh; - udata.dstbuf = io_info->store->compact.buf; - udata.srcbuf = io_info->u.wbuf; + udata.dstbuf = dset_info->store->compact.buf; + udata.srcbuf = dset_info->buf.cvp; /* Request that file driver does the memory copy */ if ((ret_value = H5VM_opvv(dset_max_nseq, dset_curr_seq, dset_size_arr, dset_offset_arr, mem_max_nseq, @@ -406,14 +409,14 @@ H5D__compact_writevv(const H5D_io_info_t *io_info, size_t dset_max_nseq, size_t } else { /* Use the vectorized memory copy routine to do actual work */ - if ((ret_value = H5VM_memcpyvv(io_info->store->compact.buf, dset_max_nseq, dset_curr_seq, - dset_size_arr, dset_offset_arr, io_info->u.wbuf, mem_max_nseq, + if ((ret_value = H5VM_memcpyvv(dset_info->store->compact.buf, dset_max_nseq, dset_curr_seq, + dset_size_arr, dset_offset_arr, dset_info->buf.cvp, mem_max_nseq, mem_curr_seq, mem_size_arr, mem_offset_arr)) < 0) HGOTO_ERROR(H5E_IO, H5E_WRITEERROR, FAIL, "vectorized memcpy failed") } /* Mark the compact dataset's buffer as dirty */ - *io_info->store->compact.dirty = TRUE; + *dset_info->store->compact.dirty = TRUE; done: FUNC_LEAVE_NOAPI(ret_value) |