summaryrefslogtreecommitdiffstats
path: root/src/H5Dpkg.h
diff options
context:
space:
mode:
authorNeil Fortner <fortnern@gmail.com>2022-10-19 16:13:15 (GMT)
committerGitHub <noreply@github.com>2022-10-19 16:13:15 (GMT)
commit93754cae33d4ed45850745664ce5e59f270f38f8 (patch)
treeac3154f44fe7969dacb452474c32fcaeb84e620d /src/H5Dpkg.h
parenta898cef6c2e277f74cc99302a3c49c39a7428b5b (diff)
downloadhdf5-93754cae33d4ed45850745664ce5e59f270f38f8.zip
hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.tar.gz
hdf5-93754cae33d4ed45850745664ce5e59f270f38f8.tar.bz2
Multi Dataset (#2120)
* Fix bug with cross platform compatibility of references within vlens. No testing yet. * Merge from multi_rd_wd_coll_io to a more recent branch from develop. Untested, probably does not work yet. * Committing clang-format changes * Committing clang-format changes * Fix many bugs in multi dataset branch. Mostly works, some issues in SWMR tests. * Committing clang-format changes * Disable test in swmr.c that was failing due to bug in HDF5 unrelated to multi dataset. * Committing clang-format changes * Fixed fortran multi-dataset tests * Fixed xlf errors * Added benchmark code for multi-datasets * loops over datasets * added missing error arg. * Added gnuplot formatting * Jonathan Kim original MD benchmarking code * updated MD benchmarking code * code clean-up * Only make files in feature test mode * misc clean-up * removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option * Committing clang-format changes * Change multi dataset API to use arrays of individual parameters instead of the parameter struct. * Committing clang-format changes * Update to new multi dataset Fortran API and tests. (#1724) * Update to new multi dataset Fortran API and tests. * Sync Fortran with develop. * skipping h5pget_mpio_actual_io_mode_f for now * Fixed issue with dxpl_id, changed to variable size dim. (#1770) * Remove "is_coll_broken" field from H5D_io_info_t struct * Committing clang-format changes * Minor cleanup in multi dataset code. * Committing clang-format changes * Clean up in multi dataset code. * Committing clang-format changes * Committing clang-format changes * Fix speeling * Fix bug in parallel compression. Switch base_maddr in io_info to be a union. * Committing clang-format changes * Implement selection I/O support with multi dataset. Will be broken in parallel until PR 1803 is merged to develop then the MDS branch. * Committing clang-format changes * Spelling * Fix bug in multi dataset that could cause errors when only some of the datasets in the multi dataset I/O used type conversion. * Committing clang-format changes * Integrate multi dataset APIs with VOL layer. Add async versions of multi dataset APIs. * Committing clang-format changes * Spelling fixes * Fix bug in non-parallel HDF5 compilation. * Committing clang-format changes * Fix potential memory/free list error. Minor performance fix. Other minor changes. * Committing clang-format changes * Fix memory leak with memory dataspace for I/O. * Committing clang-format changes * Fix stack variables too large. Rename H5D_dset_info_t to H5D_dset_io_info_t. * Committing clang-format changes * Remove mem_space_alloc field from H5D_dset_io_info_t. Each function is now responsible for freeing any spaces it adds to dset_info. * Committing clang-format changes * fixed _multi Fortran declaration * Refactor various things in (mostly) the serial I/O code path to make things more maintainable. * Committing clang-format changes * updated to array based, doxygen, and examples * Reinstate H5D_chunk_map_t, stored (via pointer) inside H5D_dset_io_info_t. * Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t. Switch temporary dset_infos to be local stack variables. * Committing clang-format changes * format cleanup * format cleanup * added coll and ind * Modify all parallel I/O paths to take dset_info instead of assuming dset_info[0]. * Committing clang-format changes * fixed output * Rework parallel I/O code to work properly with multi dataset in more cases. Fix bug in parallel compression. * Committing clang-format changes * Prevent H5D__multi_chunk_collective_io() from messing up collective opt property for other datasets in I/O. Other minor cleanup. Add new test case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now due to failures apparently unrelated to multi dataset code. * Fix spelling * Committing clang-format changes * Replace N log N algorithm for finding chunk in H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of io_info->sel_pieces in that function. * Committing clang-format changes * Replace sel_pieces skiplist in io_info with flat array of pointers, use qsort in I/O routine only when necessary. * Committing clang-format changes * Add new test case to mdset.c * Committing clang-format changes * Fix spelling * Very minor fix in H5VL__native_dataset_read() * Fix bug that could affect filtered parallel multi-dataset I/O. * Add RM entries for H5Dread_multi(), H5Dread_multi_async(), H5Dwrite_multi(), and H5Dwrite_multi_async() * Unskip test in swmr.c * Committing clang-format changes * Eliminate H5D__pre_read and H5D__pre_write * Remove examples/ph5mdsettest.c. Will fix and re-add as a test. * Enable hyperslab combinations in mdset test * Committing clang-format changes * Clarify H5Dread/write_multi documentation. * Fix bugs in multi-dataset I/O. Expand serial multi dataset test. Update macro in parallel multi dataset test. * Committing clang-format changes * Spelling * Remove obsolete entry in bin/trace * Rework type conversion buffer allocation. Only one buffer is shared between datasets in mdset mode, and it is malloced instead of calloced. * Committing clang-format changes * Fix bug in error handling in H5D__read/write * added multi-dataset fortran check with optional dataset creation id (#2150) * removed dup. dll entry * Address comments from code review. * Remove spurious changes in H5Fmpi.c * Fix issue with reading unallocated datasets in multi-dataset mode. Address other comments from code review. * Committing clang-format changes * Delay chunk index lookup from io_init to mdio_init so it doesn't add overhead to single dataset I/O. * Committing clang-format changes * Fix inappropriate use of piece_count * updated copyright on new file, removed benchmark from testing dir. Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org> Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>
Diffstat (limited to 'src/H5Dpkg.h')
-rw-r--r--src/H5Dpkg.h323
1 files changed, 169 insertions, 154 deletions
diff --git a/src/H5Dpkg.h b/src/H5Dpkg.h
index 0b98ce7..557d745 100644
--- a/src/H5Dpkg.h
+++ b/src/H5Dpkg.h
@@ -33,6 +33,7 @@
#include "H5ACprivate.h" /* Metadata cache */
#include "H5B2private.h" /* v2 B-trees */
#include "H5Fprivate.h" /* File access */
+#include "H5FLprivate.h" /* Free Lists */
#include "H5Gprivate.h" /* Groups */
#include "H5SLprivate.h" /* Skip lists */
#include "H5Tprivate.h" /* Datatypes */
@@ -44,20 +45,6 @@
/* Set the minimum object header size to create objects with */
#define H5D_MINHDR_SIZE 256
-/* [Simple] Macro to construct a H5D_io_info_t from it's components */
-#define H5D_BUILD_IO_INFO_WRT(io_info, ds, str, buf) \
- (io_info)->dset = ds; \
- (io_info)->f_sh = H5F_SHARED((ds)->oloc.file); \
- (io_info)->store = str; \
- (io_info)->op_type = H5D_IO_OP_WRITE; \
- (io_info)->u.wbuf = buf
-#define H5D_BUILD_IO_INFO_RD(io_info, ds, str, buf) \
- (io_info)->dset = ds; \
- (io_info)->f_sh = H5F_SHARED((ds)->oloc.file); \
- (io_info)->store = str; \
- (io_info)->op_type = H5D_IO_OP_READ; \
- (io_info)->u.rbuf = buf
-
/* Flags for marking aspects of a dataset dirty */
#define H5D_MARK_SPACE 0x01
#define H5D_MARK_LAYOUT 0x02
@@ -83,6 +70,9 @@
#define H5D_BT2_SPLIT_PERC 100
#define H5D_BT2_MERGE_PERC 40
+/* Macro to determine if the layout I/O callback should perform I/O */
+#define H5D_LAYOUT_CB_PERFORM_IO(IO_INFO) (!(IO_INFO)->use_select_io || (IO_INFO)->count == 1)
+
/****************************/
/* Package Private Typedefs */
/****************************/
@@ -97,23 +87,20 @@ typedef struct H5D_type_info_t {
hid_t dst_type_id; /* Destination datatype ID */
/* Computed/derived values */
- size_t src_type_size; /* Size of source type */
- size_t dst_type_size; /* Size of destination type */
- size_t max_type_size; /* Size of largest source/destination type */
- hbool_t is_conv_noop; /* Whether the type conversion is a NOOP */
- hbool_t is_xform_noop; /* Whether the data transform is a NOOP */
- const H5T_subset_info_t *cmpd_subset; /* Info related to the compound subset conversion functions */
- H5T_bkg_t need_bkg; /* Type of background buf needed */
- size_t request_nelmts; /* Requested strip mine */
- uint8_t *tconv_buf; /* Datatype conv buffer */
- hbool_t tconv_buf_allocated; /* Whether the type conversion buffer was allocated */
- uint8_t *bkg_buf; /* Background buffer */
- hbool_t bkg_buf_allocated; /* Whether the background buffer was allocated */
+ size_t src_type_size; /* Size of source type */
+ size_t dst_type_size; /* Size of destination type */
+ hbool_t is_conv_noop; /* Whether the type conversion is a NOOP */
+ hbool_t is_xform_noop; /* Whether the data transform is a NOOP */
+ const H5T_subset_info_t *cmpd_subset; /* Info related to the compound subset conversion functions */
+ H5T_bkg_t need_bkg; /* Type of background buf needed */
+ size_t request_nelmts; /* Requested strip mine */
+ uint8_t *bkg_buf; /* Background buffer */
+ hbool_t bkg_buf_allocated; /* Whether the background buffer was allocated */
} H5D_type_info_t;
/* Forward declaration of structs used below */
struct H5D_io_info_t;
-struct H5D_chunk_map_t;
+struct H5D_dset_io_info_t;
typedef struct H5D_shared_t H5D_shared_t;
/* Function pointers for I/O on particular types of dataset layouts */
@@ -121,27 +108,27 @@ typedef herr_t (*H5D_layout_construct_func_t)(H5F_t *f, H5D_t *dset);
typedef herr_t (*H5D_layout_init_func_t)(H5F_t *f, const H5D_t *dset, hid_t dapl_id);
typedef hbool_t (*H5D_layout_is_space_alloc_func_t)(const H5O_storage_t *storage);
typedef hbool_t (*H5D_layout_is_data_cached_func_t)(const H5D_shared_t *shared_dset);
-typedef herr_t (*H5D_layout_io_init_func_t)(struct H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- struct H5D_chunk_map_t *cm);
-typedef herr_t (*H5D_layout_read_func_t)(struct H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- struct H5D_chunk_map_t *fm);
-typedef herr_t (*H5D_layout_write_func_t)(struct H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- struct H5D_chunk_map_t *fm);
-typedef ssize_t (*H5D_layout_readvv_func_t)(const struct H5D_io_info_t *io_info, size_t dset_max_nseq,
+typedef herr_t (*H5D_layout_io_init_func_t)(struct H5D_io_info_t *io_info, struct H5D_dset_io_info_t *dinfo);
+typedef herr_t (*H5D_layout_mdio_init_func_t)(struct H5D_io_info_t *io_info,
+ struct H5D_dset_io_info_t *dinfo);
+typedef herr_t (*H5D_layout_read_func_t)(struct H5D_io_info_t *io_info, struct H5D_dset_io_info_t *dinfo);
+typedef herr_t (*H5D_layout_write_func_t)(struct H5D_io_info_t *io_info, struct H5D_dset_io_info_t *dinfo);
+typedef herr_t (*H5D_layout_read_md_func_t)(struct H5D_io_info_t *io_info);
+typedef herr_t (*H5D_layout_write_md_func_t)(struct H5D_io_info_t *io_info);
+typedef ssize_t (*H5D_layout_readvv_func_t)(const struct H5D_io_info_t *io_info,
+ const struct H5D_dset_io_info_t *dset_info, size_t dset_max_nseq,
size_t *dset_curr_seq, size_t dset_len_arr[],
hsize_t dset_offset_arr[], size_t mem_max_nseq,
size_t *mem_curr_seq, size_t mem_len_arr[],
hsize_t mem_offset_arr[]);
-typedef ssize_t (*H5D_layout_writevv_func_t)(const struct H5D_io_info_t *io_info, size_t dset_max_nseq,
+typedef ssize_t (*H5D_layout_writevv_func_t)(const struct H5D_io_info_t *io_info,
+ const struct H5D_dset_io_info_t *dset_info, size_t dset_max_nseq,
size_t *dset_curr_seq, size_t dset_len_arr[],
hsize_t dset_offset_arr[], size_t mem_max_nseq,
size_t *mem_curr_seq, size_t mem_len_arr[],
hsize_t mem_offset_arr[]);
typedef herr_t (*H5D_layout_flush_func_t)(H5D_t *dataset);
-typedef herr_t (*H5D_layout_io_term_func_t)(const struct H5D_chunk_map_t *cm);
+typedef herr_t (*H5D_layout_io_term_func_t)(struct H5D_io_info_t *io_info, struct H5D_dset_io_info_t *di);
typedef herr_t (*H5D_layout_dest_func_t)(H5D_t *dataset);
/* Typedef for grouping layout I/O routines */
@@ -152,27 +139,28 @@ typedef struct H5D_layout_ops_t {
H5D_layout_is_data_cached_func_t
is_data_cached; /* Query routine to determine if any raw data is cached. If routine is not present
then the layout type never caches raw data. */
- H5D_layout_io_init_func_t io_init; /* I/O initialization routine */
- H5D_layout_read_func_t ser_read; /* High-level I/O routine for reading data in serial */
- H5D_layout_write_func_t ser_write; /* High-level I/O routine for writing data in serial */
-#ifdef H5_HAVE_PARALLEL
- H5D_layout_read_func_t par_read; /* High-level I/O routine for reading data in parallel */
- H5D_layout_write_func_t par_write; /* High-level I/O routine for writing data in parallel */
-#endif /* H5_HAVE_PARALLEL */
- H5D_layout_readvv_func_t readvv; /* Low-level I/O routine for reading data */
- H5D_layout_writevv_func_t writevv; /* Low-level I/O routine for writing data */
- H5D_layout_flush_func_t flush; /* Low-level I/O routine for flushing raw data */
- H5D_layout_io_term_func_t io_term; /* I/O shutdown routine */
- H5D_layout_dest_func_t dest; /* Destroy layout info */
+ H5D_layout_io_init_func_t io_init; /* I/O initialization routine */
+ H5D_layout_mdio_init_func_t mdio_init; /* Multi Dataset I/O initialization routine - called after all
+ datasets have done io_init and sel_pieces has been allocated */
+ H5D_layout_read_func_t ser_read; /* High-level I/O routine for reading data in serial */
+ H5D_layout_write_func_t ser_write; /* High-level I/O routine for writing data in serial */
+ H5D_layout_readvv_func_t readvv; /* Low-level I/O routine for reading data */
+ H5D_layout_writevv_func_t writevv; /* Low-level I/O routine for writing data */
+ H5D_layout_flush_func_t flush; /* Low-level I/O routine for flushing raw data */
+ H5D_layout_io_term_func_t io_term; /* I/O shutdown routine for multi-dset */
+ H5D_layout_dest_func_t dest; /* Destroy layout info */
} H5D_layout_ops_t;
/* Function pointers for either multiple or single block I/O access */
-typedef herr_t (*H5D_io_single_read_func_t)(const struct H5D_io_info_t *io_info,
- const H5D_type_info_t *type_info, hsize_t nelmts,
- H5S_t *file_space, H5S_t *mem_space);
-typedef herr_t (*H5D_io_single_write_func_t)(const struct H5D_io_info_t *io_info,
- const H5D_type_info_t *type_info, hsize_t nelmts,
- H5S_t *file_space, H5S_t *mem_space);
+typedef herr_t (*H5D_io_single_read_func_t)(const struct H5D_io_info_t *io_info,
+ const struct H5D_dset_io_info_t *dset_info);
+typedef herr_t (*H5D_io_single_write_func_t)(const struct H5D_io_info_t *io_info,
+ const struct H5D_dset_io_info_t *dset_info);
+
+typedef herr_t (*H5D_io_single_read_md_func_t)(const struct H5D_io_info_t *io_info, hsize_t nelmts,
+ H5S_t *file_space, H5S_t *mem_space);
+typedef herr_t (*H5D_io_single_write_md_func_t)(const struct H5D_io_info_t *io_info, hsize_t nelmts,
+ H5S_t *file_space, H5S_t *mem_space);
/* Typedef for raw data I/O framework info */
typedef struct H5D_io_ops_t {
@@ -182,6 +170,14 @@ typedef struct H5D_io_ops_t {
H5D_io_single_write_func_t single_write; /* I/O routine for writing single block */
} H5D_io_ops_t;
+/* Typedef for raw data I/O framework info (multi-dataset I/O) */
+typedef struct H5D_md_io_ops_t {
+ H5D_layout_read_md_func_t multi_read_md; /* High-level I/O routine for reading data for multi-dset */
+ H5D_layout_write_md_func_t multi_write_md; /* High-level I/O routine for writing data for multi-dset */
+ H5D_io_single_read_md_func_t single_read_md; /* I/O routine for reading single block for multi-dset */
+ H5D_io_single_write_md_func_t single_write_md; /* I/O routine for writing single block for multi-dset */
+} H5D_md_io_ops_t;
+
/* Typedefs for dataset storage information */
typedef struct {
haddr_t dset_addr; /* Address of dataset in file */
@@ -210,25 +206,79 @@ typedef enum H5D_io_op_type_t {
H5D_IO_OP_WRITE /* Write operation */
} H5D_io_op_type_t;
+/* Piece info for a data chunk/block during I/O */
+typedef struct H5D_piece_info_t {
+ haddr_t faddr; /* File address */
+ hsize_t index; /* "Index" of chunk in dataset */
+ hsize_t piece_points; /* Number of elements selected in piece */
+ hsize_t scaled[H5O_LAYOUT_NDIMS]; /* Scaled coordinates of chunk (in file dataset's dataspace) */
+ H5S_t *fspace; /* Dataspace describing chunk & selection in it */
+ unsigned fspace_shared; /* Indicate that the file space for a chunk is shared and shouldn't be freed */
+ H5S_t *mspace; /* Dataspace describing selection in memory corresponding to this chunk */
+ unsigned mspace_shared; /* Indicate that the memory space for a chunk is shared and shouldn't be freed */
+ struct H5D_dset_io_info_t *dset_info; /* Pointer to dset_info */
+} H5D_piece_info_t;
+
+/* I/O info for a single dataset */
+typedef struct H5D_dset_io_info_t {
+ H5D_t *dset; /* Pointer to dataset being operated on */
+ H5D_storage_t *store; /* Dataset storage info */
+ H5D_layout_ops_t layout_ops; /* Dataset layout I/O operation function pointers */
+ H5_flexible_const_ptr_t buf; /* Buffer pointer */
+
+ H5D_io_ops_t io_ops; /* I/O operations for this dataset */
+
+ H5O_layout_t *layout; /* Dataset layout information*/
+ hsize_t nelmts; /* Number of elements selected in file & memory dataspaces */
+
+ H5S_t *file_space; /* Pointer to the file dataspace */
+ H5S_t *mem_space; /* Pointer to the memory dataspace */
+
+ union {
+ struct H5D_chunk_map_t *chunk_map; /* Chunk specific I/O info */
+ H5D_piece_info_t *contig_piece_info; /* Piece info for contiguous dataset */
+ } layout_io_info;
+
+ hid_t mem_type_id; /* memory datatype ID */
+ H5D_type_info_t type_info;
+ hbool_t skip_io; /* Whether to skip I/O for this dataset */
+} H5D_dset_io_info_t;
+
+/* I/O info for entire I/O operation */
typedef struct H5D_io_info_t {
- const H5D_t *dset; /* Pointer to dataset being operated on */
- /* QAK: Delete the f_sh field when oloc has a shared file pointer? */
+ /* QAK: Delete the f_sh field when oloc has a shared file pointer? */
H5F_shared_t *f_sh; /* Pointer to shared file struct that dataset is within */
#ifdef H5_HAVE_PARALLEL
- MPI_Comm comm; /* MPI communicator for file */
- hbool_t using_mpi_vfd; /* Whether the file is using an MPI-based VFD */
-#endif /* H5_HAVE_PARALLEL */
- H5D_storage_t *store; /* Dataset storage info */
- H5D_layout_ops_t layout_ops; /* Dataset layout I/O operation function pointers */
- H5D_io_ops_t io_ops; /* I/O operation function pointers */
- H5D_io_op_type_t op_type;
- hbool_t use_select_io; /* Whether to use selection I/O */
- union {
- void *rbuf; /* Pointer to buffer for read */
- const void *wbuf; /* Pointer to buffer to write */
- } u;
+ MPI_Comm comm; /* MPI communicator for file */
+ hbool_t using_mpi_vfd; /* Whether the file is using an MPI-based VFD */
+#endif /* H5_HAVE_PARALLEL */
+ H5D_md_io_ops_t md_io_ops; /* Multi dataset I/O operation function pointers */
+ H5D_io_op_type_t op_type;
+ size_t count; /* Number of datasets in I/O request */
+ H5D_dset_io_info_t *dsets_info; /* dsets info where I/O is done to/from */
+ size_t piece_count; /* Number of pieces in I/O request */
+ size_t pieces_added; /* Number of pieces added so far to arrays */
+ H5D_piece_info_t **sel_pieces; /* Array of info struct for all pieces in I/O */
+ H5S_t **mem_spaces; /* Array of chunk memory spaces */
+ H5S_t **file_spaces; /* Array of chunk file spaces */
+ haddr_t *addrs; /* Array of chunk addresses */
+ size_t *element_sizes; /* Array of element sizes */
+ void **rbufs; /* Array of read buffers */
+ const void **wbufs; /* Array of write buffers */
+ haddr_t store_faddr; /* lowest file addr for read/write */
+ H5_flexible_const_ptr_t base_maddr; /* starting mem address */
+ hbool_t use_select_io; /* Whether to use selection I/O */
+ uint8_t *tconv_buf; /* Datatype conv buffer */
+ hbool_t tconv_buf_allocated; /* Whether the type conversion buffer was allocated */
+ size_t max_type_size; /* Largest of all source and destination type sizes */
} H5D_io_info_t;
+/* Created to pass both at once for callback func */
+typedef struct H5D_io_info_wrap_t {
+ H5D_io_info_t *io_info;
+ H5D_dset_io_info_t *dinfo;
+} H5D_io_info_wrap_t;
+
/******************/
/* Chunk typedefs */
/******************/
@@ -330,46 +380,26 @@ typedef struct H5D_chunk_ops_t {
H5D_chunk_dest_func_t dest; /* Routine to destroy indexing information in memory */
} H5D_chunk_ops_t;
-/* Structure holding information about a chunk's selection for mapping */
-typedef struct H5D_chunk_info_t {
- hsize_t index; /* "Index" of chunk in dataset */
- uint32_t chunk_points; /* Number of elements selected in chunk */
- hsize_t scaled[H5O_LAYOUT_NDIMS]; /* Scaled coordinates of chunk (in file dataset's dataspace) */
- H5S_t *fspace; /* Dataspace describing chunk & selection in it */
- hbool_t fspace_shared; /* Indicate that the file space for a chunk is shared and shouldn't be freed */
- H5S_t *mspace; /* Dataspace describing selection in memory corresponding to this chunk */
- hbool_t mspace_shared; /* Indicate that the memory space for a chunk is shared and shouldn't be freed */
-} H5D_chunk_info_t;
-
/* Main structure holding the mapping between file chunks and memory */
typedef struct H5D_chunk_map_t {
- H5O_layout_t *layout; /* Dataset layout information*/
- hsize_t nelmts; /* Number of elements selected in file & memory dataspaces */
+ unsigned f_ndims; /* Number of dimensions for file dataspace */
- H5S_t *file_space; /* Pointer to the file dataspace */
- unsigned f_ndims; /* Number of dimensions for file dataspace */
-
- H5S_t *mem_space; /* Pointer to the memory dataspace */
H5S_t *mchunk_tmpl; /* Dataspace template for new memory chunks */
H5S_sel_iter_t mem_iter; /* Iterator for elements in memory selection */
unsigned m_ndims; /* Number of dimensions for memory dataspace */
H5S_sel_type msel_type; /* Selection type in memory */
H5S_sel_type fsel_type; /* Selection type in file */
- H5SL_t *sel_chunks; /* Skip list containing information for each chunk selected */
+ H5SL_t *dset_sel_pieces; /* Skip list containing information for each chunk selected */
H5S_t *single_space; /* Dataspace for single chunk */
- H5D_chunk_info_t *single_chunk_info; /* Pointer to single chunk's info */
+ H5D_piece_info_t *single_piece_info; /* Pointer to single chunk's info */
hbool_t use_single; /* Whether I/O is on a single element */
hsize_t last_index; /* Index of last chunk operated on */
- H5D_chunk_info_t *last_chunk_info; /* Pointer to last chunk's info */
+ H5D_piece_info_t *last_piece_info; /* Pointer to last chunk's info */
hsize_t chunk_dim[H5O_LAYOUT_NDIMS]; /* Size of chunk in each dimension */
-
-#ifdef H5_HAVE_PARALLEL
- H5D_chunk_info_t **select_chunk; /* Store the information about whether this chunk is selected or not */
-#endif /* H5_HAVE_PARALLEL */
} H5D_chunk_map_t;
/* Cached information about a particular chunk */
@@ -415,7 +445,7 @@ typedef struct H5D_rdcc_t {
struct H5D_rdcc_ent_t **slot; /* Chunk slots, each points to a chunk*/
H5SL_t *sel_chunks; /* Skip list containing information for each chunk selected */
H5S_t *single_space; /* Dataspace for single element I/O on chunks */
- H5D_chunk_info_t *single_chunk_info; /* Pointer to single chunk's info */
+ H5D_piece_info_t *single_piece_info; /* Pointer to single piece's info */
/* Cached information about scaled dataspace dimensions */
hsize_t scaled_dims[H5S_MAX_RANK]; /* The scaled dim sizes */
@@ -458,12 +488,13 @@ struct H5D_shared_t {
/* Buffered/cached information for types of raw data storage*/
struct {
- H5D_rdcdc_t contig; /* Information about contiguous data */
- /* (Note that the "contig" cache
- * information can be used by a chunked
- * dataset in certain circumstances)
- */
- H5D_rdcc_t chunk; /* Information about chunked data */
+ H5D_rdcdc_t contig; /* Information about contiguous data */
+ /* (Note that the "contig" cache
+ * information can be used by a chunked
+ * dataset in certain circumstances)
+ */
+ H5D_rdcc_t chunk; /* Information about chunked data */
+ H5SL_t *sel_pieces; /* Skip list containing information for each piece selected */
} cache;
H5D_append_flush_t append_flush; /* Append flush property information */
@@ -556,8 +587,8 @@ H5_DLL H5D_t *H5D__open_name(const H5G_loc_t *loc, const char *name, hid_t dapl
H5_DLL hid_t H5D__get_space(const H5D_t *dset);
H5_DLL hid_t H5D__get_type(const H5D_t *dset);
H5_DLL herr_t H5D__get_space_status(const H5D_t *dset, H5D_space_status_t *allocation);
-H5_DLL herr_t H5D__alloc_storage(const H5D_io_info_t *io_info, H5D_time_alloc_t time_alloc,
- hbool_t full_overwrite, hsize_t old_dim[]);
+H5_DLL herr_t H5D__alloc_storage(H5D_t *dset, H5D_time_alloc_t time_alloc, hbool_t full_overwrite,
+ hsize_t old_dim[]);
H5_DLL herr_t H5D__get_storage_size(const H5D_t *dset, hsize_t *storage_size);
H5_DLL herr_t H5D__get_chunk_storage_size(H5D_t *dset, const hsize_t *offset, hsize_t *storage_size);
H5_DLL herr_t H5D__chunk_index_empty(const H5D_t *dset, hbool_t *empty);
@@ -582,16 +613,12 @@ H5_DLL herr_t H5D__refresh(H5D_t *dataset, hid_t dset_id);
H5_DLL herr_t H5D__format_convert(H5D_t *dataset);
/* Internal I/O routines */
-H5_DLL herr_t H5D__read(H5D_t *dataset, hid_t mem_type_id, H5S_t *mem_space, H5S_t *file_space,
- void *buf /*out*/);
-H5_DLL herr_t H5D__write(H5D_t *dataset, hid_t mem_type_id, H5S_t *mem_space, H5S_t *file_space,
- const void *buf);
+H5_DLL herr_t H5D__read(size_t count, H5D_dset_io_info_t *dset_info);
+H5_DLL herr_t H5D__write(size_t count, H5D_dset_io_info_t *dset_info);
/* Functions that perform direct serial I/O operations */
-H5_DLL herr_t H5D__select_read(const H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts,
- H5S_t *file_space, H5S_t *mem_space);
-H5_DLL herr_t H5D__select_write(const H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space);
+H5_DLL herr_t H5D__select_read(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info);
+H5_DLL herr_t H5D__select_write(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info);
/* Functions that perform direct copying between memory buffers */
H5_DLL herr_t H5D_select_io_mem(void *dst_buf, H5S_t *dst_space, const void *src_buf, H5S_t *src_space,
@@ -601,10 +628,8 @@ H5_DLL herr_t H5D_select_io_mem(void *dst_buf, H5S_t *dst_space, const void *src
H5_DLL herr_t H5D__scatter_mem(const void *_tscat_buf, H5S_sel_iter_t *iter, size_t nelmts, void *_buf);
H5_DLL size_t H5D__gather_mem(const void *_buf, H5S_sel_iter_t *iter, size_t nelmts,
void *_tgath_buf /*out*/);
-H5_DLL herr_t H5D__scatgath_read(const H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space);
-H5_DLL herr_t H5D__scatgath_write(const H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space);
+H5_DLL herr_t H5D__scatgath_read(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info);
+H5_DLL herr_t H5D__scatgath_write(const H5D_io_info_t *io_info, const H5D_dset_io_info_t *dset_info);
/* Functions that operate on dataset's layout information */
H5_DLL herr_t H5D__layout_set_io_ops(const H5D_t *dataset);
@@ -620,25 +645,23 @@ H5_DLL herr_t H5D__layout_oh_write(const H5D_t *dataset, H5O_t *oh, unsigned upd
H5_DLL herr_t H5D__contig_alloc(H5F_t *f, H5O_storage_contig_t *storage);
H5_DLL hbool_t H5D__contig_is_space_alloc(const H5O_storage_t *storage);
H5_DLL hbool_t H5D__contig_is_data_cached(const H5D_shared_t *shared_dset);
-H5_DLL herr_t H5D__contig_fill(const H5D_io_info_t *io_info);
-H5_DLL herr_t H5D__contig_read(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts,
- H5S_t *file_space, H5S_t *mem_space, H5D_chunk_map_t *fm);
-H5_DLL herr_t H5D__contig_write(H5D_io_info_t *io_info, const H5D_type_info_t *type_info, hsize_t nelmts,
- H5S_t *file_space, H5S_t *mem_space, H5D_chunk_map_t *fm);
+H5_DLL herr_t H5D__contig_fill(H5D_t *dset);
+H5_DLL herr_t H5D__contig_read(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo);
+H5_DLL herr_t H5D__contig_write(H5D_io_info_t *io_info, H5D_dset_io_info_t *dinfo);
H5_DLL herr_t H5D__contig_copy(H5F_t *f_src, const H5O_storage_contig_t *storage_src, H5F_t *f_dst,
H5O_storage_contig_t *storage_dst, H5T_t *src_dtype, H5O_copy_t *cpy_info);
H5_DLL herr_t H5D__contig_delete(H5F_t *f, const H5O_storage_t *store);
/* Functions that operate on chunked dataset storage */
-H5_DLL htri_t H5D__chunk_cacheable(const H5D_io_info_t *io_info, haddr_t caddr, hbool_t write_op);
-H5_DLL herr_t H5D__chunk_create(const H5D_t *dset /*in,out*/);
-H5_DLL herr_t H5D__chunk_set_info(const H5D_t *dset);
+H5_DLL htri_t H5D__chunk_cacheable(const H5D_io_info_t *io_info, H5D_dset_io_info_t *dset_info, haddr_t caddr,
+ hbool_t write_op);
+H5_DLL herr_t H5D__chunk_create(const H5D_t *dset /*in,out*/);
+H5_DLL herr_t H5D__chunk_set_info(const H5D_t *dset);
H5_DLL hbool_t H5D__chunk_is_space_alloc(const H5O_storage_t *storage);
H5_DLL hbool_t H5D__chunk_is_data_cached(const H5D_shared_t *shared_dset);
H5_DLL herr_t H5D__chunk_lookup(const H5D_t *dset, const hsize_t *scaled, H5D_chunk_ud_t *udata);
H5_DLL herr_t H5D__chunk_allocated(const H5D_t *dset, hsize_t *nbytes);
-H5_DLL herr_t H5D__chunk_allocate(const H5D_io_info_t *io_info, hbool_t full_overwrite,
- const hsize_t old_dim[]);
+H5_DLL herr_t H5D__chunk_allocate(const H5D_t *dset, hbool_t full_overwrite, const hsize_t old_dim[]);
H5_DLL herr_t H5D__chunk_file_alloc(const H5D_chk_idx_info_t *idx_info, const H5F_block_t *old_chunk,
H5F_block_t *new_chunk, hbool_t *need_insert, const hsize_t *scaled);
H5_DLL void *H5D__chunk_mem_alloc(size_t size, void *pline);
@@ -651,7 +674,7 @@ H5_DLL hbool_t H5D__chunk_is_partial_edge_chunk(unsigned dset_ndims, const uint3
H5_DLL herr_t H5D__chunk_prune_by_extent(H5D_t *dset, const hsize_t *old_dim);
H5_DLL herr_t H5D__chunk_set_sizes(H5D_t *dset);
#ifdef H5_HAVE_PARALLEL
-H5_DLL herr_t H5D__chunk_addrmap(const H5D_io_info_t *io_info, haddr_t chunk_addr[]);
+H5_DLL herr_t H5D__chunk_addrmap(const H5D_t *dset, haddr_t chunk_addr[]);
#endif /* H5_HAVE_PARALLEL */
H5_DLL herr_t H5D__chunk_update_cache(H5D_t *dset);
H5_DLL herr_t H5D__chunk_copy(H5F_t *f_src, H5O_storage_chunk_t *storage_src, H5O_layout_chunk_t *layout_src,
@@ -662,8 +685,8 @@ H5_DLL herr_t H5D__chunk_bh_info(const H5O_loc_t *loc, H5O_t *oh, H5O_layout_t *
H5_DLL herr_t H5D__chunk_dump_index(H5D_t *dset, FILE *stream);
H5_DLL herr_t H5D__chunk_delete(H5F_t *f, H5O_t *oh, H5O_storage_t *store);
H5_DLL herr_t H5D__chunk_get_offset_copy(const H5D_t *dset, const hsize_t *offset, hsize_t *offset_copy);
-H5_DLL herr_t H5D__chunk_direct_write(const H5D_t *dset, uint32_t filters, hsize_t *offset,
- uint32_t data_size, const void *buf);
+H5_DLL herr_t H5D__chunk_direct_write(H5D_t *dset, uint32_t filters, hsize_t *offset, uint32_t data_size,
+ const void *buf);
H5_DLL herr_t H5D__chunk_direct_read(const H5D_t *dset, hsize_t *offset, uint32_t *filters, void *buf);
#ifdef H5D_CHUNK_DEBUG
H5_DLL herr_t H5D__chunk_stats(const H5D_t *dset, hbool_t headers);
@@ -711,39 +734,31 @@ H5_DLL herr_t H5D__fill_term(H5D_fill_buf_info_t *fb_info);
#define H5Dmpio_DEBUG
#endif /*H5Dmpio_DEBUG*/
#endif /*H5D_DEBUG*/
-/* MPI-IO function to read, it will select either regular or irregular read */
-H5_DLL herr_t H5D__mpio_select_read(const H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space);
-
-/* MPI-IO function to write, it will select either regular or irregular read */
-H5_DLL herr_t H5D__mpio_select_write(const H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space);
-
-/* MPI-IO functions to handle contiguous collective IO */
-H5_DLL herr_t H5D__contig_collective_read(H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- H5D_chunk_map_t *fm);
-H5_DLL herr_t H5D__contig_collective_write(H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- H5D_chunk_map_t *fm);
-
-/* MPI-IO functions to handle chunked collective IO */
-H5_DLL herr_t H5D__chunk_collective_read(H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- H5D_chunk_map_t *fm);
-H5_DLL herr_t H5D__chunk_collective_write(H5D_io_info_t *io_info, const H5D_type_info_t *type_info,
- hsize_t nelmts, H5S_t *file_space, H5S_t *mem_space,
- H5D_chunk_map_t *fm);
+/* MPI-IO function to read multi-dsets (Chunk, Contig), it will select either
+ * regular or irregular read */
+H5_DLL herr_t H5D__mpio_select_read(const H5D_io_info_t *io_info, hsize_t nelmts, H5S_t *file_space,
+ H5S_t *mem_space);
+
+/* MPI-IO function to write multi-dsets (Chunk, Contig), it will select either
+ * regular or irregular write */
+H5_DLL herr_t H5D__mpio_select_write(const H5D_io_info_t *io_info, hsize_t nelmts, H5S_t *file_space,
+ H5S_t *mem_space);
+
+/* MPI-IO functions to handle collective IO for multiple dsets (CONTIG, CHUNK) */
+H5_DLL herr_t H5D__collective_read(H5D_io_info_t *io_info);
+H5_DLL herr_t H5D__collective_write(H5D_io_info_t *io_info);
/* MPI-IO function to check if a direct I/O transfer is possible between
* memory and the file */
-H5_DLL htri_t H5D__mpio_opt_possible(const H5D_io_info_t *io_info, const H5S_t *file_space,
- const H5S_t *mem_space, const H5D_type_info_t *type_info);
+H5_DLL htri_t H5D__mpio_opt_possible(H5D_io_info_t *io_info);
H5_DLL herr_t H5D__mpio_get_no_coll_cause_strings(char *local_cause, size_t local_cause_len,
char *global_cause, size_t global_cause_len);
#endif /* H5_HAVE_PARALLEL */
+/* Free a piece (chunk or contiguous dataset data block) info struct */
+H5_DLL herr_t H5D__free_piece_info(void *item, void *key, void *opdata);
+
/* Testing functions */
#ifdef H5D_TESTING
H5_DLL herr_t H5D__layout_version_test(hid_t did, unsigned *version);