diff options
| author | jhendersonHDF <jhenderson@hdfgroup.org> | 2023-12-07 18:13:21 (GMT) |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-12-07 18:13:21 (GMT) |
| commit | fb84c696a476c39db2c35e7d86b818138e089358 (patch) | |
| tree | 56c0fec578e2e0d2eaee455cf3ae133dca541989 /src | |
| parent | dc8b455ac00e899b53127de0d270230f4d4e40a1 (diff) | |
| download | hdf5-fb84c696a476c39db2c35e7d86b818138e089358.zip hdf5-fb84c696a476c39db2c35e7d86b818138e089358.tar.gz hdf5-fb84c696a476c39db2c35e7d86b818138e089358.tar.bz2 | |
Sync 1.14 branch with develop (#3887)
* Preserve MPI-I/O file hints when fapl is closed (#3755)
* Fix for issue #3025: Save the MPI info in the file struct so H5Fget_access_plist() can retrieve it from there.
* Add compression tests for subfiling (#3769)
* Fix typo in comment (#3775)
* Fixed a file handle leak in the core VFD (#3779)
When opening a file with the core VFD and a file image, if the file
already exists, the file check would leak the POSIX file handle.
Fixes GitHub issue #635
* Fix a format string warning in the C++ examples (#3776)
* Cancel running GitHub workflows on push to same PR (#3772)
* Cancel running GitHub workflows on push to same PR
* Remove github.sha from workflow concurrency groups
* Print some messages in parallel tests on MPI rank 0 only (#3785)
Avoids overly verbose output from all processes emitting progress, etc. info.
* Avoid attempted use of NULL pointer in parallel compression code (#3786)
The parallel compression test code tests for the case where all MPI ranks have no selection in a dataset when writing to it. Add an early exit to the code to avoid attempting to use a NULL pointer due to there being no work to do.
* Don't install h5tools_test_utils test program on system (#3793)
* Add Doxygen to H5FDsplitter.h (#3794)
* H5FD_CURR_SPLITTER_VFD_CONFIG_VERSION
* H5FD_SPLITTER_PATH_MAX
* H5FD_SPLITTER_MAGIC
* H5FD_splitter_vfd_config_t
* H5Pset_fapl_splitter()
* H5Pget_fapl_splitter()
* Update Doxygen initializers & identifiers in VFDs (#3795)
* Add Doxygen for all H5FD_<VFD> initializers
* Add Doxygen for all H5FD_<VFD>_VALUE values
* Mark H5FD_<vfd>_init() calls private in Doxygen
* Fix memory corruption in 'MPI I/O FAPL preserve' test (#3806)
* Fix usage of h5_clean_files in t_pflush2.c (#3807)
* Fix parallel driver check in h5_fixname_real (#3808)
* Fix a couple usages of MPI_Info_get (#3809)
* Remove H5system.c warning on Windows oneAPI. (#3812)
* Add processing of NVHPC flags in linux-gnulibc1 file (#3804)
* Disable testing as tests are failing the same as in CMake
* Use the current toolchain for examples as default (#3810)
* Fix misc. warnings from GCC when compiling with -fsanitize=undefined (#3787)
* Set NVHPC maximum optimization level to -O1 for now (#3800)
* Set NVHPC maximum optimization level to -O1 for now
Compiling HDF5 with NVHPC 23.5 - 23.9 results in test failures in
4 different test files that need to be resolved. Since those tests
pass with an optimization level of -O1 (and -O0) and it is currently
unclear whether the test failures are due to issues in HDF5 or issues
in the 'nvc' compiler, set the maximum optimization level for NVHPC
to -O1 until the test failures are resolved.
* Disable nvhpc Java testing in CMake and amend known issues
* Re-enable testing of Autotools nvhpc
* Update some doxygen links to local refs (#3814)
* Rework MPI Info FAPL preserve PR to use VFD 'ctl' operations (#3782)
* Removed the use of C wrappers from H5P APIs. (#3824)
* fix seg fault on frontier/cray
* fix seg fault on frontier/cray
* fix seg fault on frontier/cray
* removed the use of h5pclose_c
* removed the use of h5pclose_c
* Fortran Wrappers H5VLnative_addr_to_token_f and H5VLnative_token_to_address_f (#3801)
* Added H5VLnative_addr_to_token_f and H5VLnative_token_to_address_f
* Added H5VLnative_addr_to_token_f and H5VLnative_token_to_address_f tests
* Create test for H5Pget_dxpl_mpio (#3825)
* Create test and add to testphdf5
* Renamed h5fuse.sh to h5fuse (#3834)
* provide an alternative to mapfile for older bash
* Disable FP exceptions in H5T init code (#3837)
The H5T floating-point datatype initialization code can raise exceptions when handling signaling NaNs. This change disables FE_INVALID exceptions during initialization.
Also removes the -ieee=full change for NAG Fortran as that shouldn't be necessary anymore.
Fixes #3831
* Add intel oneapi windows build to CI CMake (#3836)
* Remove printf format warning on Windows oneAPI. (#3838)
* Correct ENV variables (#3841)
* Remove Autotools sed hack (#3848)
configure.ac contains a sed line that cleans up incorrect library
flags which was added to paper over some bugs in earlier versions
of the Autotools. These issues are not a problem with the current
versions of the Autootols.
The sed line causes problems on MacOS, so it has been removed.
Fixes #3843
* Make filter unregister callbacks safe for VOL connectors (#3629)
* Make filter callbacks use top-level API functions
When using VOL connectors, H5I_iterate may not provide
valid object pointers to its callback. This change keeps
existing functionality in H5Zunregister() without using
potentially unsafe pointers.
* Filter callbacks use internal API
* Skip MPI work on non-native VOL
* Add extra space in comments for consistency (#3852)
* Add extra space in comments for consistency
* uncomment tfloatsattrs test
* Update Actions badges to link to relevant workflow (#3850)
* Add CMake long double cross-compile defaults (#3683)
HDF5 performs a couple of checks at build time to see if long double
values can be converted correctly (IBM's Power architecture uses a
special format for long doubles). These checks were performed using
TRY_RUN, which is a problem when cross-compiling.
These checks now use default values appropriate for most non-Power
systems when cross-compiling. The cache values can be pre-set if
necessary, which will preempt both the TRY_RUN and the default.
Affected values:
H5_LDOUBLE_TO_LONG_SPECIAL (default no)
H5_LONG_TO_LDOUBLE_SPECIAL (default no)
H5_LDOUBLE_TO_LLONG_ACCURATE (default yes)
H5_LLONG_TO_LDOUBLE_CORRECT (default yes)
H5_DISABLE_SOME_LDOUBLE_CONV (default no)
Fixes GitHub #3585
* Updates for building and testing VOL connectors
* Fix issue with HDF5_VOL_ALLOW_EXTERNAL CMake variable
* Initialize parallel testing with MPI_THREAD_MULTIPLE when testing API
* Add CMake variable to allow specifying a VOL connector's package name
* Remove call to MPI_Init in serial API tests
While previously necessary, it now interferes with VOL connectors that
may need to be initialized with MPI_THREAD_MULTIPLE
* Fixes for CI and presets (#3853)
* Change dest for doxygen (#3856)
* Implement selection vector I/O with collective chunk filling (#3826)
* Changes for ECP-344: Implement selection vector I/O with collective chunk filling.
Also fix a bug in H5FD__mpio_write_vector() to account for fixed size optimization
when computing max address.
* Fixes based on PR review comments:
For H5Dchunk.c: fix H5MM_xfree()
For H5FDmpio.c:
1) Revert the fix to H5FD__mpio_write_vector()
2) Apply the patch from Neil on the proper length of s_sizes reported by H5FD__mpio_vector_build_types()
* Put back the logic of dividing up the work among all the mpi ranks similar to the
original H5D__chunk_collective_fill() routine.
* Add a test to verify the fix for the illegal reference problem in H5FD__mpio_write_vector().
* Do not publish compression headers or docs (#3865)
* Fix typo: look -> loop (#3866)
* Moved the README to markdown and expanded its overview of the files, file generation, and other Fortran wrapper development practices as mentioned in the HDF5 architectural document. I added a new figure and included the SVG file and the original xfig file it was generated from. (#3862)
* Add HDF5_DISABLE_TESTS_REGEX option to skip tests (#3859)
* Fix typo in error message for `MPI_Type_dup`. (#3867)
* Complete the `if command line option` sentence. (#3868)
* Fix h5dump segmentation fault when --vfd-value option is used (#3873)
* Updated URL in funding.yml (#3882)
Using new shortened URL, might look better.
* Remove unused variable from unmerged changes
* Update src/H5Tinit_float.c
Diffstat (limited to 'src')
| -rw-r--r-- | src/H5Dchunk.c | 264 | ||||
| -rw-r--r-- | src/H5Dmpio.c | 7 | ||||
| -rw-r--r-- | src/H5FDcore.c | 4 | ||||
| -rw-r--r-- | src/H5FDcore.h | 12 | ||||
| -rw-r--r-- | src/H5FDdirect.h | 33 | ||||
| -rw-r--r-- | src/H5FDfamily.c | 3 | ||||
| -rw-r--r-- | src/H5FDfamily.h | 11 | ||||
| -rw-r--r-- | src/H5FDhdfs.h | 21 | ||||
| -rw-r--r-- | src/H5FDlog.c | 2 | ||||
| -rw-r--r-- | src/H5FDlog.h | 11 | ||||
| -rw-r--r-- | src/H5FDmirror.h | 11 | ||||
| -rw-r--r-- | src/H5FDmpi.c | 48 | ||||
| -rw-r--r-- | src/H5FDmpio.c | 39 | ||||
| -rw-r--r-- | src/H5FDmpio.h | 25 | ||||
| -rw-r--r-- | src/H5FDmulti.h | 8 | ||||
| -rw-r--r-- | src/H5FDonion.h | 17 | ||||
| -rw-r--r-- | src/H5FDprivate.h | 1 | ||||
| -rw-r--r-- | src/H5FDpublic.h | 1 | ||||
| -rw-r--r-- | src/H5FDros3.h | 20 | ||||
| -rw-r--r-- | src/H5FDsec2.c | 2 | ||||
| -rw-r--r-- | src/H5FDsec2.h | 13 | ||||
| -rw-r--r-- | src/H5FDsplitter.h | 121 | ||||
| -rw-r--r-- | src/H5FDstdio.h | 14 | ||||
| -rw-r--r-- | src/H5FDsubfiling/H5FDsubfiling.c | 6 | ||||
| -rw-r--r-- | src/H5FDwindows.h | 7 | ||||
| -rw-r--r-- | src/H5Fint.c | 8 | ||||
| -rw-r--r-- | src/H5Fmpi.c | 34 | ||||
| -rw-r--r-- | src/H5Fprivate.h | 1 | ||||
| -rw-r--r-- | src/H5Smpio.c | 2 | ||||
| -rw-r--r-- | src/H5T.c | 6 | ||||
| -rw-r--r-- | src/H5Tinit_float.c | 54 | ||||
| -rw-r--r-- | src/H5Tvlen.c | 2 | ||||
| -rw-r--r-- | src/H5Z.c | 142 | ||||
| -rw-r--r-- | src/H5mpi.c | 4 | ||||
| -rw-r--r-- | src/H5private.h | 1 | ||||
| -rw-r--r-- | src/H5system.c | 3 |
36 files changed, 590 insertions, 368 deletions
diff --git a/src/H5Dchunk.c b/src/H5Dchunk.c index 41d774d..c8bad76 100644 --- a/src/H5Dchunk.c +++ b/src/H5Dchunk.c @@ -5536,11 +5536,9 @@ done: /*------------------------------------------------------------------------- * Function: H5D__chunk_collective_fill * - * Purpose: Use MPIO collective write to fill the chunks (if number of - * chunks to fill is greater than the number of MPI procs; - * otherwise use independent I/O). + * Purpose: Use MPIO selection vector I/O for writing fill chunks * - * Return: Non-negative on success/Negative on failure + * Return: Non-negative on success/Negative on failure * *------------------------------------------------------------------------- */ @@ -5554,19 +5552,24 @@ H5D__chunk_collective_fill(const H5D_t *dset, H5D_chunk_coll_fill_info_t *chunk_ int mpi_code; /* MPI return code */ size_t num_blocks; /* Number of blocks between processes. */ size_t leftover_blocks; /* Number of leftover blocks to handle */ - int blocks, leftover; /* converted to int for MPI */ - MPI_Aint *chunk_disp_array = NULL; - MPI_Aint *block_disps = NULL; - int *block_lens = NULL; - MPI_Datatype mem_type = MPI_BYTE, file_type = MPI_BYTE; - H5FD_mpio_xfer_t prev_xfer_mode; /* Previous data xfer mode */ - bool have_xfer_mode = false; /* Whether the previous xffer mode has been retrieved */ - bool need_sort = false; - size_t i; /* Local index variable */ + int blocks; /* converted to int for MPI */ + int leftover; /* converted to int for MPI */ + H5FD_mpio_xfer_t prev_xfer_mode; /* Previous data xfer mode */ + bool have_xfer_mode = false; /* Whether the previous xffer mode has been retrieved */ + size_t i; /* Local index variable */ + haddr_t *io_addrs = NULL; + size_t *io_sizes = NULL; + const void **io_wbufs = NULL; + H5FD_mem_t io_types[2]; + bool all_same_block_len = true; + bool need_sort = false; + size_t io_2sizes[2]; herr_t ret_value = SUCCEED; /* Return value */ FUNC_ENTER_PACKAGE + assert(chunk_fill_info->num_chunks != 0); + /* * If a separate fill buffer is provided for partial chunks, ensure * that the "don't filter partial edge chunks" flag is set. @@ -5589,6 +5592,7 @@ H5D__chunk_collective_fill(const H5D_t *dset, H5D_chunk_coll_fill_info_t *chunk_ /* Distribute evenly the number of blocks between processes. */ if (mpi_size == 0) HGOTO_ERROR(H5E_DATASET, H5E_BADVALUE, FAIL, "Resulted in division by zero"); + num_blocks = (size_t)(chunk_fill_info->num_chunks / (size_t)mpi_size); /* value should be the same on all procs */ @@ -5602,157 +5606,97 @@ H5D__chunk_collective_fill(const H5D_t *dset, H5D_chunk_coll_fill_info_t *chunk_ H5_CHECKED_ASSIGN(leftover, int, leftover_blocks, size_t); /* Check if we have any chunks to write on this rank */ - if (num_blocks > 0 || (leftover && leftover > mpi_rank)) { - MPI_Aint partial_fill_buf_disp = 0; - bool all_same_block_len = true; - - /* Allocate buffers */ - if (NULL == (chunk_disp_array = (MPI_Aint *)H5MM_malloc((size_t)(blocks + 1) * sizeof(MPI_Aint)))) - HGOTO_ERROR(H5E_DATASET, H5E_CANTALLOC, FAIL, "couldn't allocate chunk file displacement buffer"); - - if (partial_chunk_fill_buf) { - MPI_Aint fill_buf_addr; - MPI_Aint partial_fill_buf_addr; - - /* Calculate the displacement between the fill buffer and partial chunk fill buffer */ - if (MPI_SUCCESS != (mpi_code = MPI_Get_address(fill_buf, &fill_buf_addr))) - HMPI_GOTO_ERROR(FAIL, "MPI_Get_address failed", mpi_code) - if (MPI_SUCCESS != (mpi_code = MPI_Get_address(partial_chunk_fill_buf, &partial_fill_buf_addr))) - HMPI_GOTO_ERROR(FAIL, "MPI_Get_address failed", mpi_code) - -#if H5_CHECK_MPI_VERSION(3, 1) - partial_fill_buf_disp = MPI_Aint_diff(partial_fill_buf_addr, fill_buf_addr); -#else - partial_fill_buf_disp = partial_fill_buf_addr - fill_buf_addr; -#endif + if (num_blocks > 0 || leftover > mpi_rank) { - /* - * Allocate all-zero block displacements array. If a block's displacement - * is left as zero, that block will be written to from the regular fill - * buffer. If a block represents an unfiltered partial edge chunk, its - * displacement will be set so that the block is written to from the - * unfiltered fill buffer. - */ - if (NULL == (block_disps = (MPI_Aint *)H5MM_calloc((size_t)(blocks + 1) * sizeof(MPI_Aint)))) - HGOTO_ERROR(H5E_DATASET, H5E_CANTALLOC, FAIL, "couldn't allocate block displacements buffer"); - } + if (NULL == (io_addrs = H5MM_malloc((size_t)(blocks + 1) * sizeof(*io_addrs)))) + HGOTO_ERROR(H5E_RESOURCE, H5E_CANTALLOC, FAIL, + "couldn't allocate space for I/O addresses vector"); - /* - * Perform initial scan of chunk info list to: - * - make sure that chunk addresses are monotonically non-decreasing - * - check if all blocks have the same length - */ - for (i = 1; i < chunk_fill_info->num_chunks; i++) { - if (chunk_fill_info->chunk_info[i].addr < chunk_fill_info->chunk_info[i - 1].addr) - need_sort = true; - - if (chunk_fill_info->chunk_info[i].chunk_size != chunk_fill_info->chunk_info[i - 1].chunk_size) - all_same_block_len = false; - } + if (NULL == (io_wbufs = H5MM_malloc((size_t)(blocks + 1) * sizeof(*io_wbufs)))) + HGOTO_ERROR(H5E_RESOURCE, H5E_CANTALLOC, FAIL, "couldn't allocate space for I/O buffers vector"); + } - if (need_sort) - qsort(chunk_fill_info->chunk_info, chunk_fill_info->num_chunks, - sizeof(struct chunk_coll_fill_info), H5D__chunk_cmp_coll_fill_info); + /* + * Perform initial scan of chunk info list to: + * - make sure that chunk addresses are monotonically non-decreasing + * - check if all blocks have the same length + */ + for (i = 1; i < chunk_fill_info->num_chunks; i++) { + if (chunk_fill_info->chunk_info[i].addr < chunk_fill_info->chunk_info[i - 1].addr) + need_sort = true; - /* Allocate buffer for block lengths if necessary */ - if (!all_same_block_len) - if (NULL == (block_lens = (int *)H5MM_malloc((size_t)(blocks + 1) * sizeof(int)))) - HGOTO_ERROR(H5E_DATASET, H5E_CANTALLOC, FAIL, "couldn't allocate chunk lengths buffer"); + if (chunk_fill_info->chunk_info[i].chunk_size != chunk_fill_info->chunk_info[i - 1].chunk_size) + all_same_block_len = false; + } - for (i = 0; i < (size_t)blocks; i++) { - size_t idx = i + (size_t)(mpi_rank * blocks); + /* + * Note that we sort all of the chunks here, and not just a subset + * corresponding to this rank. We do this since we have found MPI I/O to work + * better when each rank writes blocks that are contiguous in the file, + * and by sorting the full list we maximize the chance of that happening. + */ + if (need_sort) + qsort(chunk_fill_info->chunk_info, chunk_fill_info->num_chunks, sizeof(struct chunk_coll_fill_info), + H5D__chunk_cmp_coll_fill_info); - /* store the chunk address as an MPI_Aint */ - chunk_disp_array[i] = (MPI_Aint)(chunk_fill_info->chunk_info[idx].addr); + /* + * If all the chunks have the same length, use the compressed feature + * to store the size. + * Otherwise, allocate the array of sizes for storing chunk sizes. + */ + if (all_same_block_len) { + io_2sizes[0] = chunk_fill_info->chunk_info[0].chunk_size; + io_2sizes[1] = 0; + } + else { + if (NULL == (io_sizes = H5MM_malloc((size_t)(blocks + 1) * sizeof(*io_sizes)))) + HGOTO_ERROR(H5E_RESOURCE, H5E_CANTALLOC, FAIL, "couldn't allocate space for I/O sizes vector"); + } - if (!all_same_block_len) - H5_CHECKED_ASSIGN(block_lens[i], int, chunk_fill_info->chunk_info[idx].chunk_size, size_t); + /* + * Since the type of all chunks is raw data, use the compressed feature + * to store the chunk type. + */ + io_types[0] = H5FD_MEM_DRAW; + io_types[1] = H5FD_MEM_NOLIST; - if (chunk_fill_info->chunk_info[idx].unfiltered_partial_chunk) { - assert(partial_chunk_fill_buf); - block_disps[i] = partial_fill_buf_disp; - } - } /* end for */ + /* + * For the chunks corresponding to this rank, fill in the + * address, size and buf pointer for each chunk. + */ + for (i = 0; i < (size_t)blocks; i++) { + size_t idx = i + (size_t)(mpi_rank * blocks); - /* Calculate if there are any leftover blocks after evenly - * distributing. If there are, then round-robin the distribution - * to processes 0 -> leftover. - */ - if (leftover && leftover > mpi_rank) { - chunk_disp_array[blocks] = - (MPI_Aint)chunk_fill_info->chunk_info[(blocks * mpi_size) + mpi_rank].addr; - - if (!all_same_block_len) - H5_CHECKED_ASSIGN(block_lens[blocks], int, - chunk_fill_info->chunk_info[(blocks * mpi_size) + mpi_rank].chunk_size, - size_t); - - if (chunk_fill_info->chunk_info[(blocks * mpi_size) + mpi_rank].unfiltered_partial_chunk) { - assert(partial_chunk_fill_buf); - block_disps[blocks] = partial_fill_buf_disp; - } + io_addrs[i] = chunk_fill_info->chunk_info[idx].addr; - blocks++; - } + if (!all_same_block_len) + io_sizes[i] = chunk_fill_info->chunk_info[idx].chunk_size; - /* Create file and memory types for the write operation */ - if (all_same_block_len) { - int block_len; + if (chunk_fill_info->chunk_info[idx].unfiltered_partial_chunk) + io_wbufs[i] = partial_chunk_fill_buf; + else + io_wbufs[i] = fill_buf; + } - H5_CHECKED_ASSIGN(block_len, int, chunk_fill_info->chunk_info[0].chunk_size, size_t); + /* + * For the leftover chunk corresponding to this rank, fill in the + * address, size and buf pointer for the chunk. + */ + if (leftover > mpi_rank) { + io_addrs[blocks] = chunk_fill_info->chunk_info[(blocks * mpi_size) + mpi_rank].addr; - mpi_code = - MPI_Type_create_hindexed_block(blocks, block_len, chunk_disp_array, MPI_BYTE, &file_type); - if (mpi_code != MPI_SUCCESS) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_create_hindexed_block failed", mpi_code) + if (!all_same_block_len) + io_sizes[blocks] = chunk_fill_info->chunk_info[(blocks * mpi_size) + mpi_rank].chunk_size; - if (partial_chunk_fill_buf) { - /* - * If filters are disabled for partial edge chunks, those chunks could - * potentially have the same block length as the other chunks, but still - * need to be written to using the unfiltered fill buffer. Use an hindexed - * block type rather than an hvector. - */ - mpi_code = - MPI_Type_create_hindexed_block(blocks, block_len, block_disps, MPI_BYTE, &mem_type); - if (mpi_code != MPI_SUCCESS) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_create_hindexed_block failed", mpi_code) - } - else { - mpi_code = MPI_Type_create_hvector(blocks, block_len, 0, MPI_BYTE, &mem_type); - if (mpi_code != MPI_SUCCESS) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_create_hvector failed", mpi_code) - } - } - else { - /* - * Currently, different block lengths implies that there are partial - * edge chunks and the "don't filter partial edge chunks" flag is set. - */ + if (chunk_fill_info->chunk_info[(blocks * mpi_size) + mpi_rank].unfiltered_partial_chunk) { assert(partial_chunk_fill_buf); - assert(block_lens); - assert(block_disps); - - mpi_code = MPI_Type_create_hindexed(blocks, block_lens, chunk_disp_array, MPI_BYTE, &file_type); - if (mpi_code != MPI_SUCCESS) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_create_hindexed failed", mpi_code) - - mpi_code = MPI_Type_create_hindexed(blocks, block_lens, block_disps, MPI_BYTE, &mem_type); - if (mpi_code != MPI_SUCCESS) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_create_hindexed failed", mpi_code) + io_wbufs[blocks] = partial_chunk_fill_buf; } + else + io_wbufs[blocks] = fill_buf; - if (MPI_SUCCESS != (mpi_code = MPI_Type_commit(&file_type))) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_commit failed", mpi_code) - if (MPI_SUCCESS != (mpi_code = MPI_Type_commit(&mem_type))) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_commit failed", mpi_code) - } /* end if */ - - /* Set MPI-IO VFD properties */ - - /* Set MPI datatypes for operation */ - if (H5CX_set_mpi_coll_datatypes(mem_type, file_type) < 0) - HGOTO_ERROR(H5E_DATASET, H5E_CANTSET, FAIL, "can't set MPI-I/O properties"); + blocks++; + } /* Get current transfer mode */ if (H5CX_get_io_xfer_mode(&prev_xfer_mode) < 0) @@ -5763,31 +5707,24 @@ H5D__chunk_collective_fill(const H5D_t *dset, H5D_chunk_coll_fill_info_t *chunk_ if (H5CX_set_io_xfer_mode(H5FD_MPIO_COLLECTIVE) < 0) HGOTO_ERROR(H5E_DATASET, H5E_CANTSET, FAIL, "can't set transfer mode"); - /* Low-level write (collective) */ - if (H5F_shared_block_write(H5F_SHARED(dset->oloc.file), H5FD_MEM_DRAW, (haddr_t)0, - (blocks) ? (size_t)1 : (size_t)0, fill_buf) < 0) - HGOTO_ERROR(H5E_IO, H5E_WRITEERROR, FAIL, "unable to write raw data to file"); - /* Barrier so processes don't race ahead */ if (MPI_SUCCESS != (mpi_code = MPI_Barrier(mpi_comm))) HMPI_GOTO_ERROR(FAIL, "MPI_Barrier failed", mpi_code) + /* Perform the selection vector I/O for the chunks */ + if (H5F_shared_vector_write(H5F_SHARED(dset->oloc.file), (uint32_t)blocks, io_types, io_addrs, + all_same_block_len ? io_2sizes : io_sizes, io_wbufs) < 0) + HGOTO_ERROR(H5E_DATASET, H5E_WRITEERROR, FAIL, "vector write call failed"); + done: if (have_xfer_mode) - /* Set transfer mode */ + /* Restore transfer mode */ if (H5CX_set_io_xfer_mode(prev_xfer_mode) < 0) HDONE_ERROR(H5E_DATASET, H5E_CANTSET, FAIL, "can't set transfer mode"); - /* free things */ - if (MPI_BYTE != file_type) - if (MPI_SUCCESS != (mpi_code = MPI_Type_free(&file_type))) - HMPI_DONE_ERROR(FAIL, "MPI_Type_free failed", mpi_code) - if (MPI_BYTE != mem_type) - if (MPI_SUCCESS != (mpi_code = MPI_Type_free(&mem_type))) - HMPI_DONE_ERROR(FAIL, "MPI_Type_free failed", mpi_code) - H5MM_xfree(chunk_disp_array); - H5MM_xfree(block_disps); - H5MM_xfree(block_lens); + H5MM_xfree(io_addrs); + H5MM_xfree(io_wbufs); + H5MM_xfree(io_sizes); FUNC_LEAVE_NOAPI(ret_value) } /* end H5D__chunk_collective_fill() */ @@ -5805,6 +5742,7 @@ H5D__chunk_cmp_coll_fill_info(const void *_entry1, const void *_entry2) FUNC_LEAVE_NOAPI(H5_addr_cmp(entry1->addr, entry2->addr)) } /* end H5D__chunk_cmp_coll_fill_info() */ + #endif /* H5_HAVE_PARALLEL */ /*------------------------------------------------------------------------- diff --git a/src/H5Dmpio.c b/src/H5Dmpio.c index 16243fa..b6976e6 100644 --- a/src/H5Dmpio.c +++ b/src/H5Dmpio.c @@ -911,7 +911,8 @@ H5D__mpio_get_no_coll_cause_strings(char *local_cause, size_t local_cause_len, c case H5D_MPIO_COLLECTIVE: case H5D_MPIO_NO_COLLECTIVE_MAX_CAUSE: default: - assert(0 && "invalid no collective cause reason"); + cause_str = "invalid or unknown no collective cause reason"; + assert(0 && "invalid or unknown no collective cause reason"); break; } @@ -3782,6 +3783,10 @@ H5D__mpio_redistribute_shared_chunks_int(H5D_filtered_collective_io_info_t *chun counts_disps_array = H5MM_xfree(counts_disps_array); } + /* No useful work to do - exit */ + if (coll_chunk_list_num_entries == 0) + HGOTO_DONE(SUCCEED); + /* * Phase 2 - Involved ranks now redistribute any shared chunks to new * owners as necessary. diff --git a/src/H5FDcore.c b/src/H5FDcore.c index 08b714d..1aa8d4b 100644 --- a/src/H5FDcore.c +++ b/src/H5FDcore.c @@ -754,8 +754,10 @@ H5FD__core_open(const char *name, unsigned flags, hid_t fapl_id, haddr_t maxaddr ((file_image_info.buffer == NULL) && (file_image_info.size == 0))); memset(&sb, 0, sizeof(sb)); if ((file_image_info.buffer != NULL) && !(H5F_ACC_CREAT & flags)) { - if (HDopen(name, o_flags, H5_POSIX_CREATE_MODE_RW) >= 0) + if ((fd = HDopen(name, o_flags, H5_POSIX_CREATE_MODE_RW)) >= 0) { + HDclose(fd); HGOTO_ERROR(H5E_FILE, H5E_FILEEXISTS, NULL, "file already exists"); + } /* If backing store is requested, create and stat the file * Note: We are forcing the O_CREAT flag here, even though this is diff --git a/src/H5FDcore.h b/src/H5FDcore.h index 235d6fc..cd45c8d 100644 --- a/src/H5FDcore.h +++ b/src/H5FDcore.h @@ -11,17 +11,25 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the core driver. + * Purpose: The public header file for the core virtual file driver (VFD) */ #ifndef H5FDcore_H #define H5FDcore_H -#define H5FD_CORE (H5FDperform_init(H5FD_core_init)) +/** Initializer for the core VFD */ +#define H5FD_CORE (H5FDperform_init(H5FD_core_init)) + +/** Identifier for the core VFD */ #define H5FD_CORE_VALUE H5_VFD_CORE #ifdef __cplusplus extern "C" { #endif + +/** @private + * + * \brief Private initializer for the core VFD + */ H5_DLL hid_t H5FD_core_init(void); /** diff --git a/src/H5FDdirect.h b/src/H5FDdirect.h index e47ac37..1e60bb0 100644 --- a/src/H5FDdirect.h +++ b/src/H5FDdirect.h @@ -11,30 +11,47 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the direct driver. + * Purpose: The public header file for the direct virtual file driver (VFD) */ #ifndef H5FDdirect_H #define H5FDdirect_H #ifdef H5_HAVE_DIRECT -#define H5FD_DIRECT (H5FDperform_init(H5FD_direct_init)) + +/** Initializer for the direct VFD */ +#define H5FD_DIRECT (H5FDperform_init(H5FD_direct_init)) + +/** Identifier for the direct VFD */ #define H5FD_DIRECT_VALUE H5_VFD_DIRECT + #else + +/** Initializer for the direct VFD (disabled) */ #define H5FD_DIRECT (H5I_INVALID_HID) + +/** Identifier for the direct VFD (disabled) */ #define H5FD_DIRECT_VALUE H5_VFD_INVALID + #endif /* H5_HAVE_DIRECT */ +/** Default value for memory boundary */ +#define MBOUNDARY_DEF 4096 + +/** Default value for file block size */ +#define FBSIZE_DEF 4096 + +/** Default value for maximum copy buffer size */ +#define CBSIZE_DEF (16 * 1024 * 1024) + #ifdef H5_HAVE_DIRECT #ifdef __cplusplus extern "C" { #endif -/* Default values for memory boundary, file block size, and maximal copy buffer size. - * Application can set these values through the function H5Pset_fapl_direct. */ -#define MBOUNDARY_DEF 4096 -#define FBSIZE_DEF 4096 -#define CBSIZE_DEF 16 * 1024 * 1024 - +/** @private + * + * \brief Private initializer for the direct VFD + */ H5_DLL hid_t H5FD_direct_init(void); /** diff --git a/src/H5FDfamily.c b/src/H5FDfamily.c index 94805a2..3f43ae9 100644 --- a/src/H5FDfamily.c +++ b/src/H5FDfamily.c @@ -1427,7 +1427,8 @@ H5FD__family_delete(const char *filename, hid_t fapl_id) FUNC_ENTER_PACKAGE - assert(filename); + if (!filename) + HGOTO_ERROR(H5E_VFL, H5E_BADVALUE, FAIL, "invalid filename pointer"); /* Get the driver info (for the member fapl) * The family_open call accepts H5P_DEFAULT, so we'll accept that here, too. diff --git a/src/H5FDfamily.h b/src/H5FDfamily.h index 76020f0..32e885c 100644 --- a/src/H5FDfamily.h +++ b/src/H5FDfamily.h @@ -11,18 +11,25 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the family driver. + * Purpose: The public header file for the family virtual file driver (VFD) */ #ifndef H5FDfamily_H #define H5FDfamily_H -#define H5FD_FAMILY (H5FDperform_init(H5FD_family_init)) +/** Initializer for the family VFD */ +#define H5FD_FAMILY (H5FDperform_init(H5FD_family_init)) + +/** Identifier for the family VFD */ #define H5FD_FAMILY_VALUE H5_VFD_FAMILY #ifdef __cplusplus extern "C" { #endif +/** @private + * + * \brief Private initializer for the family VFD + */ H5_DLL hid_t H5FD_family_init(void); /** diff --git a/src/H5FDhdfs.h b/src/H5FDhdfs.h index c8c2c37..e5f7173 100644 --- a/src/H5FDhdfs.h +++ b/src/H5FDhdfs.h @@ -11,18 +11,29 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the hdfs driver. + * Purpose: The public header file for the Hadoop Distributed File System + * (hdfs) virtual file driver (VFD) */ #ifndef H5FDhdfs_H #define H5FDhdfs_H #ifdef H5_HAVE_LIBHDFS -#define H5FD_HDFS (H5FDperform_init(H5FD_hdfs_init)) + +/** Initializer for the hdfs VFD */ +#define H5FD_HDFS (H5FDperform_init(H5FD_hdfs_init)) + +/** Identifier for the hdfs VFD */ #define H5FD_HDFS_VALUE H5_VFD_HDFS -#else /* H5_HAVE_LIBHDFS */ + +#else + +/** Initializer for the hdfs VFD (disabled) */ #define H5FD_HDFS (H5I_INVALID_HID) + +/** Identifier for the hdfs VFD (disabled) */ #define H5FD_HDFS_VALUE H5_VFD_INVALID + #endif /* H5_HAVE_LIBHDFS */ #ifdef H5_HAVE_LIBHDFS @@ -104,6 +115,10 @@ typedef struct H5FD_hdfs_fapl_t { int32_t stream_buffer_size; } H5FD_hdfs_fapl_t; +/** @private + * + * \brief Private initializer for the hdfs VFD + */ H5_DLL hid_t H5FD_hdfs_init(void); /** diff --git a/src/H5FDlog.c b/src/H5FDlog.c index 8d43dc8..e35a6a6 100644 --- a/src/H5FDlog.c +++ b/src/H5FDlog.c @@ -545,7 +545,7 @@ H5FD__log_open(const char *name, unsigned flags, hid_t fapl_id, haddr_t maxaddr) #endif /* H5_HAVE_WIN32_API */ /* Retain a copy of the name used to open the file, for possible error reporting */ - strncpy(file->filename, name, sizeof(file->filename)); + strncpy(file->filename, name, sizeof(file->filename) - 1); file->filename[sizeof(file->filename) - 1] = '\0'; /* Get the flags for logging */ diff --git a/src/H5FDlog.h b/src/H5FDlog.h index ae4e2d0..b4af205 100644 --- a/src/H5FDlog.h +++ b/src/H5FDlog.h @@ -11,12 +11,15 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the log driver. + * Purpose: The public header file for the log virtual file driver (VFD) */ #ifndef H5FDlog_H #define H5FDlog_H -#define H5FD_LOG (H5FDperform_init(H5FD_log_init)) +/** Initializer for the log VFD */ +#define H5FD_LOG (H5FDperform_init(H5FD_log_init)) + +/** Identifier for the log VFD */ #define H5FD_LOG_VALUE H5_VFD_LOG /* Flags for H5Pset_fapl_log() */ @@ -62,6 +65,10 @@ extern "C" { #endif +/** @private + * + * \brief Private initializer for the log VFD + */ H5_DLL hid_t H5FD_log_init(void); /** diff --git a/src/H5FDmirror.h b/src/H5FDmirror.h index b196b2b..6c98e1a 100644 --- a/src/H5FDmirror.h +++ b/src/H5FDmirror.h @@ -11,7 +11,7 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: Public, shared definitions for Mirror VFD & remote Writer. + * Purpose: The public header file for the mirror virtual file driver (VFD) */ #ifndef H5FDmirror_H @@ -19,7 +19,10 @@ #ifdef H5_HAVE_MIRROR_VFD -#define H5FD_MIRROR (H5FDperform_init(H5FD_mirror_init)) +/** Initializer for the mirror VFD */ +#define H5FD_MIRROR (H5FDperform_init(H5FD_mirror_init)) + +/** Identifier for the mirror VFD */ #define H5FD_MIRROR_VALUE H5_VFD_MIRROR #ifdef __cplusplus @@ -62,6 +65,10 @@ typedef struct H5FD_mirror_fapl_t { char remote_ip[H5FD_MIRROR_MAX_IP_LEN + 1]; } H5FD_mirror_fapl_t; +/** @private + * + * \brief Private initializer for the mirror VFD + */ H5_DLL hid_t H5FD_mirror_init(void); /** diff --git a/src/H5FDmpi.c b/src/H5FDmpi.c index 127740e..f247c34 100644 --- a/src/H5FDmpi.c +++ b/src/H5FDmpi.c @@ -104,13 +104,12 @@ done: } /* end H5FD_mpi_get_size() */ /*------------------------------------------------------------------------- - * Function: H5FD_mpi_get_comm + * Function: H5FD_mpi_get_comm * - * Purpose: Retrieves the file's communicator + * Purpose: Retrieves the file's MPI_Comm communicator object * - * Return: Success: The communicator (non-negative) - * - * Failure: Negative + * Return: Success: The communicator object + * Failure: MPI_COMM_NULL * *------------------------------------------------------------------------- */ @@ -144,6 +143,45 @@ done: } /* end H5FD_mpi_get_comm() */ /*------------------------------------------------------------------------- + * Function: H5FD_mpi_get_info + * + * Purpose: Retrieves the file's MPI_Info info object + * + * Return: Success: The info object + * Failure: MPI_INFO_NULL + * + *------------------------------------------------------------------------- + */ +MPI_Info +H5FD_mpi_get_info(H5FD_t *file) +{ + const H5FD_class_t *cls; + uint64_t flags = H5FD_CTL_FAIL_IF_UNKNOWN_FLAG | H5FD_CTL_ROUTE_TO_TERMINAL_VFD_FLAG; + MPI_Info info = MPI_INFO_NULL; + void *info_ptr = (void *)(&info); + MPI_Info ret_value; + + FUNC_ENTER_NOAPI(MPI_INFO_NULL) + + assert(file); + cls = (const H5FD_class_t *)(file->cls); + assert(cls); + assert(cls->ctl); /* All MPI drivers must implement this */ + + /* Dispatch to driver */ + if ((cls->ctl)(file, H5FD_CTL_GET_MPI_INFO_OPCODE, flags, NULL, &info_ptr) < 0) + HGOTO_ERROR(H5E_VFL, H5E_CANTGET, MPI_INFO_NULL, "driver get_info request failed"); + + if (info == MPI_INFO_NULL) + HGOTO_ERROR(H5E_VFL, H5E_CANTGET, MPI_INFO_NULL, "driver get_info request failed -- bad info object"); + + ret_value = info; + +done: + FUNC_LEAVE_NOAPI(ret_value) +} /* end H5FD_mpi_get_info() */ + +/*------------------------------------------------------------------------- * Function: H5FD_mpi_MPIOff_to_haddr * * Purpose: Convert an MPI_Offset value to haddr_t. diff --git a/src/H5FDmpio.c b/src/H5FDmpio.c index 7141550..8aae79e 100644 --- a/src/H5FDmpio.c +++ b/src/H5FDmpio.c @@ -106,7 +106,7 @@ static herr_t H5FD__mpio_ctl(H5FD_t *_file, uint64_t op_code, uint64_t flags, co /* Other functions */ static herr_t H5FD__mpio_vector_build_types(uint32_t count, H5FD_mem_t types[], haddr_t addrs[], size_t sizes[], H5_flexible_const_ptr_t bufs[], - haddr_t *s_addrs[], size_t *s_sizes[], + haddr_t *s_addrs[], size_t *s_sizes[], uint32_t *s_sizes_len, H5_flexible_const_ptr_t *s_bufs[], bool *vector_was_sorted, MPI_Offset *mpi_off, H5_flexible_const_ptr_t *mpi_bufs_base, int *size_i, MPI_Datatype *buf_type, bool *buf_type_created, @@ -906,7 +906,7 @@ H5FD__mpio_open(const char *name, unsigned flags, hid_t fapl_id, haddr_t H5_ATTR /* copy over each hint */ for (i = 0; i < nkeys; i++) { - char key[MPI_MAX_INFO_KEY], value[MPI_MAX_INFO_VAL]; + char key[MPI_MAX_INFO_KEY], value[MPI_MAX_INFO_VAL + 1]; int valuelen, flag; /* retrieve the nth hint */ @@ -916,7 +916,7 @@ H5FD__mpio_open(const char *name, unsigned flags, hid_t fapl_id, haddr_t H5_ATTR if (MPI_SUCCESS != (mpi_code = MPI_Info_get_valuelen(info_used, key, &valuelen, &flag))) HMPI_GOTO_ERROR(NULL, "MPI_Info_get_valuelen failed", mpi_code) /* retrieve the value of nth hint */ - if (MPI_SUCCESS != (mpi_code = MPI_Info_get(info_used, key, valuelen + 1, value, &flag))) + if (MPI_SUCCESS != (mpi_code = MPI_Info_get(info_used, key, valuelen, value, &flag))) HMPI_GOTO_ERROR(NULL, "MPI_Info_get failed", mpi_code) /* copy the hint into info */ @@ -1675,7 +1675,8 @@ done: static herr_t H5FD__mpio_vector_build_types(uint32_t count, H5FD_mem_t types[], haddr_t addrs[], size_t sizes[], H5_flexible_const_ptr_t bufs[], haddr_t *s_addrs[], size_t *s_sizes[], - H5_flexible_const_ptr_t *s_bufs[], bool *vector_was_sorted, MPI_Offset *mpi_off, + uint32_t *s_sizes_len, H5_flexible_const_ptr_t *s_bufs[], + bool *vector_was_sorted, MPI_Offset *mpi_off, H5_flexible_const_ptr_t *mpi_bufs_base, int *size_i, MPI_Datatype *buf_type, bool *buf_type_created, MPI_Datatype *file_type, bool *file_type_created, char *unused) @@ -1716,6 +1717,10 @@ H5FD__mpio_vector_build_types(uint32_t count, H5FD_mem_t types[], haddr_t addrs[ /* Get bio I/O transition point (may be lower than 2G for testing) */ bigio_count = H5_mpi_get_bigio_count(); + /* Start with s_sizes_len at count */ + if (s_sizes_len) + *s_sizes_len = count; + if (count == 1) { /* Single block. Just use a series of MPI_BYTEs for the file view. */ @@ -1808,8 +1813,13 @@ H5FD__mpio_vector_build_types(uint32_t count, H5FD_mem_t types[], haddr_t addrs[ if (!fixed_size) { if ((*s_sizes)[i] == 0) { assert(vector_was_sorted); + assert(i > 0); fixed_size = true; size = sizes[i - 1]; + + /* Return the used length of the s_sizes buffer */ + if (s_sizes_len) + *s_sizes_len = (uint32_t)i; } else { size = (*s_sizes)[i]; @@ -2098,7 +2108,7 @@ H5FD__mpio_read_vector(H5FD_t *_file, hid_t H5_ATTR_UNUSED dxpl_id, uint32_t cou if (xfer_mode == H5FD_MPIO_COLLECTIVE) { /* Build MPI types, etc. */ if (H5FD__mpio_vector_build_types(count, types, addrs, sizes, (H5_flexible_const_ptr_t *)bufs, - &s_addrs, &s_sizes, (H5_flexible_const_ptr_t **)&s_bufs, + &s_addrs, &s_sizes, NULL, (H5_flexible_const_ptr_t **)&s_bufs, &vector_was_sorted, &mpi_off, (H5_flexible_const_ptr_t *)&mpi_bufs_base, &size_i, &buf_type, &buf_type_created, &file_type, &file_type_created, &unused) < 0) @@ -2464,17 +2474,21 @@ H5FD__mpio_write_vector(H5FD_t *_file, hid_t H5_ATTR_UNUSED dxpl_id, uint32_t co HGOTO_ERROR(H5E_VFL, H5E_CANTGET, FAIL, "can't get MPI-I/O transfer mode"); if (xfer_mode == H5FD_MPIO_COLLECTIVE) { + uint32_t s_sizes_len; + /* Build MPI types, etc. */ if (H5FD__mpio_vector_build_types(count, types, addrs, sizes, (H5_flexible_const_ptr_t *)bufs, - &s_addrs, &s_sizes, (H5_flexible_const_ptr_t **)&s_bufs, - &vector_was_sorted, &mpi_off, + &s_addrs, &s_sizes, &s_sizes_len, + (H5_flexible_const_ptr_t **)&s_bufs, &vector_was_sorted, &mpi_off, (H5_flexible_const_ptr_t *)&mpi_bufs_base, &size_i, &buf_type, &buf_type_created, &file_type, &file_type_created, &unused) < 0) HGOTO_ERROR(H5E_VFL, H5E_CANTGET, FAIL, "can't build MPI datatypes for I/O"); - /* Compute max address written to */ + /* Compute max address written to. Note s_sizes is indexed according to the length of that array as + * reported by H5FD__mpio_vector_build_types(), which may be shorter if using the compressed arrays + * feature. */ if (count > 0) - max_addr = s_addrs[count - 1] + (haddr_t)(s_sizes[count - 1]); + max_addr = s_addrs[count - 1] + (haddr_t)(s_sizes[s_sizes_len - 1]); /* free sorted vectors if they exist */ if (!vector_was_sorted) { @@ -3795,6 +3809,7 @@ done: * At present, the supported op codes are: * * H5FD_CTL_GET_MPI_COMMUNICATOR_OPCODE + * H5FD_CTL_GET_MPI_INFO_OPCODE * H5FD_CTL_GET_MPI_RANK_OPCODE * H5FD_CTL_GET_MPI_SIZE_OPCODE * H5FD_CTL_GET_MPI_FILE_SYNC_OPCODE @@ -3827,6 +3842,12 @@ H5FD__mpio_ctl(H5FD_t *_file, uint64_t op_code, uint64_t flags, const void H5_AT **((MPI_Comm **)output) = file->comm; break; + case H5FD_CTL_GET_MPI_INFO_OPCODE: + assert(output); + assert(*output); + **((MPI_Info **)output) = file->info; + break; + case H5FD_CTL_GET_MPI_RANK_OPCODE: assert(output); assert(*output); diff --git a/src/H5FDmpio.h b/src/H5FDmpio.h index 60deec2..5e7ecf3 100644 --- a/src/H5FDmpio.h +++ b/src/H5FDmpio.h @@ -11,35 +11,42 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the mpio driver. + * Purpose: The public header file for the MPI-I/O (mpio) virtual file driver (VFD) */ #ifndef H5FDmpio_H #define H5FDmpio_H -/* Macros */ - #ifdef H5_HAVE_PARALLEL + +/** Initializer for the mpio VFD */ #define H5FD_MPIO (H5FDperform_init(H5FD_mpio_init)) + #else + +/** Initializer for the mpio VFD (disabled) */ #define H5FD_MPIO (H5I_INVALID_HID) -#endif /* H5_HAVE_PARALLEL */ + +#endif #ifdef H5_HAVE_PARALLEL -/*Turn on H5FDmpio_debug if H5F_DEBUG is on */ -#ifdef H5F_DEBUG -#ifndef H5FDmpio_DEBUG + +#if defined(H5F_DEBUG) && !defined(H5FDmpio_DEBUG) +/** Turn mpio VFD debugging on (requires H5F_DEBUG) */ #define H5FDmpio_DEBUG #endif -#endif /* Global var whose value comes from environment variable */ /* (Defined in H5FDmpio.c) */ H5_DLLVAR hbool_t H5FD_mpi_opt_types_g; -/* Function prototypes */ #ifdef __cplusplus extern "C" { #endif + +/** @private + * + * \brief Private initializer for the mpio VFD + */ H5_DLL hid_t H5FD_mpio_init(void); /** diff --git a/src/H5FDmulti.h b/src/H5FDmulti.h index a85f2df..d89a3e2 100644 --- a/src/H5FDmulti.h +++ b/src/H5FDmulti.h @@ -11,16 +11,22 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the "multi" driver. + * Purpose: The public header file for the multi virtual file driver (VFD) */ #ifndef H5FDmulti_H #define H5FDmulti_H +/** Initializer for the multi VFD */ #define H5FD_MULTI (H5FDperform_init(H5FD_multi_init)) #ifdef __cplusplus extern "C" { #endif + +/** @private + * + * \brief Private initializer for the multi VFD + */ H5_DLL hid_t H5FD_multi_init(void); /** diff --git a/src/H5FDonion.h b/src/H5FDonion.h index 09b290e..4aaab6d 100644 --- a/src/H5FDonion.h +++ b/src/H5FDonion.h @@ -11,19 +11,18 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Onion Virtual File Driver (VFD) - * - * Purpose: The public header file for the Onion VFD. + * Purpose: The public header file for the onion virtual file driver (VFD) */ #ifndef H5FDonion_H #define H5FDonion_H -#define H5FD_ONION (H5FDperform_init(H5FD_onion_init)) +/** Initializer for the onion VFD */ +#define H5FD_ONION (H5FDperform_init(H5FD_onion_init)) + +/** Identifier for the onion VFD */ #define H5FD_ONION_VALUE H5_VFD_ONION -/** - * Current version of the onion VFD fapl info struct. - */ +/** Current version of the onion VFD fapl info struct */ #define H5FD_ONION_FAPL_INFO_VERSION_CURR 1 #define H5FD_ONION_FAPL_INFO_CREATE_FLAG_ENABLE_PAGE_ALIGNMENT \ @@ -114,6 +113,10 @@ typedef struct H5FD_onion_fapl_info_t { extern "C" { #endif +/** @private + * + * \brief Private initializer for the onion VFD + */ H5_DLL hid_t H5FD_onion_init(void); /** diff --git a/src/H5FDprivate.h b/src/H5FDprivate.h index 5330077..2fe54a5 100644 --- a/src/H5FDprivate.h +++ b/src/H5FDprivate.h @@ -214,6 +214,7 @@ H5_DLL herr_t H5FD_get_mpio_atomicity(H5FD_t *file, bool *flag); H5_DLL int H5FD_mpi_get_rank(H5FD_t *file); H5_DLL int H5FD_mpi_get_size(H5FD_t *file); H5_DLL MPI_Comm H5FD_mpi_get_comm(H5FD_t *file); +H5_DLL MPI_Info H5FD_mpi_get_info(H5FD_t *file); H5_DLL herr_t H5FD_mpi_get_file_sync_required(H5FD_t *file, bool *file_sync_required); #endif /* H5_HAVE_PARALLEL */ diff --git a/src/H5FDpublic.h b/src/H5FDpublic.h index 5f40bff..d8d77d6 100644 --- a/src/H5FDpublic.h +++ b/src/H5FDpublic.h @@ -179,6 +179,7 @@ #define H5FD_CTL_INVALID_OPCODE 0 #define H5FD_CTL_TEST_OPCODE 1 #define H5FD_CTL_GET_MPI_COMMUNICATOR_OPCODE 2 +#define H5FD_CTL_GET_MPI_INFO_OPCODE 9 #define H5FD_CTL_GET_MPI_RANK_OPCODE 3 #define H5FD_CTL_GET_MPI_SIZE_OPCODE 4 #define H5FD_CTL_MEM_ALLOC 5 diff --git a/src/H5FDros3.h b/src/H5FDros3.h index ecd2678..217af2d 100644 --- a/src/H5FDros3.h +++ b/src/H5FDros3.h @@ -11,20 +11,24 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Read-Only S3 Virtual File Driver (VFD) - * - * Purpose: The public header file for the ros3 driver. + * Purpose: The public header file for the read-only S3 (ros3) virtual file driver (VFD) */ #ifndef H5FDros3_H #define H5FDros3_H #ifdef H5_HAVE_ROS3_VFD -#define H5FD_ROS3 (H5FDperform_init(H5FD_ros3_init)) +/** Initializer for the ros3 VFD */ +#define H5FD_ROS3 (H5FDperform_init(H5FD_ros3_init)) + +/** Identifier for the ros3 VFD */ #define H5FD_ROS3_VALUE H5_VFD_ROS3 #else +/** Initializer for the ros3 VFD (disabled) */ #define H5FD_ROS3 (H5I_INVALID_HID) + +/** Identifier for the ros3 VFD (disabled) */ #define H5FD_ROS3_VALUE H5_VFD_INVALID -#endif /* H5_HAVE_ROS3_VFD */ +#endif #ifdef H5_HAVE_ROS3_VFD @@ -139,9 +143,9 @@ typedef struct H5FD_ros3_fapl_t { extern "C" { #endif -/** - * \brief Internal routine to initialize #H5FD_ROS3 driver. Not meant to be - * called directly by an HDF5 application. +/** @private + * + * \brief Private initializer for the ros3 VFD */ H5_DLL hid_t H5FD_ros3_init(void); diff --git a/src/H5FDsec2.c b/src/H5FDsec2.c index 2961684..15accf7 100644 --- a/src/H5FDsec2.c +++ b/src/H5FDsec2.c @@ -368,7 +368,7 @@ H5FD__sec2_open(const char *name, unsigned flags, hid_t fapl_id, haddr_t maxaddr } /* Retain a copy of the name used to open the file, for possible error reporting */ - strncpy(file->filename, name, sizeof(file->filename)); + strncpy(file->filename, name, sizeof(file->filename) - 1); file->filename[sizeof(file->filename) - 1] = '\0'; /* Check for non-default FAPL */ diff --git a/src/H5FDsec2.h b/src/H5FDsec2.h index a2590ae..dd0a4d8 100644 --- a/src/H5FDsec2.h +++ b/src/H5FDsec2.h @@ -11,18 +11,26 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the sec2 driver + * Purpose: The public header file for the POSOX I/O (sec2 - "POSIX section 2") + * virtual file driver (VFD) */ #ifndef H5FDsec2_H #define H5FDsec2_H -#define H5FD_SEC2 (H5FDperform_init(H5FD_sec2_init)) +/** Initializer for the sec2 VFD */ +#define H5FD_SEC2 (H5FDperform_init(H5FD_sec2_init)) + +/** Identifier for the sec2 VFD */ #define H5FD_SEC2_VALUE H5_VFD_SEC2 #ifdef __cplusplus extern "C" { #endif +/** @private + * + * \brief Private initializer for the sec2 VFD + */ H5_DLL hid_t H5FD_sec2_init(void); /** @@ -38,7 +46,6 @@ H5_DLL hid_t H5FD_sec2_init(void); * #H5FD_SEC2 driver. * * \since 1.4.0 - * */ H5_DLL herr_t H5Pset_fapl_sec2(hid_t fapl_id); diff --git a/src/H5FDsplitter.h b/src/H5FDsplitter.h index c8751c8..99a471e 100644 --- a/src/H5FDsplitter.h +++ b/src/H5FDsplitter.h @@ -11,96 +11,103 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the "splitter" driver. + * Purpose: The public header file for the splitter virtual file driver (VFD) */ #ifndef H5FDsplitter_H #define H5FDsplitter_H -#define H5FD_SPLITTER (H5FDperform_init(H5FD_splitter_init)) +/** Initializer for the splitter VFD */ +#define H5FD_SPLITTER (H5FDperform_init(H5FD_splitter_init)) + +/** Identifier for the splitter VFD */ #define H5FD_SPLITTER_VALUE H5_VFD_SPLITTER -/* The version of the H5FD_splitter_vfd_config_t structure used */ +/** The version of the H5FD_splitter_vfd_config_t structure used */ #define H5FD_CURR_SPLITTER_VFD_CONFIG_VERSION 1 -/* Maximum length of a filename/path string in the Write-Only channel, +/** + * Maximum length of a filename/path string in the Write-Only channel, * including the NULL-terminator. */ #define H5FD_SPLITTER_PATH_MAX 4096 -/* Semi-unique constant used to help identify structure pointers */ +/** Semi-unique constant used to help identify structure pointers */ #define H5FD_SPLITTER_MAGIC 0x2B916880 -/* ---------------------------------------------------------------------------- - * Structure: H5FD_spliiter_vfd_config_t - * - * One-stop shopping for configuring a Splitter VFD (rather than many - * parameters passed into H5Pset/get functions). - * - * magic (int32_t) - * Semi-unique number, used to sanity-check that a given pointer is - * likely (or not) to be this structure type. MUST be first. - * If magic is not H5FD_SPLITTER_MAGIC, the structure (and/or pointer to) - * must be considered invalid. - * - * version (unsigned int) - * Version number of this structure -- informs component membership. - * If not H5FD_CURR_SPLITTER_VFD_CONFIG_VERSION, the structure (and/or - * pointer to) must be considered invalid. - * - * rw_fapl_id (hid_t) - * Library-given identification number of the Read/Write channel driver - * File Access Property List. - * The driver must support read/write access. - * Must be set to H5P_DEFAULT or a valid FAPL ID. - * - * wo_fapl_id (hid_t) - * Library-given identification number of the Read/Write channel driver - * File Access Property List. - * The driver feature flags must include H5FD_FEAT_DEFAULT_VFD_COMPAITBLE. - * Must be set to H5P_DEFAULT or a valid FAPL ID. - * - * wo_file_path (char[H5FD_SPLITTER_PATH_MAX + 1]) - * String buffer for the Write-Only channel target file. - * Must be null-terminated, cannot be empty. - * - * log_file_path (char[H5FD_SPLITTER_PATH_MAX + 1]) - * String buffer for the Splitter VFD logging output. - * Must be null-terminated. - * If null, no logfile is created. - * - * ignore_wo_errors (hbool_t) - * Toggle flag for how judiciously to respond to errors on the Write-Only - * channel. - * - * ---------------------------------------------------------------------------- +//! <!-- [H5FD_splitter_vfd_config_t_snip] --> +/** + * Configuration options for setting up the Splitter VFD */ typedef struct H5FD_splitter_vfd_config_t { - int32_t magic; - unsigned int version; - hid_t rw_fapl_id; - hid_t wo_fapl_id; - char wo_path[H5FD_SPLITTER_PATH_MAX + 1]; - char log_file_path[H5FD_SPLITTER_PATH_MAX + 1]; - hbool_t ignore_wo_errs; + int32_t magic; /**< Magic number to identify this struct. Must be \p H5FD_SPLITTER_MAGIC. */ + unsigned int version; /**< Version number of this struct. Currently must be \p + H5FD_CURR_SPLITTER_VFD_CONFIG_VERSION. */ + hid_t rw_fapl_id; /**< File-access property list for setting up the read/write channel. Can be \p + H5P_DEFAULT. */ + hid_t wo_fapl_id; /**< File-access property list for setting up the read-only channel. The selected VFD + must support the \p H5FD_FEAT_DEFAULT_VFD_COMPATIBLE flag. Can be \p H5P_DEFAULT. */ + char wo_path[H5FD_SPLITTER_PATH_MAX + 1]; /**< Path to the write-only file */ + char log_file_path[H5FD_SPLITTER_PATH_MAX + 1]; /**< Path to the log file, which will be created on HDF5 + file open (existing files will be clobbered). Can be + NULL, in which case no logging output is generated. */ + hbool_t ignore_wo_errs; /**< Whether to ignore errors on the write-only channel */ } H5FD_splitter_vfd_config_t; +//! <!-- [H5FD_splitter_vfd_config_t_snip] --> #ifdef __cplusplus extern "C" { #endif + +/** @private + * + * \brief Private initializer for the splitter VFD + */ H5_DLL hid_t H5FD_splitter_init(void); /** * \ingroup FAPL * - * \todo Add missing documentation + * \brief Sets the file access property list to use the splitter driver + * + * \fapl_id + * \param[in] config_ptr Configuration options for the VFD + * \returns \herr_t + * + * \details H5Pset_fapl_splitter() sets the file access property list identifier, + * \p fapl_id, to use the splitter driver. + * + * The splitter VFD echoes file manipulation (e.g. create, truncate) + * and write calls to a second, write-only file. + * + * \note The splitter VFD should not be confused with the split VFD, + * which is a simplification of the multi VFD and creates separate + * files for metadata and data. + * + * \since 1.10.7, 1.12.1 */ H5_DLL herr_t H5Pset_fapl_splitter(hid_t fapl_id, H5FD_splitter_vfd_config_t *config_ptr); /** * \ingroup FAPL * - * \todo Add missing documentation + * \brief Gets splitter driver properties from the the file access property list + * + * \fapl_id + * \param[out] config_ptr Configuration options for the VFD + * \returns \herr_t + * + * \details H5Pset_fapl_splitter() sets the file access property list identifier, + * \p fapl_id, to use the splitter driver. + * + * The splitter VFD echoes file manipulation (e.g. create, truncate) + * and write calls to a second file. + * + * \note The splitter VFD should not be confused with the split VFD, + * which is a simplification of the multi VFD and creates separate + * files for metadata and data. + * + * \since 1.10.7, 1.12.1 */ H5_DLL herr_t H5Pget_fapl_splitter(hid_t fapl_id, H5FD_splitter_vfd_config_t *config_ptr); diff --git a/src/H5FDstdio.h b/src/H5FDstdio.h index e2e05a7..794fe31 100644 --- a/src/H5FDstdio.h +++ b/src/H5FDstdio.h @@ -11,20 +11,26 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the C stdio driver + * Purpose: The public header file for the C stdio virtual file driver (VFD) */ #ifndef H5FDstdio_H #define H5FDstdio_H #include "H5Ipublic.h" +/** Initializer for the stdio VFD */ #define H5FD_STDIO (H5FDperform_init(H5FD_stdio_init)) #ifdef __cplusplus extern "C" { #endif +/** @private + * + * \brief Private initializer for the stdio VFD + */ H5_DLL hid_t H5FD_stdio_init(void); + /** * \ingroup FAPL * @@ -34,7 +40,11 @@ H5_DLL hid_t H5FD_stdio_init(void); * \returns \herr_t * * \details H5Pset_fapl_stdio() modifies the file access property list to use - * the standard I/O driver, H5FDstdio(). + * the stdio VFD, which uses I/O calls from stdio.h. + * + * \note This VFD was designed to be a "demo" VFD that shows how to write + * your own VFD. Most applications should not use this VFD and should instead + * use the POSIX I/O VFD (sec2). * * \since 1.4.0 * diff --git a/src/H5FDsubfiling/H5FDsubfiling.c b/src/H5FDsubfiling/H5FDsubfiling.c index a2daba0..461fa16 100644 --- a/src/H5FDsubfiling/H5FDsubfiling.c +++ b/src/H5FDsubfiling/H5FDsubfiling.c @@ -2551,6 +2551,12 @@ H5FD__subfiling_ctl(H5FD_t *_file, uint64_t op_code, uint64_t flags, const void **((MPI_Comm **)output) = file->ext_comm; break; + case H5FD_CTL_GET_MPI_INFO_OPCODE: + assert(output); + assert(*output); + **((MPI_Info **)output) = file->info; + break; + case H5FD_CTL_GET_MPI_RANK_OPCODE: assert(output); assert(*output); diff --git a/src/H5FDwindows.h b/src/H5FDwindows.h index 14f6985..673d1c9 100644 --- a/src/H5FDwindows.h +++ b/src/H5FDwindows.h @@ -11,11 +11,16 @@ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ /* - * Purpose: The public header file for the Windows driver + * Purpose: The public header file for the Windows virtual file driver (VFD) + * + * This VFD uses no Win32 API calls directly (though it may be + * rewritten to do so in the future). It is currently defined to + * be the sec2 VFD. */ #ifndef H5FDwindows_H #define H5FDwindows_H +/** Initializer for the Windows VFD */ #define H5FD_WINDOWS (H5FD_sec2_init()) #ifdef __cplusplus diff --git a/src/H5Fint.c b/src/H5Fint.c index 4093b4b..8738026 100644 --- a/src/H5Fint.c +++ b/src/H5Fint.c @@ -410,11 +410,11 @@ H5F_get_access_plist(H5F_t *f, bool app_ref) if (H5P_set(new_plist, H5F_ACS_MPI_PARAMS_COMM_NAME, &mpi_comm) < 0) HGOTO_ERROR(H5E_FILE, H5E_CANTSET, H5I_INVALID_HID, "can't set MPI communicator"); - /* Retrieve and set MPI info object */ - if (H5P_get(old_plist, H5F_ACS_MPI_PARAMS_INFO_NAME, &mpi_info) < 0) - HGOTO_ERROR(H5E_FILE, H5E_CANTGET, H5I_INVALID_HID, "can't get MPI info object"); + /* Retrieve and set MPI info */ + if (MPI_INFO_NULL == (mpi_info = H5F_mpi_get_info(f))) + HGOTO_ERROR(H5E_FILE, H5E_CANTGET, H5I_INVALID_HID, "can't get MPI info"); if (H5P_set(new_plist, H5F_ACS_MPI_PARAMS_INFO_NAME, &mpi_info) < 0) - HGOTO_ERROR(H5E_FILE, H5E_CANTSET, H5I_INVALID_HID, "can't set MPI info object"); + HGOTO_ERROR(H5E_FILE, H5E_CANTSET, H5I_INVALID_HID, "can't set MPI info"); } #endif /* H5_HAVE_PARALLEL */ if (H5P_set(new_plist, H5F_ACS_META_CACHE_INIT_IMAGE_CONFIG_NAME, &(f->shared->mdc_initCacheImageCfg)) < diff --git a/src/H5Fmpi.c b/src/H5Fmpi.c index 8a8fdc1..7a535e9 100644 --- a/src/H5Fmpi.c +++ b/src/H5Fmpi.c @@ -97,11 +97,10 @@ done: /*------------------------------------------------------------------------- * Function: H5F_mpi_get_comm * - * Purpose: Retrieves the file's communicator + * Purpose: Retrieves the file's MPI_Comm communicator object * - * Return: Success: The communicator (non-negative) - * - * Failure: Negative + * Return: Success: The communicator object + * Failure: MPI_COMM_NULL * *------------------------------------------------------------------------- */ @@ -123,6 +122,33 @@ done: } /* end H5F_mpi_get_comm() */ /*------------------------------------------------------------------------- + * Function: H5F_mpi_get_info + * + * Purpose: Retrieves the file's MPI_Info info object + * + * Return: Success: The info object + * Failure: MPI_INFO_NULL + * + *------------------------------------------------------------------------- + */ +MPI_Info +H5F_mpi_get_info(const H5F_t *f) +{ + MPI_Info ret_value = MPI_INFO_NULL; + + FUNC_ENTER_NOAPI(MPI_INFO_NULL) + + assert(f && f->shared); + + /* Dispatch to driver */ + if ((ret_value = H5FD_mpi_get_info(f->shared->lf)) == MPI_INFO_NULL) + HGOTO_ERROR(H5E_FILE, H5E_CANTGET, MPI_INFO_NULL, "driver get_info request failed"); + +done: + FUNC_LEAVE_NOAPI(ret_value) +} /* end H5F_mpi_get_info() */ + +/*------------------------------------------------------------------------- * Function: H5F_shared_mpi_get_size * * Purpose: Retrieves the size of an MPI process. diff --git a/src/H5Fprivate.h b/src/H5Fprivate.h index 9adbf3a..682e938 100644 --- a/src/H5Fprivate.h +++ b/src/H5Fprivate.h @@ -640,6 +640,7 @@ H5_DLL herr_t H5F_eoa_dirty(H5F_t *f); #ifdef H5_HAVE_PARALLEL H5_DLL int H5F_mpi_get_rank(const H5F_t *f); H5_DLL MPI_Comm H5F_mpi_get_comm(const H5F_t *f); +H5_DLL MPI_Info H5F_mpi_get_info(const H5F_t *f); H5_DLL int H5F_shared_mpi_get_size(const H5F_shared_t *f_sh); H5_DLL int H5F_mpi_get_size(const H5F_t *f); H5_DLL herr_t H5F_mpi_retrieve_comm(hid_t loc_id, hid_t acspl_id, MPI_Comm *mpi_comm); diff --git a/src/H5Smpio.c b/src/H5Smpio.c index 1e0fef5..46e6c33 100644 --- a/src/H5Smpio.c +++ b/src/H5Smpio.c @@ -989,7 +989,7 @@ H5S__mpio_span_hyper_type(const H5S_t *space, size_t elmt_size, MPI_Datatype *ne &type_list, 0, op_gen) < 0) HGOTO_ERROR(H5E_DATASPACE, H5E_BADTYPE, FAIL, "couldn't obtain MPI derived data type"); if (MPI_SUCCESS != (mpi_code = MPI_Type_dup(span_type, new_type))) - HMPI_GOTO_ERROR(FAIL, "MPI_Type_commit failed", mpi_code) + HMPI_GOTO_ERROR(FAIL, "MPI_Type_dup failed", mpi_code) if (MPI_SUCCESS != (mpi_code = MPI_Type_commit(new_type))) HMPI_GOTO_ERROR(FAIL, "MPI_Type_commit failed", mpi_code) @@ -2500,7 +2500,7 @@ H5T__register(H5T_pers_t pers, const char *name, H5T_t *src, H5T_t *dst, H5T_con H5T_g.asoft = na; H5T_g.soft = x; } /* end if */ - strncpy(H5T_g.soft[H5T_g.nsoft].name, name, (size_t)H5T_NAMELEN); + strncpy(H5T_g.soft[H5T_g.nsoft].name, name, (size_t)H5T_NAMELEN - 1); H5T_g.soft[H5T_g.nsoft].name[H5T_NAMELEN - 1] = '\0'; H5T_g.soft[H5T_g.nsoft].src = src->shared->type; H5T_g.soft[H5T_g.nsoft].dst = dst->shared->type; @@ -2549,7 +2549,7 @@ H5T__register(H5T_pers_t pers, const char *name, H5T_t *src, H5T_t *dst, H5T_con /* Create a new conversion path */ if (NULL == (new_path = H5FL_CALLOC(H5T_path_t))) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed"); - strncpy(new_path->name, name, (size_t)H5T_NAMELEN); + strncpy(new_path->name, name, (size_t)H5T_NAMELEN - 1); new_path->name[H5T_NAMELEN - 1] = '\0'; if (NULL == (new_path->src = H5T_copy(old_path->src, H5T_COPY_ALL)) || NULL == (new_path->dst = H5T_copy(old_path->dst, H5T_COPY_ALL))) @@ -4952,7 +4952,7 @@ H5T__path_find_real(const H5T_t *src, const H5T_t *dst, const char *name, H5T_co if (NULL == (path = H5FL_CALLOC(H5T_path_t))) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed for type conversion path"); if (name && *name) { - strncpy(path->name, name, (size_t)H5T_NAMELEN); + strncpy(path->name, name, (size_t)H5T_NAMELEN - 1); path->name[H5T_NAMELEN - 1] = '\0'; } /* end if */ else diff --git a/src/H5Tinit_float.c b/src/H5Tinit_float.c index 3b9e127..02bb3ba 100644 --- a/src/H5Tinit_float.c +++ b/src/H5Tinit_float.c @@ -51,19 +51,23 @@ * Function: DETECT_F * * Purpose: This macro takes a floating point type like `double' and - * a base name like `natd' and detects byte order, mantissa - * location, exponent location, sign bit location, presence or - * absence of implicit mantissa bit, and exponent bias and - * initializes a detected_t structure with those properties. + * detects byte order, mantissa location, exponent location, + * sign bit location, presence or absence of implicit mantissa + * bit, and exponent bias and initializes a detected_t structure + * with those properties. + * + * Note that these operations can raise floating-point + * exceptions and building with some compiler options + * (especially Fortran) can cause problems. *------------------------------------------------------------------------- */ -#define DETECT_F(TYPE, VAR, INFO) \ +#define DETECT_F(TYPE, INFO) \ do { \ - TYPE _v1, _v2, _v3; \ - unsigned char _buf1[sizeof(TYPE)], _buf3[sizeof(TYPE)]; \ - unsigned char _pad_mask[sizeof(TYPE)]; \ - unsigned char _byte_mask; \ - int _i, _j, _last = (-1); \ + TYPE _v1, _v2, _v3; \ + uint8_t _buf1[sizeof(TYPE)], _buf3[sizeof(TYPE)]; \ + uint8_t _pad_mask[sizeof(TYPE)]; \ + uint8_t _byte_mask; \ + int _i, _j, _last = -1; \ \ memset(&INFO, 0, sizeof(INFO)); \ INFO.size = sizeof(TYPE); \ @@ -81,7 +85,7 @@ _v1 = (TYPE)4.0L; \ H5MM_memcpy(_buf1, (const void *)&_v1, sizeof(TYPE)); \ for (_i = 0; _i < (int)sizeof(TYPE); _i++) \ - for (_byte_mask = (unsigned char)1; _byte_mask; _byte_mask = (unsigned char)(_byte_mask << 1)) { \ + for (_byte_mask = (uint8_t)1; _byte_mask; _byte_mask = (uint8_t)(_byte_mask << 1)) { \ _buf1[_i] ^= _byte_mask; \ H5MM_memcpy((void *)&_v2, (const void *)_buf1, sizeof(TYPE)); \ H5_GCC_CLANG_DIAG_OFF("float-equal") \ @@ -118,7 +122,7 @@ _v1 = (TYPE)1.0L; \ _v2 = (TYPE)-1.0L; \ if (H5T__bit_cmp(sizeof(TYPE), INFO.perm, &_v1, &_v2, _pad_mask, &(INFO.sign)) < 0) \ - HGOTO_ERROR(H5E_DATATYPE, H5E_CANTINIT, FAIL, "failed to detect byte order"); \ + HGOTO_ERROR(H5E_DATATYPE, H5E_CANTINIT, FAIL, "failed to determine sign bit"); \ \ /* Mantissa */ \ INFO.mpos = 0; \ @@ -126,12 +130,11 @@ _v1 = (TYPE)1.0L; \ _v2 = (TYPE)1.5L; \ if (H5T__bit_cmp(sizeof(TYPE), INFO.perm, &_v1, &_v2, _pad_mask, &(INFO.msize)) < 0) \ - HGOTO_ERROR(H5E_DATATYPE, H5E_CANTINIT, FAIL, "failed to detect byte order"); \ + HGOTO_ERROR(H5E_DATATYPE, H5E_CANTINIT, FAIL, "failed to determine mantissa"); \ INFO.msize += 1 + (unsigned)(INFO.imp ? 0 : 1) - INFO.mpos; \ \ /* Exponent */ \ - INFO.epos = INFO.mpos + INFO.msize; \ - \ + INFO.epos = INFO.mpos + INFO.msize; \ INFO.esize = INFO.sign - INFO.epos; \ \ _v1 = (TYPE)1.0L; \ @@ -456,17 +459,24 @@ H5T__set_precision(H5T_fpoint_det_t *d) herr_t H5_NO_UBSAN H5T__init_native_float_types(void) { + fenv_t saved_fenv; H5T_fpoint_det_t det; H5T_t *dt = NULL; herr_t ret_value = SUCCEED; FUNC_ENTER_PACKAGE + /* Turn off floating-point exceptions while initializing to avoid + * tripping over signaling NaNs while looking at "don't care" bits. + */ + if (feholdexcept(&saved_fenv) != 0) + HSYS_GOTO_ERROR(H5E_DATATYPE, H5E_CANTSET, FAIL, "can't save floating-point environment"); + /* H5T_NATIVE_FLOAT */ /* Get the type's characteristics */ memset(&det, 0, sizeof(H5T_fpoint_det_t)); - DETECT_F(float, FLOAT, det); + DETECT_F(float, det); /* Allocate and fill type structure */ if (NULL == (dt = H5T__alloc())) @@ -497,7 +507,7 @@ H5T__init_native_float_types(void) /* Get the type's characteristics */ memset(&det, 0, sizeof(H5T_fpoint_det_t)); - DETECT_F(double, DOUBLE, det); + DETECT_F(double, det); /* Allocate and fill type structure */ if (NULL == (dt = H5T__alloc())) @@ -528,7 +538,7 @@ H5T__init_native_float_types(void) /* Get the type's characteristics */ memset(&det, 0, sizeof(H5T_fpoint_det_t)); - DETECT_F(long double, LDOUBLE, det); + DETECT_F(long double, det); /* Allocate and fill type structure */ if (NULL == (dt = H5T__alloc())) @@ -561,6 +571,14 @@ H5T__init_native_float_types(void) H5T_native_order_g = det.order; done: + /* Clear any FE_INVALID exceptions from NaN handling */ + if (feclearexcept(FE_INVALID) != 0) + HSYS_GOTO_ERROR(H5E_DATATYPE, H5E_CANTSET, FAIL, "can't clear floating-point exceptions"); + + /* Restore the original environment */ + if (feupdateenv(&saved_fenv) != 0) + HSYS_GOTO_ERROR(H5E_DATATYPE, H5E_CANTSET, FAIL, "can't restore floating-point environment"); + if (ret_value < 0) { if (dt != NULL) { dt->shared = H5FL_FREE(H5T_shared_t, dt->shared); diff --git a/src/H5Tvlen.c b/src/H5Tvlen.c index 68087a8..651ff8f 100644 --- a/src/H5Tvlen.c +++ b/src/H5Tvlen.c @@ -503,7 +503,7 @@ H5T__vlen_mem_seq_write(H5VL_object_t H5_ATTR_UNUSED *file, const H5T_vlen_alloc if (seq_len) { size_t len = seq_len * base_size; /* Sequence size */ - /* Use the user's memory allocation routine is one is defined */ + /* Use the user's memory allocation routine if one is defined */ if (vl_alloc_info->alloc_func != NULL) { if (NULL == (vl.p = (vl_alloc_info->alloc_func)(len, vl_alloc_info->alloc_info))) HGOTO_ERROR(H5E_DATATYPE, H5E_CANTALLOC, FAIL, @@ -485,19 +485,30 @@ done: *------------------------------------------------------------------------- */ static int -H5Z__check_unregister_group_cb(void *obj_ptr, hid_t H5_ATTR_UNUSED obj_id, void *key) +H5Z__check_unregister_group_cb(void H5_ATTR_UNUSED *obj_ptr, hid_t obj_id, void *key) { - hid_t ocpl_id = -1; - H5Z_object_t *object = (H5Z_object_t *)key; - htri_t filter_in_pline = false; - int ret_value = false; /* Return value */ + hid_t ocpl_id = -1; + H5Z_object_t *object = (H5Z_object_t *)key; + H5VL_object_t *vol_obj; /* Object for loc_id */ + H5VL_group_get_args_t vol_cb_args; /* Arguments to VOL callback */ + htri_t filter_in_pline = false; + int ret_value = false; /* Return value */ FUNC_ENTER_PACKAGE - assert(obj_ptr); - /* Get the group creation property */ - if ((ocpl_id = H5G_get_create_plist((H5G_t *)obj_ptr)) < 0) + if (NULL == (vol_obj = (H5VL_object_t *)H5I_object_verify(obj_id, H5I_GROUP))) + HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, H5I_INVALID_HID, "invalid group identifier"); + + /* Set up VOL callback arguments */ + vol_cb_args.op_type = H5VL_GROUP_GET_GCPL; + vol_cb_args.args.get_gcpl.gcpl_id = H5I_INVALID_HID; + + /* Get the group creation property list */ + if (H5VL_group_get(vol_obj, &vol_cb_args, H5P_DATASET_XFER_DEFAULT, H5_REQUEST_NULL) < 0) + HGOTO_ERROR(H5E_PLINE, H5E_CANTGET, H5I_INVALID_HID, "unable to get group creation properties"); + + if ((ocpl_id = vol_cb_args.args.get_gcpl.gcpl_id) < 0) HGOTO_ERROR(H5E_PLINE, H5E_CANTGET, FAIL, "can't get group creation property list"); /* Check if the filter is in the group creation property list */ @@ -535,19 +546,30 @@ done: *------------------------------------------------------------------------- */ static int -H5Z__check_unregister_dset_cb(void *obj_ptr, hid_t H5_ATTR_UNUSED obj_id, void *key) +H5Z__check_unregister_dset_cb(void H5_ATTR_UNUSED *obj_ptr, hid_t obj_id, void *key) { - hid_t ocpl_id = -1; - H5Z_object_t *object = (H5Z_object_t *)key; - htri_t filter_in_pline = false; - int ret_value = false; /* Return value */ + hid_t ocpl_id = -1; + H5Z_object_t *object = (H5Z_object_t *)key; + H5VL_object_t *vol_obj; /* Object for loc_id */ + H5VL_dataset_get_args_t vol_cb_args; /* Arguments to VOL callback */ + htri_t filter_in_pline = false; + int ret_value = false; /* Return value */ FUNC_ENTER_PACKAGE - assert(obj_ptr); - /* Get the dataset creation property */ - if ((ocpl_id = H5D_get_create_plist((H5D_t *)obj_ptr)) < 0) + if (NULL == (vol_obj = (H5VL_object_t *)H5I_object_verify(obj_id, H5I_DATASET))) + HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, H5I_INVALID_HID, "invalid dataset identifier"); + + /* Set up VOL callback arguments */ + vol_cb_args.op_type = H5VL_DATASET_GET_DCPL; + vol_cb_args.args.get_dcpl.dcpl_id = H5I_INVALID_HID; + + /* Get the dataset creation property list */ + if (H5VL_dataset_get(vol_obj, &vol_cb_args, H5P_DATASET_XFER_DEFAULT, H5_REQUEST_NULL) < 0) + HGOTO_ERROR(H5E_DATASET, H5E_CANTGET, H5I_INVALID_HID, "unable to get dataset creation properties"); + + if ((ocpl_id = vol_cb_args.args.get_dcpl.dcpl_id) < 0) HGOTO_ERROR(H5E_PLINE, H5E_CANTGET, FAIL, "can't get dataset creation property list"); /* Check if the filter is in the dataset creation property list */ @@ -581,51 +603,83 @@ done: *------------------------------------------------------------------------- */ static int -H5Z__flush_file_cb(void *obj_ptr, hid_t H5_ATTR_UNUSED obj_id, void H5_ATTR_PARALLEL_USED *key) +H5Z__flush_file_cb(void H5_ATTR_UNUSED *obj_ptr, hid_t obj_id, void H5_ATTR_PARALLEL_USED *key) { - H5F_t *f = (H5F_t *)obj_ptr; /* File object for operations */ + #ifdef H5_HAVE_PARALLEL H5Z_object_t *object = (H5Z_object_t *)key; -#endif /* H5_HAVE_PARALLEL */ - int ret_value = false; /* Return value */ +#endif /* H5_HAVE_PARALLEL */ + int ret_value = false; /* Return value */ + H5VL_file_specific_args_t vol_cb_args_specific; /* Arguments to VOL callback */ + H5VL_object_t *vol_obj; /* File for file_id */ + H5VL_file_get_args_t vol_cb_args; /* Arguments to VOL callback */ + bool is_native_vol_obj = true; + unsigned int intent = 0; FUNC_ENTER_PACKAGE /* Sanity checks */ - assert(obj_ptr); assert(key); + /* Get the internal file structure */ + if (NULL == (vol_obj = (H5VL_object_t *)H5I_object(obj_id))) + HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "invalid file identifier"); + + /* Get intent */ + vol_cb_args.op_type = H5VL_FILE_GET_INTENT; + vol_cb_args.args.get_intent.flags = &intent; + + /* Get the flags */ + if (H5VL_file_get(vol_obj, &vol_cb_args, H5P_DATASET_XFER_DEFAULT, H5_REQUEST_NULL) < 0) + HGOTO_ERROR(H5E_FILE, H5E_CANTGET, FAIL, "unable to get file's intent flags"); + + if (H5VL_object_is_native(vol_obj, &is_native_vol_obj) < 0) + HGOTO_ERROR(H5E_OHDR, H5E_CANTGET, H5I_INVALID_HID, + "can't determine if VOL object is native connector object"); + /* Do a global flush if the file is opened for write */ - if (H5F_ACC_RDWR & H5F_INTENT(f)) { + if (H5F_ACC_RDWR & intent) { #ifdef H5_HAVE_PARALLEL - /* Check if MPIO driver is used */ - if (H5F_HAS_FEATURE(f, H5FD_FEAT_HAS_MPI)) { - /* Sanity check for collectively calling H5Zunregister, if requested */ - /* (Sanity check assumes that a barrier on one file's comm - * is sufficient (i.e. that there aren't different comms for - * different files). -QAK, 2018/02/14) - */ - if (H5_coll_api_sanity_check_g && !object->sanity_checked) { - MPI_Comm mpi_comm; /* File's communicator */ + /* Checking MPI flag requires native VOL */ + if (is_native_vol_obj) { + H5F_t *f = (H5F_t *)obj_ptr; /* File object for native VOL operation */ + + /* Check if MPIO driver is used */ + if (H5F_HAS_FEATURE(f, H5FD_FEAT_HAS_MPI)) { - /* Retrieve the file communicator */ - if (MPI_COMM_NULL == (mpi_comm = H5F_mpi_get_comm(f))) - HGOTO_ERROR(H5E_PLINE, H5E_CANTGET, FAIL, "can't get MPI communicator"); + /* Sanity check for collectively calling H5Zunregister, if requested */ + /* (Sanity check assumes that a barrier on one file's comm + * is sufficient (i.e. that there aren't different comms for + * different files). -QAK, 2018/02/14) + */ + if (H5_coll_api_sanity_check_g && !object->sanity_checked) { + MPI_Comm mpi_comm; /* File's communicator */ - /* Issue the barrier */ - if (mpi_comm != MPI_COMM_NULL) - MPI_Barrier(mpi_comm); + /* Retrieve the file communicator */ + if (H5F_mpi_retrieve_comm(obj_id, H5P_DEFAULT, &mpi_comm) < 0) + HGOTO_ERROR(H5E_PLINE, H5E_CANTGET, FAIL, "can't get MPI communicator"); - /* Set the "sanity checked" flag */ - object->sanity_checked = true; - } /* end if */ - } /* end if */ -#endif /* H5_HAVE_PARALLEL */ + /* Issue the barrier */ + if (mpi_comm != MPI_COMM_NULL) + MPI_Barrier(mpi_comm); + + /* Set the "sanity checked" flag */ + object->sanity_checked = true; + } /* end if */ + } /* end if */ + } +#endif /* H5_HAVE_PARALLEL */ /* Call the flush routine for mounted file hierarchies */ - if (H5F_flush_mounts((H5F_t *)obj_ptr) < 0) - HGOTO_ERROR(H5E_PLINE, H5E_CANTFLUSH, FAIL, "unable to flush file hierarchy"); + vol_cb_args_specific.op_type = H5VL_FILE_FLUSH; + vol_cb_args_specific.args.flush.obj_type = H5I_FILE; + vol_cb_args_specific.args.flush.scope = H5F_SCOPE_GLOBAL; + + /* Flush the object */ + if (H5VL_file_specific(vol_obj, &vol_cb_args_specific, H5P_DATASET_XFER_DEFAULT, NULL) < 0) + HGOTO_ERROR(H5E_FILE, H5E_CANTFLUSH, FAIL, "unable to flush file hierarchy"); + } /* end if */ done: diff --git a/src/H5mpi.c b/src/H5mpi.c index 2725ec5..cf7e33d 100644 --- a/src/H5mpi.c +++ b/src/H5mpi.c @@ -380,9 +380,9 @@ H5_mpi_info_cmp(MPI_Info info1, MPI_Info info2, int *result) /* Allocate buffers for iteration */ if (NULL == (key = (char *)H5MM_malloc(MPI_MAX_INFO_KEY * sizeof(char)))) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed"); - if (NULL == (value1 = (char *)H5MM_malloc(MPI_MAX_INFO_VAL * sizeof(char)))) + if (NULL == (value1 = (char *)H5MM_malloc((MPI_MAX_INFO_VAL + 1) * sizeof(char)))) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed"); - if (NULL == (value2 = (char *)H5MM_malloc(MPI_MAX_INFO_VAL * sizeof(char)))) + if (NULL == (value2 = (char *)H5MM_malloc((MPI_MAX_INFO_VAL + 1) * sizeof(char)))) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed"); /* Iterate over the keys, comparing them */ diff --git a/src/H5private.h b/src/H5private.h index 14a0ac3..3aaa0d5 100644 --- a/src/H5private.h +++ b/src/H5private.h @@ -26,6 +26,7 @@ #include <ctype.h> #include <errno.h> #include <fcntl.h> +#include <fenv.h> #include <float.h> #include <math.h> #include <setjmp.h> diff --git a/src/H5system.c b/src/H5system.c index 30a89a1..be886ae 100644 --- a/src/H5system.c +++ b/src/H5system.c @@ -807,13 +807,12 @@ H5_nanosleep(uint64_t nanosec) #ifdef H5_HAVE_WIN32_API DWORD dwMilliseconds = (DWORD)ceil(nanosec / 1.0e6); - DWORD ignore; /* Windows can't sleep at a ns resolution. Best we can do is ~1 ms. We * don't care about the return value since the second parameter * (bAlertable) is false, so it will always be zero. */ - ignore = SleepEx(dwMilliseconds, false); + SleepEx(dwMilliseconds, false); #else |