diff options
author | jhendersonHDF <jhenderson@hdfgroup.org> | 2023-10-24 02:08:06 (GMT) |
---|---|---|
committer | GitHub <noreply@github.com> | 2023-10-24 02:08:06 (GMT) |
commit | 8cff88c5073b0bbd66c5876e0ce3da16214af39c (patch) | |
tree | f1dd8065cc17fbbb749b1c30e5d91e9fe6dbd998 /release_docs/RELEASE.txt | |
parent | af49eb5b8647e8d9ffb527fd533def0910eb535c (diff) | |
download | hdf5-8cff88c5073b0bbd66c5876e0ce3da16214af39c.zip hdf5-8cff88c5073b0bbd66c5876e0ce3da16214af39c.tar.gz hdf5-8cff88c5073b0bbd66c5876e0ce3da16214af39c.tar.bz2 |
Fix some issues with collective metadata reads for chunked datasets (#3716)
Add functions/callbacks for explicit control over chunk index open/close
Add functions/callbacks to check if chunk index is open or not so
that it can be opened if necessary before temporarily disabling
collective metadata reads in the library
Add functions/callbacks for requesting loading of additional chunk
index metadata beyond the chunk index itself
Diffstat (limited to 'release_docs/RELEASE.txt')
-rw-r--r-- | release_docs/RELEASE.txt | 40 |
1 files changed, 40 insertions, 0 deletions
diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt index e5d53e4..e5cb097 100644 --- a/release_docs/RELEASE.txt +++ b/release_docs/RELEASE.txt @@ -392,6 +392,46 @@ Bug Fixes since HDF5-1.14.0 release =================================== Library ------- + - Fixed some issues with chunk index metadata not getting read + collectively when collective metadata reads are enabled + + When looking up dataset chunks during I/O, the parallel library + temporarily disables collective metadata reads since it's generally + unlikely that the application will read the same chunks from all + MPI ranks. Leaving collective metadata reads enabled during + chunk lookups can lead to hangs or other bad behavior depending + on the chunk indexing structure used for the dataset in question. + However, due to the way that dataset chunk index metadata was + previously loaded in a deferred manner, this could mean that + the metadata for the main chunk index structure or its + accompanying pieces of metadata (e.g., fixed array data blocks) + could end up being read independently if these chunk lookup + operations are the first chunk index-related operation that + occurs on a dataset. This behavior is generally observed when + opening a dataset for which the metadata isn't in the metadata + cache yet and then immediately performing I/O on that dataset. + This behavior is not generally observed when creating a dataset + and then performing I/O on it, as the relevant metadata will + usually be in the metadata cache as a side effect of creating + the chunk index structures during dataset creation. + + This issue has been fixed by adding callbacks to the different + chunk indexing structure classes that allow more explicit control + over when chunk index metadata gets loaded. When collective + metadata reads are enabled, the necessary index metadata will now + get loaded collectively by all MPI ranks at the start of dataset + I/O to ensure that the ranks don't unintentionally read this + metadata independently further on. These changes fix collective + loading of the main chunk index structure, as well as v2 B-tree + root nodes, extensible array index blocks and fixed array data + blocks. There are still pieces of metadata that cannot currently + be loaded collectively, however, such as extensible array data + blocks, data block pages and super blocks, as well as fixed array + data block pages. These pieces of metadata are not necessarily + read in by all MPI ranks since this depends on which chunks the + ranks have selected in the dataset. Therefore, reading of these + pieces of metadata remains an independent operation. + - Fixed potential hangs in parallel library during collective I/O with independent metadata writes |