| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
|
|
|
|
| |
time in H5PB_dest(). While we're in H5PB_dest(), mark deleted shadow-index
entries as "garbage" and skip the O(n) shadow index-entries copy.
Rename shadow index-entry member `moved_to_hdf5_file` to `moved_to_lower_file`
while I'm in here---NFCI.
|
|
|
|
| |
with other code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add to the H5F_shared_t (!) a new member that tells the index in
the shadow file where the index should be written.
Allocate shadow filespace for the header and the index separately so
that the index can float. Update tests to match the expected original
location of the index.
Introduce vfd_swmr_enlarge_shadow_index(), a routine that allocates space in
the shadow file for a new index that has (up to) twice as many entries as the
old index, allocates a new in-core index of the same size, and copies the old
in-core index to the new. Call vfd_swmr_enlarge_shadow_index() in
H5PB_vfd_swmr__update_index() when the in-core index has too few slots.
In the comment at the top of H5FD__vfd_swmr_load_hdr_and_idx(), describe the
protocol that it follows, now, when it reads the shadow header and index.
Delete some dead code in the function and add a bit of diagnostic code.
TBD quiet the diagnostic code.
In H5F_vfd_swmr_init(), follow the protocol: write the index, first, then
the header.
Modify property-list checks and tests to reserve no fewer than two pages at the
front of the shadow file for the header and index.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
the way that the shadow header and shadow index are loaded.
In H5FD__vfd_swmr_load_hdr_and_idx(), adopt a new protocol for reading
the shadow file:
0 If the maximum number of retries have been attempted, then exit
with an error.
1 Try to read the shadow file *header*. If successful, continue to 2.
If there is a hard failure, then return an error. If there is a failure
that may be transient, then sleep and retry at 0.
2 If the tick number in the header is less than the tick last read by the VFD,
then return an error.
3 If the tick number in the header is equal to the last tick read by the
VFD, then exit without doing anything.
4 Try to read the shadow file *index*. If successful, continue to 5.
If there is a hard failure, then return an error. If there is a failure
that may be transient, then sleep and retry at 0.
5 If a different tick number was read from the index than from the index,
then continue at 0.
6 Try to *re-read* the shadow file *header*. If successful, continue to 7.
If there is a hard failure, then return an error. If there is a failure
that may be transient, then sleep and retry at 0.
7 Compare the header that was read previously with the new header. If
the new header is different than the old, then we may not have read
the index at the right shadow-file offset, or the index may have been
read in an inconsistent state, so sleep and retry at 0. Otherwise,
return success.
Simplify H5FD__vfd_swmr_header_deserialize() and
H5FD__vfd_swmr_index_deserialize(). Remove their retry loops. Make
each return TRUE on success, FALSE on an error that may be transient,
and FAIL on an irrecoverable error.
In H5FD__vfd_swmr_header_deserialize(), do not check the size of the
shadow file with fstat(2), since the read(2) will fail if the file is
too small. This saves us a system call.
Lightly consti-ify H5FD__vfd_swmr_index_deserialize() arguments.
In H5FD__vfd_swmr_load_hdr_and_idx():
Consolidate all of the retry-looping. Increase the initial
retry delay from 1ns to 1/10s. Delete the disused maximum-retry
constants.
Use #if 0 to disable some error-checking code that ought to be
unnecessary under the new protocol.
Don't memset() the header and index header, but make sure
they're fully initialized with real content, instead.
|
|
|
|
|
|
|
|
| |
`uint64_t` to `size_t` because it describes the size of an in-core
structure as well as an on-disk one, and `size_t` is wide enough
to store the size of any in-core structure, while `uint64_t` may
be much too wide. Check that `index_length` is no more than SIZE_MAX
after we read it.
|
|
|
|
|
| |
shadow_image_defer_free(), which is also a better description of what the
routine does.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Delete the little-used free-list length, dl_len, and just count up the list
entries when diagnostic code needs the length.
Extract the code for deferring shadow-image free into a new subroutine,
`vfd_swmr_idx_entry_defer_free()`.
Rename type `deferred_free_head_t` as `deferred_free_queue_t`.
Remove the disused H5F__LL_{REMOVE,PREPEND} macros.
Add some diagnostic code and #if 0'd assertions.
Change `qsort(ptr, n, sizeof(type), cmp)` to
`qsort(ptr, n, sizeof(*ptr), cmp)`.
Use a `continue` statement to lower a staircase in
H5F_update_vfd_swmr_metadata_file().
Add vfd_swmr_mdf_idx_entry_remove() to delete a shadow index entry and add the
image at that entry to a deferred-free list. Call it whenever a page is
evicted.
Update the comment in H5PB_remove_entry() that asks if we need to remove shadow
index entries: now we *do* remove them. Remove shadow index entries in
H5PB__evict_entry(). Also mention in the comment that the index-entry removal
performed by H5PB__evict_entry() ought to be sufficient.
|
|
|
|
|
|
|
|
| |
including the merge of `hdffv/hdf5/develop`, back to the branch that Vailin and
I share.
Now I need to put this branch on a fork with a less confusing name than
vchoi_fork!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
open in the VFD SWMR reader case.
Note that the following failures in testvfdswmr.sh:
1) Unable to find metadata file on VFD SWMR reader open.
2) Occasional sanity check failures in the page buffer on raw data write.
3) Filter failures on raw data read in VFD SWMR readers when compression
is enabled.
4) Unexpected data errors in VFD SWMR readers when compression is
disabled.
Note that I expect that items 3 & 4 two aspects of the same issues -- the
fact that we don't guarantee that raw data is consistent with metadata.
Item 2) must be addressed, but it is so infrequent that it isn't doesn't
affect the conclusion VFD SWMR seems to work, and thus it can wait until
phase 2.
I am given to understand that Vailin has largely addressed item 1),
and will be checking in her solution to this soon.
Tested on Charis and Jelly.
|
| |
|
|
|
|
|
|
|
|
| |
(1) Assertion failure in the vfd_swmr test
(2) Reader error in the vfd swmr concurrent tests
Also fixes for:
(a) Use H5MV_alloc() to allocate space for md_pages_reserved when creating the metadata file in H5F__vfd_swmr_init()
(b) Remove a multi-page (when vfd_swmr_writer is true) from the page buffer in H5MF_xfree()
|
|
|
|
|
|
| |
This is necessary to allow access to Vailin's recent changes
This version passes non-swmr tests in a serial / debug build on charis.
|
|
|
|
| |
(2) Bug fixes
|
|
|
|
|
|
|
|
|
|
| |
1) Free space manager for the metadata file
2) Delayed free space release linked list
3) H5F_update_vfd_swmr_metadata_file()
3) VFD SWMR driver: read callback
4) Flushing for VFD SWMR
5) Port one concurrent test from swmr test set
6) Bug fixes and refactoring
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
https://bitbucket.hdfgroup.org/scm/~vchoi/my_hdf5_fork into feature/vfd_swmr
Conflicts:
src/H5AC.c
src/H5F.c
src/H5Pfapl.c
Resolve conflicts for H5AC.c, H5F.c and H5Pfapl.c
|
| |
| |
| |
| | |
other misc).
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| | |
1) Define driver for the VFD SWMR reader
2) Implement VFD SWMR open callback
3) Implement H5FD_vfd_swmr_get_tick_and_idx()
4) Load and decode metadata file header and index
4) Closing for VFD SWMR
|
|/
|
|
|
|
|
|
|
|
|
|
| |
1) Public routines: H5Pget/set_vfd_swmr_config
2) Public routine: H5Fvfd_swmr_end_tick
3) Initialization when opening a file with VFD SWMR writer
4) Tests in test/vfd_swmr.c for VFD SWMR
5) Fix a bug in src/H5Fint.c: when error is encountered after the root group is created
Note the following:
--This is WORK IN PROGRESS and will subject to change as implementation goes.
--There is test failure form enc_dec_plist.c: I will fix this when changes to the property list are settled.
--The branch is updated with the latest from develop as of 8/14/2018
|
| |
|
|
|
|
|
|
| |
* Changed return type of H5FD_driver_query() to herr_t.
* Updated swmr.c to use H5FDdriver_query() instead of the private call.
* Added some flags tests to vfd.c.
|
|
|
|
|
|
|
| |
* Moved H5FD_driver_query() to H5FDint.c.
* Minor typo fixes in H5I.c.
* Updated swmr.c to check the VFD feature flags and skip the
retries test when the VFD is not compatible with H5P_DEFAULT.
|
|
|
|
|
|
|
|
|
|
| |
hdf5_1_10
* commit '54957d37f5aa73912763dbb6e308555e863c43f4':
Commit copyright header change for src/H5PLpkg.c which was added after running script to make changes.
Add new files in release_docs to MANIFEST. Cimmit changes to Makefile.in(s) and H5PL.c that resulted from running autogen.sh.
Merge pull request #407 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10_1 to hdf5_1_10_1
Change copyright headers to replace url referring to file to be removed and replace it with new url for COPYING file.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(https://jira.hdfgroup.org/browse/HDFFV-10055).
Briefly, in H5C_collective_write() in H5Cmpio.c,
the metadata cache attempts to perform a collective
write of metadata cache entries.
This worked fine as long as all processes had at
least one entry to write.
However, when the process has no entries, the
function tries to participate in the collective write
by calling MPI_File_set_view(),
MPI_File_write_all() and then MPI_File_set_view()
again, to match the calls in H5FD_mpio_write().
After pull request 183, the CGNS test benchmark_hdf5
started failing. On investigation, I determined that
the failure occurred in the first call to MPI_File_set_view()
in the "no data to write" path through H5C_collective_write().
Note that pull request 183 did not create the problem,
it only exposed it. The bug can be observed after pull
request 182 if one executes the CGNS progam
src/ptests/benchmark_hdf5 with 90 processes.
The problem appears to have been that the calls to
MPI_File_set_view() in H5C_collective_write() and
H5FD_mpio_write() were using different values for the
info parameter. I patched the problem by adding a
MPI specific VFD call allowing me to get the MPI_Info
used in H5FD_mpio_write() for use in
MPI_File_set_view() calls in H5C_collective_write().
Tested serial & parallel, debug & production on
Jelly.
|
| |
|
| |
|
| |
|
|
|
|
| |
and the tool and API wrappers will be coming in over the weekend.
|
| |
|
|
|
|
|
|
|
|
| |
Clean up warnings (from 2774 -> 1560, with my standard debug build)
Tested on:
MacOSX/64 10.11.5 (amazon) w/serial, parallel & production
(h5committest forthcoming)
|
|
|
|
|
|
| |
(H5_DEBUG_BUILD)
tested on bb-8 with Serial and Parallel, debug and production builds.
|
|
|
|
|
|
|
|
|
|
| |
checks from metadata dxpls
- remove H5AC_ind_dxpl_id and use only H5AC_dxpl_id everywhere instead.
- remove flush_me_collectively flag from cache entries
- add a collective sanity check (MPI_Barrier) for every HDF5 API routine that could possibly touch the file. This is trigerred when the environment variable H5_COLL_API_SANITY_CHECK is set to a non 0 digit.
tested on BB-8 with serial and parallel.
|
|
|
|
|
|
|
| |
Tested on:
Ubuntu 15.10 (Linux 4.2.0 x86_64) gcc 5.2.1
serial only
(these changes have been in revise_chunks for a long time)
|
|
|
|
|
|
|
|
|
|
|
| |
Refactor property list code to "deep copy" properties in the correct way,
retraining the rest of the library to copy & release things correctly. This
cleans up another batch of memory leaks, etc. within the library.
Tested on:
MacOSX/64 10.10.5 (amazon) w/serial & parallel
Linux/32 2.6.x (jam) w/serial & parallel
(h5committest forthcoming)
|
|
|
|
|
|
|
|
| |
Clean up H5FD interface, to align w/v3 metadata cache changes
Tested on:
MacOSX/64 10.10.3 (amazon) w/serial & parallel
Linux/32 2.6.* (jam) w/serial & parallel
|
| |
|
|
|
|
|
|
|
|
| |
and replace with driver feature flag(s).
Add an MPI atomicity feature flag and remove the macro from the library.
tested Parallel on Jam.
|
|
|
|
|
|
|
|
|
|
| |
Clean up EOF code within library and add 'mem_type' parameter to 'get_eof'
VFD callback, to avoid various ambiguous situations, particularly with the
multi VFD. (Supports changes for 'avoid_truncate' feature also)
Tested on:
MacOSX/64 10.10.1 (amazon) w/serial & parallel
h5committest forthcoming
|
|
|
|
|
|
|
|
|
|
|
| |
Make progress toward moving from DXPL IDs to property list structures
within the library. Also move the signature location code from the H5F
package to the H5FD package, where it's a better fit. Also, clean up some
more compiler warnings along the way.
Tested on:
Mac OSX/64 10.9.2 (amazon) w/C++, FORTRAN & parallel
(h5committest forthcoming)
|
|
|
|
|
|
|
|
|
| |
Begin process of migrating from using property list IDs internally to the
library to using the internal generic property list data structure.
Tested on:
Mac OSX/64 10.9.2 (amazon) w/C++, FORTRAN & parallel
(h5committest forthcoming)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove all traces of MPI-POSIX VFD and GPFS detection/code.
Remove remaining traces of stream VFD.
Remove testpar/t_posix_compliant test (it's not actually verifying anything).
Clean up H5D__mpio_opt_possible() further.
Moved environment variable that disables MPI collective operations into
MPI-IO VFD (instead of it being in src/H5S.c).
A few other small code cleanups.
Tested on:
Mac OSX/64 10.9.2 (amazon) w/parallel & serial
|
|
|
|
|
|
| |
https://jira.hdfgroup.uiuc.edu/browse/HDFFV-8203
tested: h5committest
|
|
|
|
|
|
|
|
| |
Clean up file driver property handling, along with some misc. warnings.
Tested on:
Mac OSX/64 10.7.4 (amazon) w/debug, gcc 4.7.1, C++, FORTRAN & threadsafe
(h5committest upcoming)
|
|
|
|
|
|
|
|
| |
Merge "file image" changes from feature branch back to trunk.
Tested on:
Mac OSX/64 10.7.3 (amazon) w/debug
(h5committest upcoming)
|
|
|
|
|
|
|
| |
opened with the MPI-IO VFD
Add test cases for these two routines
Jira issue HDFFV-7961
|
|
|
|
|
|
|
|
|
|
|
| |
Bring sec2 and log VFDs back into agreement. Correct several compiler
warnings. Also add some additional statistics to the log VFD and promote the
parameter for H5Pset_fapl_log from 'unsigned' to 'unsigned long long'.
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
FreeBSD/32 8.2 (loyalty) w/debug
FreeBSD/64 8.2 (freedom) w/debug
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix a few more problems when the user block is not aligned properly.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.5.8 (amazon) in debug mode
Mac OS X/32 10.5.8 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Switch from using a 'flags' parameter to the flush routine(s), to just
using a single 'closing' boolean parameter, since that's all we're doing with
the flags anyway and this makes things more obvious.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.5.8 (amazon) in debug mode
Mac OS X/32 10.5.8 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
|