summaryrefslogtreecommitdiffstats
path: root/src/H5FDprivate.h
Commit message (Collapse)AuthorAgeFilesLines
* Update some comments.David Young2020-02-261-3/+3
|
* Where n is the number of page-table/shadow-index entries, avoid spending O(n^2)David Young2020-02-261-4/+5
| | | | | | | | time in H5PB_dest(). While we're in H5PB_dest(), mark deleted shadow-index entries as "garbage" and skip the O(n) shadow index-entries copy. Rename shadow index-entry member `moved_to_hdf5_file` to `moved_to_lower_file` while I'm in here---NFCI.
* Make H5FD_vfd_swmr_dump_status() take an unsigned page number to be consistentDavid Young2020-02-061-1/+1
| | | | with other code.
* Numerous changes supporting a floating shadow index:David Young2020-02-031-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Add to the H5F_shared_t (!) a new member that tells the index in the shadow file where the index should be written. Allocate shadow filespace for the header and the index separately so that the index can float. Update tests to match the expected original location of the index. Introduce vfd_swmr_enlarge_shadow_index(), a routine that allocates space in the shadow file for a new index that has (up to) twice as many entries as the old index, allocates a new in-core index of the same size, and copies the old in-core index to the new. Call vfd_swmr_enlarge_shadow_index() in H5PB_vfd_swmr__update_index() when the in-core index has too few slots. In the comment at the top of H5FD__vfd_swmr_load_hdr_and_idx(), describe the protocol that it follows, now, when it reads the shadow header and index. Delete some dead code in the function and add a bit of diagnostic code. TBD quiet the diagnostic code. In H5F_vfd_swmr_init(), follow the protocol: write the index, first, then the header. Modify property-list checks and tests to reserve no fewer than two pages at the front of the shadow file for the header and index.
* Fix typo. NFCI.David Young2020-01-301-1/+1
|
* Prepare the VFD SWMR reader for a "floating" shadow index by overhaulingDavid Young2020-01-241-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the way that the shadow header and shadow index are loaded. In H5FD__vfd_swmr_load_hdr_and_idx(), adopt a new protocol for reading the shadow file: 0 If the maximum number of retries have been attempted, then exit with an error. 1 Try to read the shadow file *header*. If successful, continue to 2. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 2 If the tick number in the header is less than the tick last read by the VFD, then return an error. 3 If the tick number in the header is equal to the last tick read by the VFD, then exit without doing anything. 4 Try to read the shadow file *index*. If successful, continue to 5. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 5 If a different tick number was read from the index than from the index, then continue at 0. 6 Try to *re-read* the shadow file *header*. If successful, continue to 7. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 7 Compare the header that was read previously with the new header. If the new header is different than the old, then we may not have read the index at the right shadow-file offset, or the index may have been read in an inconsistent state, so sleep and retry at 0. Otherwise, return success. Simplify H5FD__vfd_swmr_header_deserialize() and H5FD__vfd_swmr_index_deserialize(). Remove their retry loops. Make each return TRUE on success, FALSE on an error that may be transient, and FAIL on an irrecoverable error. In H5FD__vfd_swmr_header_deserialize(), do not check the size of the shadow file with fstat(2), since the read(2) will fail if the file is too small. This saves us a system call. Lightly consti-ify H5FD__vfd_swmr_index_deserialize() arguments. In H5FD__vfd_swmr_load_hdr_and_idx(): Consolidate all of the retry-looping. Increase the initial retry delay from 1ns to 1/10s. Delete the disused maximum-retry constants. Use #if 0 to disable some error-checking code that ought to be unnecessary under the new protocol. Don't memset() the header and index header, but make sure they're fully initialized with real content, instead.
* Change the blah_blah_blah_md_header `index_length` member fromDavid Young2020-01-101-1/+1
| | | | | | | | `uint64_t` to `size_t` because it describes the size of an in-core structure as well as an on-disk one, and `size_t` is wide enough to store the size of any in-core structure, while `uint64_t` may be much too wide. Check that `index_length` is no more than SIZE_MAX after we read it.
* Change the unwieldy name vfd_swmr_idx_entry_defer_free() toDavid Young2019-12-191-1/+1
| | | | | shadow_image_defer_free(), which is also a better description of what the routine does.
* Use the "tail queue" from queue(3) for the delayed free list of shadow images.David Young2019-12-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Delete the little-used free-list length, dl_len, and just count up the list entries when diagnostic code needs the length. Extract the code for deferring shadow-image free into a new subroutine, `vfd_swmr_idx_entry_defer_free()`. Rename type `deferred_free_head_t` as `deferred_free_queue_t`. Remove the disused H5F__LL_{REMOVE,PREPEND} macros. Add some diagnostic code and #if 0'd assertions. Change `qsort(ptr, n, sizeof(type), cmp)` to `qsort(ptr, n, sizeof(*ptr), cmp)`. Use a `continue` statement to lower a staircase in H5F_update_vfd_swmr_metadata_file(). Add vfd_swmr_mdf_idx_entry_remove() to delete a shadow index entry and add the image at that entry to a deferred-free list. Call it whenever a page is evicted. Update the comment in H5PB_remove_entry() that asks if we need to remove shadow index entries: now we *do* remove them. Remove shadow index entries in H5PB__evict_entry(). Also mention in the comment that the index-entry removal performed by H5PB__evict_entry() ought to be sufficient.
* Merge all of my changes from merge-back-to-feature-vfd_swmr-attempt-1,David Young2019-12-091-11/+33
| | | | | | | | including the merge of `hdffv/hdf5/develop`, back to the branch that Vailin and I share. Now I need to put this branch on a fork with a less confusing name than vchoi_fork!
* Checkin of fixes for a number of bugs in reads to page 0, and in filemainzer2019-01-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | open in the VFD SWMR reader case. Note that the following failures in testvfdswmr.sh: 1) Unable to find metadata file on VFD SWMR reader open. 2) Occasional sanity check failures in the page buffer on raw data write. 3) Filter failures on raw data read in VFD SWMR readers when compression is enabled. 4) Unexpected data errors in VFD SWMR readers when compression is disabled. Note that I expect that items 3 & 4 two aspects of the same issues -- the fact that we don't guarantee that raw data is consistent with metadata. Item 2) must be addressed, but it is so infrequent that it isn't doesn't affect the conclusion VFD SWMR seems to work, and thus it can wait until phase 2. I am given to understand that Vailin has largely addressed item 1), and will be checking in her solution to this soon. Tested on Charis and Jelly.
* Intermin commit to allow update from Vailin's recent changesmainzer2019-01-151-1/+3
|
* Fixes for issues #1 and #2 listed in John's email dated Dec 7 2018:Vailin Choi2018-12-191-0/+1
| | | | | | | | (1) Assertion failure in the vfd_swmr test (2) Reader error in the vfd swmr concurrent tests Also fixes for: (a) Use H5MV_alloc() to allocate space for md_pages_reserved when creating the metadata file in H5F__vfd_swmr_init() (b) Remove a multi-page (when vfd_swmr_writer is true) from the page buffer in H5MF_xfree()
* local commit of first cut at vfd swmr writer EOT code.mainzer2018-11-191-40/+62
| | | | | | This is necessary to allow access to Vailin's recent changes This version passes non-swmr tests in a serial / debug build on charis.
* (1) Add tests for VFD SWMR writerVailin Choi2018-10-221-15/+18
| | | | (2) Bug fixes
* Third batch of checkin:Vailin Choi2018-09-281-20/+46
| | | | | | | | | | 1) Free space manager for the metadata file 2) Delayed free space release linked list 3) H5F_update_vfd_swmr_metadata_file() 3) VFD SWMR driver: read callback 4) Flushing for VFD SWMR 5) Port one concurrent test from swmr test set 6) Bug fixes and refactoring
* Merge branch 'develop' of ↵Vailin Choi2018-09-261-5/+3
|\ | | | | | | | | | | | | | | | | | | https://bitbucket.hdfgroup.org/scm/~vchoi/my_hdf5_fork into feature/vfd_swmr Conflicts: src/H5AC.c src/H5F.c src/H5Pfapl.c Resolve conflicts for H5AC.c, H5F.c and H5Pfapl.c
| * Normalization with vol_integration (property lists, file drivers,Dana Robinson2018-09-211-5/+3
| | | | | | | | other misc).
* | Bug fixes for the previous checkin.Vailin Choi2018-09-041-4/+4
| |
* | Second batch of checkin:Vailin Choi2018-09-041-3/+80
| | | | | | | | | | | | | | | | 1) Define driver for the VFD SWMR reader 2) Implement VFD SWMR open callback 3) Implement H5FD_vfd_swmr_get_tick_and_idx() 4) Load and decode metadata file header and index 4) Closing for VFD SWMR
* | Preliminary checkins for implementation done so far:Vailin Choi2018-08-141-0/+43
|/ | | | | | | | | | | | 1) Public routines: H5Pget/set_vfd_swmr_config 2) Public routine: H5Fvfd_swmr_end_tick 3) Initialization when opening a file with VFD SWMR writer 4) Tests in test/vfd_swmr.c for VFD SWMR 5) Fix a bug in src/H5Fint.c: when error is encountered after the root group is created Note the following: --This is WORK IN PROGRESS and will subject to change as implementation goes. --There is test failure form enc_dec_plist.c: I will fix this when changes to the property list are settled. --The branch is updated with the latest from develop as of 8/14/2018
* Add API context interface and use it throughout the library.Quincey Koziol2018-03-151-36/+10
|
* * Added new public H5FDdriver_query() API call.Dana Robinson2017-05-241-1/+1
| | | | | | * Changed return type of H5FD_driver_query() to herr_t. * Updated swmr.c to use H5FDdriver_query() instead of the private call. * Added some flags tests to vfd.c.
* * Made H5FD_driver_query() a private function instead of static.Dana Robinson2017-05-241-0/+1
| | | | | | | * Moved H5FD_driver_query() to H5FDint.c. * Minor typo fixes in H5I.c. * Updated swmr.c to check the VFD feature flags and skip the retries test when the VFD is not compatible with H5P_DEFAULT.
* Merge pull request #426 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10 to ↵Larry Knox2017-04-251-6/+4
| | | | | | | | | | hdf5_1_10 * commit '54957d37f5aa73912763dbb6e308555e863c43f4': Commit copyright header change for src/H5PLpkg.c which was added after running script to make changes. Add new files in release_docs to MANIFEST. Cimmit changes to Makefile.in(s) and H5PL.c that resulted from running autogen.sh. Merge pull request #407 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10_1 to hdf5_1_10_1 Change copyright headers to replace url referring to file to be removed and replace it with new url for COPYING file.
* Checkin of fix for CGNS bugmainzer2017-04-061-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (https://jira.hdfgroup.org/browse/HDFFV-10055). Briefly, in H5C_collective_write() in H5Cmpio.c, the metadata cache attempts to perform a collective write of metadata cache entries. This worked fine as long as all processes had at least one entry to write. However, when the process has no entries, the function tries to participate in the collective write by calling MPI_File_set_view(), MPI_File_write_all() and then MPI_File_set_view() again, to match the calls in H5FD_mpio_write(). After pull request 183, the CGNS test benchmark_hdf5 started failing. On investigation, I determined that the failure occurred in the first call to MPI_File_set_view() in the "no data to write" path through H5C_collective_write(). Note that pull request 183 did not create the problem, it only exposed it. The bug can be observed after pull request 182 if one executes the CGNS progam src/ptests/benchmark_hdf5 with 90 processes. The problem appears to have been that the calls to MPI_File_set_view() in H5C_collective_write() and H5FD_mpio_write() were using different values for the info parameter. I patched the problem by adding a MPI specific VFD call allowing me to get the MPI_Info used in H5FD_mpio_write() for use in MPI_File_set_view() calls in H5C_collective_write(). Tested serial & parallel, debug & production on Jelly.
* Final merge of page buffering branch to developQuincey Koziol2017-03-141-2/+3
|
* Bring changes to I/O parameters from page_buffering branch.Quincey Koziol2017-03-131-16/+16
|
* Bring file shutdown code from cache image branchQuincey Koziol2017-01-051-1/+1
|
* Bring SWMR support in to the main development branch. (Finally!) More testsQuincey Koziol2016-12-021-0/+1
| | | | and the tool and API wrappers will be coming in over the weekend.
* Warning and whitespace cleanup.Quincey Koziol2016-11-031-3/+15
|
* [svn-r30113] Description:Quincey Koziol2016-06-281-1/+1
| | | | | | | | Clean up warnings (from 2774 -> 1560, with my standard debug build) Tested on: MacOSX/64 10.11.5 (amazon) w/serial, parallel & production (h5committest forthcoming)
* [svn-r29057] added dxpl type checking when debug mode is enabled ↵Mohamad Chaarawi2016-02-071-0/+11
| | | | | | (H5_DEBUG_BUILD) tested on bb-8 with Serial and Parallel, debug and production builds.
* [svn-r28950] - remove META_FLUSH_COLLECTIVELY property for delayed sanity ↵Mohamad Chaarawi2016-01-211-0/+8
| | | | | | | | | | checks from metadata dxpls - remove H5AC_ind_dxpl_id and use only H5AC_dxpl_id everywhere instead. - remove flush_me_collectively flag from cache entries - add a collective sanity check (MPI_Barrier) for every HDF5 API routine that could possibly touch the file. This is trigerred when the environment variable H5_COLL_API_SANITY_CHECK is set to a non 0 digit. tested on BB-8 with serial and parallel.
* [svn-r28626] Brought VFD-level file locking code over from revise_chunks.Dana Robinson2015-12-141-0/+2
| | | | | | | Tested on: Ubuntu 15.10 (Linux 4.2.0 x86_64) gcc 5.2.1 serial only (these changes have been in revise_chunks for a long time)
* [svn-r27811] Description:Quincey Koziol2015-09-161-2/+7
| | | | | | | | | | | Refactor property list code to "deep copy" properties in the correct way, retraining the rest of the library to copy & release things correctly. This cleans up another batch of memory leaks, etc. within the library. Tested on: MacOSX/64 10.10.5 (amazon) w/serial & parallel Linux/32 2.6.x (jam) w/serial & parallel (h5committest forthcoming)
* [svn-r27077] Description:Quincey Koziol2015-05-151-1/+1
| | | | | | | | Clean up H5FD interface, to align w/v3 metadata cache changes Tested on: MacOSX/64 10.10.3 (amazon) w/serial & parallel Linux/32 2.6.* (jam) w/serial & parallel
* [svn-r27009] Fix mismatching prototypes in H5FD/H5B2Jerome Soumagne2015-05-041-1/+1
|
* [svn-r25983] HDFFV-1125: Eliminate IS_H5FD_MPI() macro use in the library ↵Mohamad Chaarawi2015-01-201-4/+0
| | | | | | | | and replace with driver feature flag(s). Add an MPI atomicity feature flag and remove the macro from the library. tested Parallel on Jam.
* [svn-r25929] Description:Quincey Koziol2014-12-291-1/+1
| | | | | | | | | | Clean up EOF code within library and add 'mem_type' parameter to 'get_eof' VFD callback, to avoid various ambiguous situations, particularly with the multi VFD. (Supports changes for 'avoid_truncate' feature also) Tested on: MacOSX/64 10.10.1 (amazon) w/serial & parallel h5committest forthcoming
* [svn-r25097] Description:Quincey Koziol2014-04-251-0/+1
| | | | | | | | | | | Make progress toward moving from DXPL IDs to property list structures within the library. Also move the signature location code from the H5F package to the H5FD package, where it's a better fit. Also, clean up some more compiler warnings along the way. Tested on: Mac OSX/64 10.9.2 (amazon) w/C++, FORTRAN & parallel (h5committest forthcoming)
* [svn-r25084] Description:Quincey Koziol2014-04-231-2/+3
| | | | | | | | | Begin process of migrating from using property list IDs internally to the library to using the internal generic property list data structure. Tested on: Mac OSX/64 10.9.2 (amazon) w/C++, FORTRAN & parallel (h5committest forthcoming)
* [svn-r24864] Description:Quincey Koziol2014-03-211-3/+0
| | | | | | | | | | | | | | | | | | Remove all traces of MPI-POSIX VFD and GPFS detection/code. Remove remaining traces of stream VFD. Remove testpar/t_posix_compliant test (it's not actually verifying anything). Clean up H5D__mpio_opt_possible() further. Moved environment variable that disables MPI collective operations into MPI-IO VFD (instead of it being in src/H5S.c). A few other small code cleanups. Tested on: Mac OSX/64 10.9.2 (amazon) w/parallel & serial
* [svn-r23165] HDFF-8203 Break H5FDmpi.h header into public and private componentsMohamad Chaarawi2013-01-151-0/+41
| | | | | | https://jira.hdfgroup.uiuc.edu/browse/HDFFV-8203 tested: h5committest
* [svn-r22681] Description:Quincey Koziol2012-08-151-2/+0
| | | | | | | | Clean up file driver property handling, along with some misc. warnings. Tested on: Mac OSX/64 10.7.4 (amazon) w/debug, gcc 4.7.1, C++, FORTRAN & threadsafe (h5committest upcoming)
* [svn-r22219] Description:Quincey Koziol2012-03-311-3/+23
| | | | | | | | Merge "file image" changes from feature branch back to trunk. Tested on: Mac OSX/64 10.7.3 (amazon) w/debug (h5committest upcoming)
* [svn-r22115] Add 2 new API routines to set/unset file atomicity for files ↵Mohamad Chaarawi2012-03-211-0/+4
| | | | | | | opened with the MPI-IO VFD Add test cases for these two routines Jira issue HDFFV-7961
* [svn-r20050] Description:Quincey Koziol2011-02-051-2/+29
| | | | | | | | | | | Bring sec2 and log VFDs back into agreement. Correct several compiler warnings. Also add some additional statistics to the log VFD and promote the parameter for H5Pset_fapl_log from 'unsigned' to 'unsigned long long'. Tested on: Mac OS X/32 10.6.6 (amazon) w/debug & production FreeBSD/32 8.2 (loyalty) w/debug FreeBSD/64 8.2 (freedom) w/debug
* [svn-r17461] Description:Quincey Koziol2009-09-101-0/+1
| | | | | | | | | | | | | | | | | | | | | Fix a few more problems when the user block is not aligned properly. Tested on: FreeBSD/32 6.3 (duty) in debug mode FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (jam) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode Mac OS X/32 10.5.8 (amazon) in debug mode Mac OS X/32 10.5.8 (amazon) w/C++ & FORTRAN, w/threadsafe, in production mode
* [svn-r17402] Description:Quincey Koziol2009-08-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | Switch from using a 'flags' parameter to the flush routine(s), to just using a single 'closing' boolean parameter, since that's all we're doing with the flags anyway and this makes things more obvious. Tested on: FreeBSD/32 6.3 (duty) in debug mode FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode Mac OS X/32 10.5.8 (amazon) in debug mode Mac OS X/32 10.5.8 (amazon) w/C++ & FORTRAN, w/threadsafe, in production mode
generated by cgit v0.12 at 2024-09-05 23:59:51 (GMT)