summaryrefslogtreecommitdiffstats
path: root/src/H5FDvfd_swmr.c
Commit message (Collapse)AuthorAgeFilesLines
...
* On the reader, zero the shadow-index entries before filling fields from theDavid Young2020-02-261-1/+1
| | | | | shadow file so that members like `garbage` are not filled with garbage that fools us, later.
* Where n is the number of page-table/shadow-index entries, avoid spending O(n^2)David Young2020-02-261-1/+2
| | | | | | | | time in H5PB_dest(). While we're in H5PB_dest(), mark deleted shadow-index entries as "garbage" and skip the O(n) shadow index-entries copy. Rename shadow index-entry member `moved_to_hdf5_file` to `moved_to_lower_file` while I'm in here---NFCI.
* Delete misleading "Programmer:" field and excessive decoration from theDavid Young2020-02-121-22/+17
| | | | | H5FD_vfd_swmr_write() comment header. Delete a superfluous comment. Delete some whitespace at EOL. NFCI.
* Use an early exit to lower a staircase.David Young2020-02-061-132/+130
|
* Reduce code duplication by using vfd_swmr_pageno_to_mdf_idx_entry() to look upDavid Young2020-02-061-98/+61
| | | | | | | | | | | | | shadow pages. Reduce casts by choosing correct format strings and compatible variable types. Poison writes to addr by making it const. Don't increase addr in the read(2) loop because it's never used afterward. Delete some more dead code. Rename read_ptr as p and declare it much closer to its use. Change its type to `char *` so that no casts are necessary to increase it.
* Delete unused code.David Young2020-02-061-36/+0
|
* Fix a typo in a diagnostic printf and, while I'm here, use the right formatDavid Young2020-02-061-6/+6
| | | | strings.
* Use proper format string.David Young2020-02-061-2/+4
|
* Make H5FD_vfd_swmr_dump_status() take an unsigned page number to be consistentDavid Young2020-02-061-2/+2
| | | | with other code.
* Numerous changes supporting a floating shadow index:David Young2020-02-031-48/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | Add to the H5F_shared_t (!) a new member that tells the index in the shadow file where the index should be written. Allocate shadow filespace for the header and the index separately so that the index can float. Update tests to match the expected original location of the index. Introduce vfd_swmr_enlarge_shadow_index(), a routine that allocates space in the shadow file for a new index that has (up to) twice as many entries as the old index, allocates a new in-core index of the same size, and copies the old in-core index to the new. Call vfd_swmr_enlarge_shadow_index() in H5PB_vfd_swmr__update_index() when the in-core index has too few slots. In the comment at the top of H5FD__vfd_swmr_load_hdr_and_idx(), describe the protocol that it follows, now, when it reads the shadow header and index. Delete some dead code in the function and add a bit of diagnostic code. TBD quiet the diagnostic code. In H5F_vfd_swmr_init(), follow the protocol: write the index, first, then the header. Modify property-list checks and tests to reserve no fewer than two pages at the front of the shadow file for the header and index.
* Remove some dead code and out-of-date comments.David Young2020-01-301-18/+0
|
* Prepare the VFD SWMR reader for a "floating" shadow index by overhaulingDavid Young2020-01-241-218/+200
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the way that the shadow header and shadow index are loaded. In H5FD__vfd_swmr_load_hdr_and_idx(), adopt a new protocol for reading the shadow file: 0 If the maximum number of retries have been attempted, then exit with an error. 1 Try to read the shadow file *header*. If successful, continue to 2. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 2 If the tick number in the header is less than the tick last read by the VFD, then return an error. 3 If the tick number in the header is equal to the last tick read by the VFD, then exit without doing anything. 4 Try to read the shadow file *index*. If successful, continue to 5. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 5 If a different tick number was read from the index than from the index, then continue at 0. 6 Try to *re-read* the shadow file *header*. If successful, continue to 7. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 7 Compare the header that was read previously with the new header. If the new header is different than the old, then we may not have read the index at the right shadow-file offset, or the index may have been read in an inconsistent state, so sleep and retry at 0. Otherwise, return success. Simplify H5FD__vfd_swmr_header_deserialize() and H5FD__vfd_swmr_index_deserialize(). Remove their retry loops. Make each return TRUE on success, FALSE on an error that may be transient, and FAIL on an irrecoverable error. In H5FD__vfd_swmr_header_deserialize(), do not check the size of the shadow file with fstat(2), since the read(2) will fail if the file is too small. This saves us a system call. Lightly consti-ify H5FD__vfd_swmr_index_deserialize() arguments. In H5FD__vfd_swmr_load_hdr_and_idx(): Consolidate all of the retry-looping. Increase the initial retry delay from 1ns to 1/10s. Delete the disused maximum-retry constants. Use #if 0 to disable some error-checking code that ought to be unnecessary under the new protocol. Don't memset() the header and index header, but make sure they're fully initialized with real content, instead.
* Change md_pages_reserved and pb_expansion_threshold parameters to uint32_t fromDavid Young2020-01-161-1/+1
| | | | int32_t.
* Prepare a bit to let the shadow index "float": drasticallyDavid Young2020-01-101-53/+83
| | | | | | | | | | | | | | simplify H5FD__vfd_swmr_index_deserialize(): reuse h5_retry_init()/h5_retry_next() for retry loops. Don't wait for the fstat(2) to read the correct size, because the read(2) will return short if the file isn't long enough. (This change should save at least one system call, always.) Leave a bunch of comments about the changes that I will have to make so that the shadow index will float. NFCI: do not cast H5MM_malloc() return values, this is not C++.
* Simplify nested logic in H5FD_vfd_swmr_get_tick_and_idx().David Young2020-01-101-9/+12
| | | | | assert() our expectation that either a NULL index was passed, or a pointer to the index length was passed in, too.
* Correct some irregular indentation, delete an unused line, split anDavid Young2020-01-101-12/+10
| | | | | | | `if ((p = allocate(...)) == NULL) { }` into two statements, `p = allocate(...); if (p == NULL) { }`, put a semicolon at the end of an HGOTO_ERROR(), remove comments /* end if */, /* end for */ after closing curly braces.
* Make H5FD__vfd_swmr_load_hdr_and_idx() simpler and faster: copy aDavid Young2020-01-101-56/+30
| | | | | | pointer to the metadata index instead of copying the index itself. Use struct assignment instead of copying individual struct members. Lower a staircase.
* Clean up the code a bit: remove some unnecessary casts and such. NFCI.David Young2020-01-101-4/+4
|
* Change the blah_blah_blah_md_header `index_length` member fromDavid Young2020-01-101-1/+7
| | | | | | | | `uint64_t` to `size_t` because it describes the size of an in-core structure as well as an on-disk one, and `size_t` is wide enough to store the size of any in-core structure, while `uint64_t` may be much too wide. Check that `index_length` is no more than SIZE_MAX after we read it.
* Merge all of my changes from merge-back-to-feature-vfd_swmr-attempt-1,David Young2019-12-091-78/+73
| | | | | | | | including the merge of `hdffv/hdf5/develop`, back to the branch that Vailin and I share. Now I need to put this branch on a fork with a less confusing name than vchoi_fork!
* Modifications for the following items in the punch list:Vailin Choi2019-11-141-1/+1
| | | | | | | | | (A) #5: Add the "pb_expansion_threshold" field to the "H5F_vfd_swmr_config_t" structure and update H5Pset_vfd_swmr_config() and H5Pget_vfd_swmr_config() accordingly (B) #13 bullet 2: Comment H5F_vfd_swmr_config_t in H5Fpublic.h properly (copied from John's description in the RFC) (C) Change the field name "vfd_swmr_writer" to "writer" in "struct H5F_vfd_swmr_config_t" (as indicated on page 11 in the RFC) and all references to it
* Good-bye md_open_tries.David Young2019-11-141-1/+1
|
* Use h5_retry_init/_next to retry loading the SWMR shadow file.David Young2019-09-121-10/+9
|
* Checkin of fixes for a number of bugs in reads to page 0, and in filemainzer2019-01-301-10/+142
| | | | | | | | | | | | | | | | | | | | | | | | | | | | open in the VFD SWMR reader case. Note that the following failures in testvfdswmr.sh: 1) Unable to find metadata file on VFD SWMR reader open. 2) Occasional sanity check failures in the page buffer on raw data write. 3) Filter failures on raw data read in VFD SWMR readers when compression is enabled. 4) Unexpected data errors in VFD SWMR readers when compression is disabled. Note that I expect that items 3 & 4 two aspects of the same issues -- the fact that we don't guarantee that raw data is consistent with metadata. Item 2) must be addressed, but it is so infrequent that it isn't doesn't affect the conclusion VFD SWMR seems to work, and thus it can wait until phase 2. I am given to understand that Vailin has largely addressed item 1), and will be checking in her solution to this soon. Tested on Charis and Jelly.
* Intermin commit to allow update from Vailin's recent changesmainzer2019-01-151-101/+389
|
* Interim checkin to allow Vailin to address assertion failure inmainzer2019-01-031-71/+215
| | | | | | | | | | | | | | | | the memory manager -- details shown below. Note that there are other issues as well -- this is not a working version. [mainzer@jelly test]$ ./vfd_swmr Testing Configure VFD SWMR with fapl PASSED Testing VFD SWMR configuration for the file and fapl PASSED Testing H5Fvfd_swmr_end_tick() for VFD SWMR PASSED Testing Create/Open/Flush an HDF5 file for VFD SWMR PASSED Testing Verify the metadata file for VFD SWMR writer vfd_swmr: H5MVsection.c:233: H5MV__sect_can_merge: Assertion `((sect1->sect_info.addr)!=((haddr_t)(long)(-1)) && (sect2->sect_info.addr)!=((haddr_t)(long)(-1)) && (sect1->sect_info.addr)<(sect2->sect_info.addr))' failed. Abort (core dumped) [mainzer@jelly test]$
* Fixes for issues #1 and #2 listed in John's email dated Dec 7 2018:Vailin Choi2018-12-191-1/+1
| | | | | | | | (1) Assertion failure in the vfd_swmr test (2) Reader error in the vfd swmr concurrent tests Also fixes for: (a) Use H5MV_alloc() to allocate space for md_pages_reserved when creating the metadata file in H5F__vfd_swmr_init() (b) Remove a multi-page (when vfd_swmr_writer is true) from the page buffer in H5MF_xfree()
* (A) Fixes for the assertion failures described in issue #2 (see John ↵Vailin Choi2018-11-141-1/+1
| | | | | | | | | | | | Mainzer's last checkin message): --src/H5PB.c: checks for size >= page size --src/H5MF.c: disable/enable page buffering in H5MF_tidy_self_referential_fsm_hack() --src/H5MFsection.c: call H5PB_remove_entry() for both raw/metadata pages in H5MF__sect_small_merge() (B) Port and modify existing concurrent swmr tests to VFD SWMR. Also modify the following: --remove flushes from VFD SWMR writer tests --set Nreaders to 0 in test/testvfdswmr.sh.in to test for writers only Please enter the commit message for your changes. Lines starting
* 1) Add concurrent test for VFD SWMR readerVailin Choi2018-10-291-11/+29
| | | | 2) Bug fixes in H5FDvfd_swmr.c
* (1) Add tests for VFD SWMR writerVailin Choi2018-10-221-42/+93
| | | | (2) Bug fixes
* Third batch of checkin:Vailin Choi2018-09-281-166/+149
| | | | | | | | | | 1) Free space manager for the metadata file 2) Delayed free space release linked list 3) H5F_update_vfd_swmr_metadata_file() 3) VFD SWMR driver: read callback 4) Flushing for VFD SWMR 5) Port one concurrent test from swmr test set 6) Bug fixes and refactoring
* Bug fixes for the previous checkin.Vailin Choi2018-09-041-79/+119
|
* Second batch of checkin:Vailin Choi2018-09-041-0/+1099
1) Define driver for the VFD SWMR reader 2) Implement VFD SWMR open callback 3) Implement H5FD_vfd_swmr_get_tick_and_idx() 4) Load and decode metadata file header and index 4) Closing for VFD SWMR
generated by cgit v0.12 at 2024-07-19 23:22:15 (GMT)