summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorJohn Mainzer <mainzer@hdfgroup.org>2020-09-23 21:12:57 (GMT)
committerJohn Mainzer <mainzer@hdfgroup.org>2020-09-23 21:12:57 (GMT)
commitac2b0e55b649b95c5893bc90d62e5b77facdc703 (patch)
tree75e8d3cdb34fa61701ce9dd884c0b5102caf5b48
parent1f8482247a69368c9bfbe78d2d181236cf273e4d (diff)
parent1bbec8d69d8c7d3aa46e44b5448733ef396eece1 (diff)
downloadhdf5-ac2b0e55b649b95c5893bc90d62e5b77facdc703.zip
hdf5-ac2b0e55b649b95c5893bc90d62e5b77facdc703.tar.gz
hdf5-ac2b0e55b649b95c5893bc90d62e5b77facdc703.tar.bz2
Merge pull request #2897 in HDFFV/hdf5 from ~MAINZER/vfd_swmr_alpha_1:feature/vfd_swmr_alpha_1 to feature/vfd_swmr_alpha_1vfd_swmr_alpha_1
* commit '1bbec8d69d8c7d3aa46e44b5448733ef396eece1': Deleted extraneous files from doc, and updated the contact email address in doc/vfd-swmr-user-guide.phg to swmr@forum.hdfgroup.org
-rw-r--r--doc/SWMR_Example.pngbin81265 -> 0 bytes
-rw-r--r--doc/VFD_SWMR_Punch_List.md192
-rw-r--r--doc/VFD_SWMR_RFC_2020-02-03.docxbin138626 -> 0 bytes
-rw-r--r--doc/vfd-swmr-user-guide.md2
4 files changed, 1 insertions, 193 deletions
diff --git a/doc/SWMR_Example.png b/doc/SWMR_Example.png
deleted file mode 100644
index e35624c..0000000
--- a/doc/SWMR_Example.png
+++ /dev/null
Binary files differ
diff --git a/doc/VFD_SWMR_Punch_List.md b/doc/VFD_SWMR_Punch_List.md
deleted file mode 100644
index 5674116..0000000
--- a/doc/VFD_SWMR_Punch_List.md
+++ /dev/null
@@ -1,192 +0,0 @@
-VFD SWMR Punch List
-
-<span style="background:green">This text should be on a green background </span>
-
-This document lives at [https://bitbucket.hdfgroup.org/users/dyoung/repos/vchoi_fork/browse/doc/VFD_SWMR_Punch_List.md?at=refs%2Fheads%2Ffeature%2Fvfd_swmr](https://bitbucket.hdfgroup.org/users/dyoung/repos/vchoi_fork/browse/doc/VFD_SWMR_Punch_List.md?at=refs%2Fheads%2Ffeature%2Fvfd_swmr)
-
-An [interactive tutorial](https://www.markdowntutorial.com/) on Markdown
-covers the basics quickly and effectively.
-
-I know I have used this [Markdown "cheat
-sheet"](https://commonmark.org/help/) before---it provides a nice visual
-guide to the syntax.
-
-22 November 2019
-
-1. Design optimizations in index writes. Perhaps write deltas only,
- with a full index every n ticks?
-
-2. Design work for NFS and object store versions.
-
-3. Design work for journaling variation.
-
-4. **Vailin, complete** Add support for opening multiple files in
- either VFD SWMR writer or VFD SWMR writer mode. See addition of
- the EOT queue in section 3.2.2 of the RFC, and related changes in
- sections 3.3 and 3.3.2.
-
-5. **Vailin, complete** Add the pb\_expansion\_threshold field
- to the H5F\_vfd\_swmr\_config\_t structure, and update
- H5Pset\_vfd\_swmr\_config() and H5Pget\_vfd\_swmr\_config()
- accordingly.
-
-6. **Vailin, in progress** Implement end tick now API call. See section
- 3.1.2 of the RFC for specifications.
-
-7. **Vailin, in progress** Implement enable / disable EOT call. See
- section 3.1.3 of the RFC for specifications.
-
-8. Implement option to flush raw data as part of EOT -- write
- performance test to characterize the performance hit.
-
- Currently, the raw data is always flushed at EOT. We need a knob
- that turns on and off the raw-data flush.
-
-9. **Vailin, complete** Move VFD SWMR specific H5F code to its own file
- -- say H5Fvfd\_swmr.c.
-
-10. Update the new page buffer to support the pb\_expansion\_threshold,
- and trigger an early end of tick if the threshold is exceeded.
-
-11. **David, complete** Modify metadata file write call to allow the
- location of the index to float and thus be of arbitrary size.
-
-12. **David, complete** Add code to remove entries from the index after
- they have been written to the HDF5 file, and have not been modified
- for at least max\_lag ticks.
-
-13. Tidy short cuts in the initial implementation. These include:
-
- - **Vailin complete** Odd behavior in the superblock refresh
- routine (see comments in code). Figure out what is going on, and
- then either bypass the issue or fix it as seems appropriate.
-
- - **Vailin complete** Comment H5F\_vfd\_swmr\_config\_t in
- H5Fpublic.h properly.
-
- - Cleanup EOA hack in H5FD\_read().
-
- - **David, complete** Address file open failures in SWMR
- tests. These appear to be cases in which the writer finishes
- before the reader opens -- causing the reader to fail as
- the metadata file no longer exists. If so, handle this more
- gracefully.
-
-14. Add support for specifying the VFD that sits under the VFD SWMR
- reader VFD. Probably do this as part of the pluggable VFD design
- that Jake is working on.
-
-15. Modify the metadata cache so that we don't allocate space for the
- page hash table unless the file is opened in VFD SWMR reader mode.
-
-16. Implement the logging facility (section 3.14 RFC)
-
-17. **David, probably fixed by various changes since September 2019**
- Test code to expose existing page buffer bugs and fix same. Note
- that they seem to appear primarily on Jelly.
-
- - DLL pre remove sanity check assertion failure
-
- - vfd\_swmr\_addrem\_writer: H5PB.c:2981:
- H5PB\_\_mark\_entry\_dirty: Assertion
- '(entry\_ptr)-\>delay\_write\_until \> (pb\_ptr)-\>cur\_tick'
- failed.
-
-18. Flesh out designs for unit, integration and performance tests suites
- as outlined in the RFC. Implement same.
-
-19. **David, complete** Fix memory leak in sparse-reader test.
-
- David found that the shadow index was leaked by VFD SWMR readers and
- plugged the leak. Now the sparse reader tests do not use up all of
- the memory on `jelly`.
-
-20. **David, complete** Test John's patch that repairs the superblock
- flags mismatch that crashes the reader.
-
- David found that the patch fixed the demo crashes.
-
-21. Investigate a potential time-of-check, time-of-use race condition
- involving EOA/EOF and the skip\_read variable in some of the H5PB
- routines.
-
- This race condition may only affect raw-data access.
-
- It may be necessary to poll the superblock for the current EOA. The
- reader's refresh routine for the superblock should propagate the new
- EOA to the VFDs.
-
-22. Understand use of H5F\_t on branch feature/vfd\_swmr instead of
- H5C\_t as on develop branch.
-
-23. *Temporarily* reserve a new superblock flag for VFD SWMR for
- development purposes.
-
-24. Prior to feature merge, *permanently* reserve a new superblock flag.
-
-25. Revisit the global heap. The global heap receives data types (a form
- of metadata) and variable-length (VL) data such as strings. It is
- translated to raw data on its way down the HDF5 software stack. If
- the global heap is modified/replaced as currently planned, then VFD
- SWMR does not have to deal with it. However, if the global heap
- overhaul does not take place, then we have more work to do.
-
-26. **David, complete** Fix the expand/shrink test.
-
- The test appeared to fail because the dataset extent was enlarged
- before the data was written, so arbitrary data was present. If a
- tick snuck in between the `H5Dset_extent` and the `H5Dwrite`, then
- the reader read the arbitrary data.
-
- In the `gaussian` test, I have a heuristic that avoids reading
- arbitrary data. Replicating that in the expand/shrink test has
- fixed it. Essentially, the reader trails the writer by a bit.
-
- Ultimately, we should suspend ticks over the H5Dset_extent/H5Dwrite.
-
-27. **Vailin, complete** Change the field name "vfd\_swmr\_writer" to
- "writer" in "struct H5F_vfd_swmr_config_t" and all references to it.
- See page 11 in the RFC.
-
-28. **Vailin, complete** Fix bug as stated on page 9 in the RFC section
- 3.1.1: Given that the VFD SWMR configuration FAPL property is
- set, the writer field must be consistent with the flags passed in
- the H5Fopen() (either H5F_ACC_RDWR for the VFD SWMR writer, or
- H5F_ACC_RDONLY for the VFD SWMR readers).
-
-29. **Vailin, complete** Fix the FSM bug when setting the FSM
- threshold to a non-default value. The problem was manifested
- when setting the threshold value in vfd_swmr_generator.c via
- H5Pset_file_space_strategy(fcpl, H5F_FSPACE_STRATEGY_PAGE, FALSE,
- 1024 * 1024 * 1024) and then run the vfd_swmr_addrem_writer.c.
-
-30. **David, merged, needs unit test** Add a delay to the FSM so that
- freed regions are not reused before max\_lag ticks are up.
-
-31. Create a large dataset with a small chunk size to verify that page
- management in the extensible array is handled properly by VFD SWMR.
-
-32. Make a fixed-size dataset with a small chunk size to verify ... see
- #31.
-
-33. Make automated tests out of the `credel` and `gaussians` tests.
-
-34. Conserve space in the shadow file. In
- vfd_swmr_enlarge_shadow_index(), we postpone reclamation of the old
- shadow index for max\_lag ticks, but it's not necessary to wait that
- long. David traded some space efficiency for simplicity, for now.
- See the note in the source code.
-
-35. In the documentation, we should note that if a writer deletes an
- object that the reader has open (possesses an hid_t), and max_lag
- ticks later the reader tries to access the object by hid_t, the
- reader gets an error result, or crashes, or accesses the wrong
- object.
-
-36. Make variable-length data work. It cannot work in a simple and
- reliable fashion with VFD SWMR while structural metadata is mixed
- with user data and stored in raw data pages. For a simple example,
- see the variable-length string test, `vfd_swmr_vlstr`.
-
-37. Let a writer start writing in VFD SWMR mode after the reader has started
- reading in VFD SWMR mode. Let the reader start reading right away.
diff --git a/doc/VFD_SWMR_RFC_2020-02-03.docx b/doc/VFD_SWMR_RFC_2020-02-03.docx
deleted file mode 100644
index ef9a9db..0000000
--- a/doc/VFD_SWMR_RFC_2020-02-03.docx
+++ /dev/null
Binary files differ
diff --git a/doc/vfd-swmr-user-guide.md b/doc/vfd-swmr-user-guide.md
index 6a14487..d0d3e06 100644
--- a/doc/vfd-swmr-user-guide.md
+++ b/doc/vfd-swmr-user-guide.md
@@ -568,4 +568,4 @@ VFD SWMR is still under development, so we expect that you will encounter
bugs. Please report them, along with any performance or design issues you
encounter.
-To contact the VFD SWMR developers, email vfdswmr@hdfgroup.org.
+To contact the VFD SWMR developers, email swmr@forum.hdfgroup.org.