# Welcome to VFD SWMR Thank you for volunteering to test VFD SWMR. SWMR, which stands for Single Writer/Multiple Reader, is a feature of the HDF5 library that lets a process write data to an HDF5 file while one or more processes read the file. Use cases range from monitoring data collection and/or steering experiments in progress to financial applications. The following diagram illustrates how SWMR works.

VFD SWMR is designed to be a more flexible, more modular, better-performing replacement for the existing SWMR feature. * VFD SWMR allows HDF5 objects (groups, datasets, attributes) to be created and destroyed in the course of a reader-writer session. Creating objects is not possible using the existing SWMR feature. * It compartmentalizes much of the SWMR functionality in a virtual-file driver (VFD), thus easing The HDF Group's software-maintenance burden. * And it makes guarantees for the maximum time from write to availability of data for read, provided that the reading and writing systems and their interconnections can keep up with the data flow. For details on how VFD SWMR is implemented, see [TBD: LINK to RFC]. # Quick start Follow these instructions to download, configure, and build the VFD SWMR project in a jiffy. Then install the HDF5 library and utilites built by the VFD SWMR project. ## Download The latest source code here for VFD SWMR is found on the `multi` branch of [the VFD SWMR repository](https://bitbucket.hdfgroup.org/scm/~dyoung/vchoi_fork.git). Clone the repository in a new directory, then switch to the VFD SWMR branch: ``` % git clone https://bitbucket.hdfgroup.org/scm/~dyoung/vchoi_fork.git swmr % cd swmr % git checkout multi ``` ## Build Setup for autotools: ``` % sh ./autogen.sh ``` Create a build directory, change to that directory, and run the configure script: ``` % mkdir -p ../build/swmr % cd ../build/swmr % ../../swmr/configure ``` Build the project: ``` % make ``` ## Test We recommend that you run the full HDF5 test suite to make sure that VFD SWMR works correctly on your system. To test the library, utilities, run ``` % make check ``` If the tests don't pass, please let the developers know! # Sample programs ## Extensible datasets For an example of a program that uses VFD SWMR to write/read many extensible datasets, have a look at `test/vfd_swmr_bigset_writer.c`, the "bigset" test. We compile two binaries from that source file, one that operates in write mode, and a second that operates in read mode. In write mode, "bigset" creates an HDF5 file containing one or more datasets that are extensible in either one dimension or two. Then it runs for several steps, increasing the size of each dataset in each dimension once every step. The dimensions, number of datasets, the step increase in dataset size, and the number of steps are configurable using command-line options -d, -s, -r and -c, and -n, respectively---use the -h option to get a usage message. Each dataset is written with a predictable pattern. In read mode, "bigset" reads each dataset from an HDF5 file created by a "bigset" writer and verifies the patterns. It takes the same command-line parameters as the "bigset" writer. The reader and writer may run concurrently; the reader "polls" the content until it is just shy of complete, given the number of steps expected. To run a bigset test, I open a couple of terminal windows, one for the reader and one for the writer. I change to the `test` directory under my build directory, and I run the writer in one window: ``` % ./vfd_swmr_bigset_writer -n 50 -d 2 ``` and in the other window, I run the reader: ``` % ./vfd_swmr_bigset_reader -n 50 -d 2 -W ``` The writer will wait for a signal before it quits. You may tap Control-C to make it quit. The reader and writer programs support several command-line options: * `-h`: show program usage * `-W`: stop the program from waiting for a signal before it quits. * `-q`: suppress the progress messages that the programs write to the standard error stream. * `-V`: create a virtual dataset with content in three source datasets in the same HDF5 file---only available when the writer creates a dataset extensible in one dimension (`-d 1`) * `-M`: like `-V`, the writer creates the virtual dataset on three source datasets, but each source dataset is in a different HDF5 file. ## The VFD SWMR demos The VFD SWMR demos are in a [separate repository](https://bitbucket.hdfgroup.org/scm/~dyoung/swmr-demo.git). Before you build the demos, you will need to install the HDF5 library and utilities built from the VFD SWMR branch in your home directory somewhere. In the ./configure step, use the command-line option `--prefix=$HOME/path/for/library` to set the directory you prefer. In the demo Makefiles, update the `H5CC` variable with the path to the `h5cc` installed from the VFD SWMR branch. Then you should be able to `make` and `make clean` the demos. Under `gaussians/`, two programs are built, `wgaussians` and `rgaussians`. If you start both from the same directory in different terminals, you should see the "bouncing 2-D Gaussian distributions" in the `rgaussians` terminal. The creation-deletion (`credel`) demo is also run in two terminals. The two command lines are given in `credel/README.md`. You need to use the `h5ls` installed from the VFD SWMR branch, since only that version has the `--poll` option. # Developer tips ## Configuring VFD SWMR ### File-creation properties To use VFD SWMR, creating your HDF5 file with paged allocation strategy is mandatory. This call enables the paged allocation strategy: ``` ret = H5Pset_file_space_strategy(fcpl, H5F_FSPACE_STRATEGY_PAGE, false, 1); ``` Allocated storage that is smaller than the page size will not overlap a page boundary, and allocated storage that is one page or greater in size will start on a page boundary. VFD SWMR relies on that allocation strategy. ### File-access properties In this section we dissect `vfd_swmr_create_fapl()`, a helper routine in the VFD SWMR tests, to show how to configure your application to use VFD SWMR. ``` hid_t vfd_swmr_create_fapl(bool writer, bool only_meta_pages, bool use_vfd_swmr) { H5F_vfd_swmr_config_t config; hid_t fapl; ``` `h5_fileaccess()` is also a helper routine for the tests. In your program, you can replace the `h5_fileaccess()` call with a call to `H5Pcreate(H5P_FILE_ACCESS)`. ``` /* Create file access property list */ if((fapl = h5_fileaccess()) < 0) { warnx("h5_fileaccess"); return badhid; } ``` VFD SWMR has only been tested with the latest file format. It may malfunction with older formats, we just don't know. We force the latest version here. ``` /* FOR NOW: set to use latest format, the "old" parameter is not used */ if(H5Pset_libver_bounds(fapl, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST) < 0) { warnx("H5Pset_libver_bounds"); return badhid; } /* * Set up to open the file with VFD SWMR configured. */ ``` VFD SWMR relies on metadata reads and writes to go through the page buffer. Note that the default page size is 4096 bytes. This call sets the total page buffer size to 4096 bytes. So we have effectively created a one-page page buffer! That is adequate for testing, but it may not be best for your application. If `only_meta_pages` is true, then the entire page buffer is dedicated to metadata. That's fine for VFD SWMR. *Note well*: when VFD SWMR is enabled, the meta-/raw-data pages proportion set by `H5Pset_page_buffer_size()` does not actually control the pages reserved for raw data. *All* pages are dedicated to buffering metadata. ``` /* Enable page buffering */ if(H5Pset_page_buffer_size(fapl, 4096, only_meta_pages ? 100 : 0, 0) < 0) { warnx("H5Pset_page_buffer_size"); return badhid; } ``` Add VFD SWMR-specific configuration to the file-access property list (`fapl`) using an `H5Pset_vfd_swmr_config()` call. When VFD SWMR is enabled, changes to the HDF5 metadata accumulate in RAM until a configurable unit of time known as a *tick* has passed. At the end of each tick, a snapshot of the metadata at the end of the tick is "published"---that is, made visible to the readers. The length of a *tick* is configurable in units of 100 milliseconds using the `tick_len` parameter. Below, `tick_len` is set to `4` to select a tick length of 400ms. A snapshot does not persist forever, but it expires after a number of ticks, given by the *maximum lag*, has passed. Below, `max_lag` is set to `7` to select a maximum lag of 7 ticks. After a snapshot has expired, the writer may overwrite it. When a reader first enters the API, it starts to use, or "selects," the metadata in the newest snapshot, and on every subsequent API entry, if a tick has passed since the last selection, and if new snapshots are available, then the reader selects the latest. If a reader spends longer than `max_lag - 1` ticks (2400ms with the example configuration) inside the HDF5 API, then its snapshot may expire, resulting in undefined behavior. When a snapshot expires while the reader is using it, we say that the writer has "overrun" the reader. The writer cannot currently detect overruns. Frequently the reader will detect an overrun and force the program to exit with a diagnostic assertion failure. The application tells VFD SWMR whether or not to configure for reading or writing a file by setting the `writer` parameter to `true` for writing or `false` for reading. VFD SWMR snapshots are stored in a "shadow file" that is shared between writer and readers. On a POSIX system, the shadow file may be placed on any *local* filesystem that the reader and writer share. The `md_file_path` parameter tells where to put the shadow file. The `md_pages_reserved` parameter tells how many pages to reserve at the beginning of the shadow file for the shadow-file header and the shadow index. The header has an entire page to itself. The remaining `md_pages_reserved - 1` pages are reserved for the shadow index. If the index grows larger than its initial allocation, then it will move to a new location in the shadow file, and the initial allocation will be reclaimed. `md_pages_reserved` must be at least 2. The `version` parameter tells what version of VFD SWMR configuration the parameter struct `config` contains. For now, it should be initialized to `H5F__CURR_VFD_SWMR_CONFIG_VERSION`. ``` memset(&config, 0, sizeof(config)); config.version = H5F__CURR_VFD_SWMR_CONFIG_VERSION; config.tick_len = 4; config.max_lag = 7; config.writer = writer; config.md_pages_reserved = 128; HDstrcpy(config.md_file_path, "./my_md_file"); /* Enable VFD SWMR configuration */ if(use_vfd_swmr && H5Pset_vfd_swmr_config(fapl, &config) < 0) { warnx("H5Pset_vfd_swmr_config"); return badhid; } return fapl; } ``` ## Using virtual datasets (VDS) An application may want to use VFD SWMR to create, read, or write a virtual dataset. Unfortunately, VDS does not work properly with VFD SWMR at this time. In this section, we describe some workarounds that can be used with great care to make VDS and VFD SWMR cooperate. A virtual dataset, when it is read or written, will open files on an application's behalf in order to access the source datasets inside. If a virtual dataset resides on file `v.h5`, and one of its source datasets resides on a second file, `s1.h5`, then the virtual dataset will try to open `s1.h5` using the same file-access properties as `v.h5`. Thus, if `v.h5` is open with VFD SWMR with shadow file `v.shadow`, then the virtual dataset will try to open `s1.h5` with the same shadow file, which will fail. Suppose that `v.h5` is *not* open with VFD SWMR, but it was opened with default file-access properties. Then the virtual dataset will open the source dataset on `s1.h5` with default file-access properties, too. This default virtual-dataset behavior is not helpful to the application that wants to use VFD SWMR to read or write source datasets. To use VFD SWMR with VDS, an application should *pre-open* each file using its preferred file-access properties, including independent shadow filenames for each source file. As long as the virtual dataset remains in use, the application should leave each of the pre-opened files open. In this way the library, when it tries to open the source files, will always find them already open and re-use the already-open files with the file-access properties established on first open. ## Pushing HDF5 content to reader visibility With VFD SWMR, ordinarily it should not be necessary to call H5Fflush(). In fact, when VFD SWMR is active, calling H5Fflush() may slow down your program considerably because the call will not return until after `max_lag` ticks have passed. A writer can make its last changes to an HDF5 file visible to all readers immediately using the new call, `H5Fvfd_swmr_end_tick()`. A writer should use `H5Fvfd_swmr_end_tick()` carefully: by calling it more frequently than once a tick, a writer may corrupt a reader's view of the HDF5 file. When VFD SWMR is enabled, raw data is not cached in the page buffer. On each tick, the content of chunk caches and other unwritten raw data is flushed directly to the HDF5 file, so that raw data is always available before the HDF5 structural metadata that describes it. ## Reading up-to-date content The HDF Group (THG) expects that in one class of VFD SWMR application, instruments on a particle accelerator will continuously generate 2-dimensional data frames and add them to HDF5 datasets while an experiment is ongoing. The datasets will be written to an HDF5 file opened in VFD SWMR mode. Experimenters will monitor a real-time display of the datasets while the experiment takes place. A second program, possibly running on a second computer, will generate the display. The second program will open the HDF5 file in VFD SWMR mode, too. THG developed a demonstration program for class of application, and we have some advice based on that experience. The writer typically will increase a dataset's dimensions by a frame, using `H5Dset_extent()`, before it writes the data of that frame with `H5Dwrite()`. It's possible that a snapshot of the HDF5 file will propagate to the reader between the `H5Dset_extent()` call and the `H5Dwrite()`. Values `H5Dread()` from the last frame at that juncture will not reflect the actual experimental data. Instead, the reader will see arbitrary values or the fill value. To display those values would be distracting and misleading to the experimenter. On the reader, a strategy for displaying the most current, bonafide application data is to read the dimensions of the frames dataset, `d`, compute the number `n` of full frames contained in `d`, and read the next-to-last frame, `n - 2`. THG uses a variant of this strategy in its `gaussians` demo. On the writer, a strategy for protecting against snapshots between the `H5Dset_extent()` and `H5Dwrite()` calls is to suspend VFD SWMR's clock across both of the calls. The `H5Fvfd_swmr_disable_end_of_tick()` call takes a file identifier and stops new snapshots from being taken on the given file until `H5Fvfd_swmr_enable_end_of_tick()` is called on the same file. # Known issues ## Variable-length data A VFD SWMR reader cannot reliably read back a variable-length dataset written by VFD SWMR. For example, a variable-length string created and written as follows ``` hid_t dset, space, type; char data[] = "content"; type = H5Tcopy(H5T_C_S1); H5Tset_size(type, H5T_VARIABLE); space = H5Screate(H5S_SCALAR); dset = H5Dcreate2(..., "string", type, space, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); H5Dwrite(dset, type, space, space, H5P_DEFAULT, &data); ``` and read back like this, ``` char *data; herr_t ret; ret = H5Dread(..., ..., H5S_ALL, H5S_ALL, H5P_DEFAULT, &data); ``` may produce either an error return from `H5Dread` (`ret < 0`) or a `NULL` pointer (`data == NULL`). Planned improvements to the HDF5 *global heap* may alleviate this problem. There is no schedule for those improvements. Improvements to VFD SWMR may also alleviate the problem. ## Microsoft Windows VFD SWMR is not officially supported on Microsoft Windows at this time. The feature should in theory work on Windows and NTFS, however it has not been tested as the existing VFD SWMR tests rely on shell scripts. Note that Windows file shares are not supported as there is no write ordering guarantee (as with NFS, et al.). ## Supported filesystems A VFD SWMR writer and readers share a couple of files, the HDF5 (`.h5`) file and the shadow file. VFD SWMR relies on writes to the files to take effect in the order described in the POSIX documentation for `read(2)` and `write(2)` system calls. If the VFD SWMR readers and the writer run on the same POSIX host, this ordering should take effect, regardless of the underlying filesystem. If the VFD SWMR reader and the writer run on *different* hosts, then the write-ordering rules depend on the shared filesystem. VFD SWMR is not generally expected to work with NFS at this time. GPFS is reputed to order writes according to POSIX convention, so we expect VFD SWMR to work with GPFS. (Caveat: we are still looking for an authoritative description of GPFS I/O semantics.) The HDF Group plans to add support for NFS to VFD SWMR in the future. ## File-opening order If an application tries to open a file in VFD SWMR reader mode, and the file is not already open by a VFD SWMR writer, then the application will sleep in the `H5Fopen()` call until either the writer opens the same file (using the same shadow file) or the reader times out after several seconds. # Reporting bugs VFD SWMR is still under construction, so I think that you will find some bugs. Please do not hesitate to report them. To contact the VFD SWMR developers, email vfdswmr@hdfgroup.org.