summaryrefslogtreecommitdiffstats
path: root/src/H5FDprivate.h
Commit message (Collapse)AuthorAgeFilesLines
* 1) Core changes for adding NFS/updater support as described in the RFC.vchoi2021-11-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | src/H5Pfapl.c src/H5Fvfd_swmr.c src/H5Fpublic.h src/H5Fpkg.h src/H5Fprivate.h 2) For VFD SWMR testing, add private property for checksum generation of metadata files: src/H5Fint.c src/H5Fvfd_swmr.c src/H5Pfapl.c src/H5Fpkg.h src/H5Fprivate.h 3) Fix the following in H5F_vfd_swmr_init() and H5F_vfd_swmr_close_or_flush(): (a) Allocate metadata file index right after metadata file header. (b) Set tick number to 0 when creating header and index for file open case. (c) Remove tick number increment at file close. src/H5Fvfd_swmr.c src/H5Ftest.c 4) To be consistent with the RFC, change the name for field "chksum" to "checksum" in struct H5FD_vfd_swmr_idx_entry_t: src/H5FDprivate.h src/H5FDtest.c src/H5FDvfd_swmr.c src/H5Ftest.c src/H5PB.c 4) Add tests for NFS/updater test/vfd_swmr.c 5) Modify common routine init_vfd_swmr_config() to accept updater_file_path test/vfd_swmr_common.c test/vfd_swmr_common.h 6) Changes to the tests due to the common routine init_vfd_swmr_config(): test/vfd_swmr_addrem_writer.c test/vfd_swmr_attrdset_writer.c test/vfd_swmr_bigset_writer.c test/vfd_swmr_dsetchks_writer.c test/vfd_swmr_dsetops_writer.c test/vfd_swmr_generator.c test/vfd_swmr_gfail_writer.c test/vfd_swmr_gperf_writer.c test/vfd_swmr_group_writer.c test/vfd_swmr_reader.c test/vfd_swmr_remove_reader.c test/vfd_swmr_remove_writer.c test/vfd_swmr_sparse_reader.c test/vfd_swmr_sparse_writer.c test/vfd_swmr_vlstr_reader.c test/vfd_swmr_vlstr_writer.c test/vfd_swmr_writer.c test/page_buffer.c
* VFD SWMR: Removes dedup callback + cleanup (#842)Dana Robinson2021-07-171-0/+1
| | | | | | | | | | | * HDF5-ification of dedup calls * Removes dedup callback from VFD struct * VFD SWMR library code cleanup * Committing clang-format changes Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* VFD SWMR: Normalization with develop (#765)Dana Robinson2021-06-171-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Brings CMake updates from develop * Brings reduction in pedantic casts from develop * Purges UFAIL from the library (#637) * Committing clang-format changes * Purges UFAIL from the library * H5HL_insert change requested in PR Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Removes gratuitous (double)x.yF casts (#632) * Committing clang-format changes * Removes gratuitous (double)x.yF casts * Committing clang-format changes Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Committing clang-format changes * Cleans up a const warning left over from previous constification (#633) * Committing clang-format changes * Adds consts to a few global variables * Cleans up a const warning left over from previous constification Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Formatted source * Bring over some VOL registration changes * VOL cleanup * H5VL_request_status_t substitutions * H5F.c cleanup * Minor API tweaks from develop * Moves H5G package init/teardown to H5Gint.c * H5G cleanup * H5M cleanup * H5SM cleanup * H5T cleanup * H5R cleanup * H5Lpublic.h cleanup * H5L cleanup * H5O cleanup * H5A, H5CS, and H5AC cleanup * Moved H5A init/teardown code to H5Aint.c * Moves H5D I/O functions to H5D.c * H5D cleanup * Misc minor cleanup * H5P close cleanup * H5Tpublic.h cleanup * Fixes err_compat test * H5PLpublic.h cleanup * Updates H5Ppublic.h * H5Fpublic.h updates * H5A.c cleanup * Brings over H5Aexists and related changes * Brings CMake shell testing changes from develop * Close callback changes * H5R and H5Tcommit normalization * err_compat test works now * H5O tweaks * Updates VOL registration code * Brings over H5VL_create_object * H5Tconv.c reformatting * H5T.c tweaks * Brings datatype and reference updates from develop * Brings VOL plugin loading changes from develop * Brings event sets from develop * Brings async functions over * Tools changes * Brings over many tools changes from develop * Brings VOL flags from develop * Fixes h5dump double/float tests * Updates h5repack tests * Brings h5diff test changes from develop * Last h5dump changes * Brings test changes from develop * Committing clang-format changes * Tidied h5_testing() * Brings chunk iteration code + misc from develop * Updates vds test * Enables external link parallel test * Brings updated property lists from develop * H5G changes from develop * H5MF cleanup * Brings vfd_swmr test back into CMake * Updates threadsafe test * Updates plist test * Brings recent changes from develop * Require a C++11 compiler to build the C++ wrappers (#693) * Committing clang-format changes * Requires a C++11 compiler to build the C++ wrappers Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Fix H5FD_mpio_Debug="rw" can report MPI_count instead of bytes (#699) * Removes pre-C99 build and header cruft (#700) * Assumes ANSI C headers exist * Assumes stdbool.h, stdint.h, and inttypes.h are present * Assumes the C++ compiler can handle stdint.h * Removes all work-arounds for missing functionality, especially stdbool.h Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Updates release documents * Brings parallel changes from develop * Adds a VFD/parallel check program for the VFD SWMR shell test * Updates MANIFEST * Disable building VFD SWMR acceptance tests on CMake * Removes err.h from genall5.c * Partial cleanup of H5Cpkg.h * Fixes printf specifiers in H5Cdbg.c * Java and fortran cmake changes (#694) * OESS-98 convert plugin option to FetchContent, add tests * Fixes for pkcfg files because of plugin option * OESS-98 fix tools test for plugins * Keep doxygen comments under 100 chars long - format hint * Whitespace * HDFFV-11144 - Reclassify CMake messages * HDFFV-11099/11100 added help text * Reworked switch statement to compare string instead * Fix typo * Update CDash mode * Correct name of threadsafe * Correct option name * Undo accidental commit * Note LLVM 10 to 11 format default changes * Update format plugin * Undo clang-format version 11 changes * One more correction * Update supported platforms * Revert whitespace changes * Correct whitespace * Changes from PR#3 * HDFFV-11213 added option to control gcc10 warnings diagnostics * HDFFV-11212 Use the new references correctly in JNI utility and tests * format source * Fix typo * Add new test file * HDFFV-11212 - update test and remove unused arg * Minor non-space formatting changes * Use H5I_INVALID_ID instead of "-1" * source formatting * add missing testfile, update jni function * Undo commit of debug code * remove mislocated file * Fix h5repack test for handling of fapls and id close * Update h5diff test files usage text * HDFFV-11212 add new ref tests for JNI export dataset * src format update * Remove blank line typo * src format typo * long double requires %Lg * Another long double foramt specifer S.B. %Lg * issue with t128bit test * Windows issue with h5dump and type. * Fix review issues * refactor function nesting and fix error checks * format fixes * Remove untested functions and javadoc quiet comments * Restore TRY block. * Change string append errors to memory exception * revert to H5_JNI_FATAL_ERROR - support functions need work * Add assertion error for h5util functions * remove duplicate function * format fix * Revert HD function error handling * Update copyright comments * GH #386 java folder copyright corrections * Whitespace * GH #359 implement and fix tools 1.6 API usage * remove excessive comments * Flip inits to correct ifdef section * rework ifdef to be simpler * format issue * Reformat ifdef inits * remove static attribute * format compliance * Update names * Revert because logic relies on float not being int * Changes noticed from creating merge of #412 * Double underscore change * Correct compiler version variable used * Remove header guard underscores * Whitespace cleanup * Split format source and commit changes on repo push * remove pre-split setting * Change windows TS to use older VS. * correct window os name * HDFFV-11212 JNI export util and Javadoc * Suggested review changes * Another change found * Committing clang-format changes * HDFFV-11113 long double in tools * HDFFV-11113 add note * Disable long double tests for now * HDFFV-11228 remove arbitrary CMake warning groups. * Make each flag a string * Some Javadoc warning fixes * Updated javadoc fixes * # WARNING: head commit changed in the meantime HDFFV-11229 Fix long double usage in tools and java Changed h5dump and h5ls to just print 128-bit for long double type. Added test and file for dataset and attributes with all three float types. * Committing clang-format changes * HDFFV-11229 Add release note * HDFFV-11229 windows testfile needed * fix typo * Remove non supported message text * HDFFV-11229 - change ldouble test to check both native and general * HDFFV-11229 add second file compare * HDFFV-11229 fix reference file * HDFFV-11229 autotools check two refs * HDFFV-11229 revert back to removal of NATIVE_LDOUBLE in tools output * Committing clang-format changes * Update release note * Update attribute type of ref file * Change source of ninja for macs * try port instead of brew * Recommended is to use brew. * Undo non long double changes * remove unneeded file * Fix perl and doxygen CMake code * Add "option" command for clang options * Rework CMake add_custom to use the BYPRODUCTS argument * Add stamp files to BYPRODUCTS * Only one copy of file exists * Fix custom cmmand depends targets * Fix fortran custom command DEPENDS * Add LD_LIBRARY_PATH to tests * Add custom target for DEPENDS * Add h5detect conditionaly to generated target DEPENDS * Correct DEPENDS targets * Parallel builds need the mpi compiler for pkgconfig scripts. * install only if MPI build * Fortran target depends * Remove incorrect source attribute * doxygen adjustments * doxygen build updates * Correct version * Correct function version - function has been merged to 1.12 * Correct version string for map functions * Cleanup warnings for tools debug builds * TRILAB-227 - fix indexing for h5diff selections * Correct location of pos to index function call * TRILAB-227 Enable test * Quote subset args * Use MATCHES because of AppleClang * if blocks needed for build modes * Update list of DT platforms * VS2019 correctly displays float values * revert VS2019 change * Issue #669 remove version from pkgcfg filename * remove version from h5cc script * Java reference functions updated enabled fortran in cmake gcc action yaml file java reference test changed to correctly test refs jni reference functions that create ids changed to use jni id logging * Correct BYPRODUCTS argument * Correct more genereated files BYPRODUCTS * BYPRODUCTS must have unique locations * Fix typo Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Removes alternative function enter/leave macro scheme (#678) * Committing clang-format changes * Converted BEGIN_FUNC, etc. macros to FUNC_ENTER * Rips out the BEGIN_FUNC, etc. macros * Removes 'end if', etc. comments from H5HL package * Committing clang-format changes * Fixes an H5EA iterate issue * Fixes an issue in the H5FA iterator code * Further cleanup in bin/trace after macro removal * Iterator changes in H5EA and H5FA Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Split H5L.c internal code to H5Lint.c (#691) * Committing clang-format changes * Splits internal parts of H5L.c to H5Lint.c Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Clean up type size checks in configure.ac (#702) * Committing clang-format changes * C++ warning and build fixes (#707) * Committing clang-format changes * C++ build and warning updates * Fixes all warnings on C++ (with gcc 9.3) * Updates CMake and Autotools C++ builds * Undo warning clobber Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Removes some DEC Alpha cruft from H5detect and H5private.h (#708) * Cleans up hsize_t and haddr_t size guessing (#709) * Cleans up definitions of haddr_t and hsize_t + cruft removal * Formatted source * Updates PGI C/C++ configurations (#715) * Updates PGI C/C++ configurations Tweaks some settings and configures pgc++ to use C++11 * Updated copyright information * Adds an Intel C++ configuration file (#716) * Adds an Intel C++ configuration file * Committing clang-format changes Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Updates bin/trace script to correctly wrap H5TRACE macros near clang-format column limit (#719) * Removes clang-format comments from H5O.c call * Fixes bin/trace to correctly wrap lines near the clang-format limit * Removed unused variable from bin/trace * Fixes issue with ccmake that prevents building Fortran (#723) ccmake runs iteratively, and the check_fortran_source_runs macros were clobbering a single output file that did not get updated on further configure iterations * Updates CMake size_t and ssize_t checks for MacOS * Moved a little boilerplate over from develop * Fixes some warnings * Committing clang-format changes * Moves gcc warning macros from H5public.h to H5private.h (#724) * Moves gcc warning suppression macros out of H5public.h * Update RELEASE.txt * Formatted source * Adds __STDC_FORMAT_MACROS before including inttypes.h w/ C++ (#726) * Adds __STDC_FORMAT_MACROS before including inttypes.h w/ C++ Allows C++ library to be built on older versions of gcc. * Committing clang-format changes Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> * Moves H5PacketTable default ctor inside source file (#731) * Brings (most) H5Xdevelop.h headers from develop * Trivial H5F and H5O changes during normalization * Renames buf_ptr back to page_buf to better align with develop * Removed H5HGtrap.c * Brings changes from develop * Normalization (mainly cache) with develop * Fix Direct VFD Testing in CMake (#742) * Fixes C++ warnings when passing hsize_t values to printf in test code (#745) * Organize and align compiler flags (#741) * OESS-98 convert plugin option to FetchContent, add tests * Fixes for pkcfg files because of plugin option * OESS-98 fix tools test for plugins * Keep doxygen comments under 100 chars long - format hint * Whitespace * HDFFV-11144 - Reclassify CMake messages * HDFFV-11099/11100 added help text * Reworked switch statement to compare string instead * Fix typo * Update CDash mode * Correct name of threadsafe * Correct option name * Undo accidental commit * Note LLVM 10 to 11 format default changes * Update format plugin * Undo clang-format version 11 changes * One more correction * Update supported platforms * Revert whitespace changes * Correct whitespace * Changes from PR#3 * HDFFV-11213 added option to control gcc10 warnings diagnostics * HDFFV-11212 Use the new references correctly in JNI utility and tests * format source * Fix typo * Add new test file * HDFFV-11212 - update test and remove unused arg * Minor non-space formatting changes * Use H5I_INVALID_ID instead of "-1" * source formatting * add missing testfile, update jni function * Undo commit of debug code * remove mislocated file * Fix h5repack test for handling of fapls and id close * Update h5diff test files usage text * HDFFV-11212 add new ref tests for JNI export dataset * src format update * Remove blank line typo * src format typo * long double requires %Lg * Another long double foramt specifer S.B. %Lg * issue with t128bit test * Windows issue with h5dump and type. * Fix review issues * refactor function nesting and fix error checks * format fixes * Remove untested functions and javadoc quiet comments * Restore TRY block. * Change string append errors to memory exception * revert to H5_JNI_FATAL_ERROR - support functions need work * Add assertion error for h5util functions * remove duplicate function * format fix * Revert HD function error handling * Update copyright comments * GH #386 java folder copyright corrections * Whitespace * GH #359 implement and fix tools 1.6 API usage * remove excessive comments * Flip inits to correct ifdef section * rework ifdef to be simpler * format issue * Reformat ifdef inits * remove static attribute * format compliance * Update names * Revert because logic relies on float not being int * Changes noticed from creating merge of #412 * Double underscore change * Correct compiler version variable used * Remove header guard underscores * Whitespace cleanup * Split format source and commit changes on repo push * remove pre-split setting * Change windows TS to use older VS. * correct window os name * HDFFV-11212 JNI export util and Javadoc * Suggested review changes * Another change found * Committing clang-format changes * HDFFV-11113 long double in tools * HDFFV-11113 add note * Disable long double tests for now * HDFFV-11228 remove arbitrary CMake warning groups. * Make each flag a string * Some Javadoc warning fixes * Updated javadoc fixes * # WARNING: head commit changed in the meantime HDFFV-11229 Fix long double usage in tools and java Changed h5dump and h5ls to just print 128-bit for long double type. Added test and file for dataset and attributes with all three float types. * Committing clang-format changes * HDFFV-11229 Add release note * HDFFV-11229 windows testfile needed * fix typo * Remove non supported message text * HDFFV-11229 - change ldouble test to check both native and general * HDFFV-11229 add second file compare * HDFFV-11229 fix reference file * HDFFV-11229 autotools check two refs * HDFFV-11229 revert back to removal of NATIVE_LDOUBLE in tools output * Committing clang-format changes * Update release note * Update attribute type of ref file * Change source of ninja for macs * try port instead of brew * Recommended is to use brew. * Undo non long double changes * remove unneeded file * Fix perl and doxygen CMake code * Add "option" command for clang options * Rework CMake add_custom to use the BYPRODUCTS argument * Add stamp files to BYPRODUCTS * Only one copy of file exists * Fix custom cmmand depends targets * Fix fortran custom command DEPENDS * Add LD_LIBRARY_PATH to tests * Add custom target for DEPENDS * Add h5detect conditionaly to generated target DEPENDS * Correct DEPENDS targets * Parallel builds need the mpi compiler for pkgconfig scripts. * install only if MPI build * Fortran target depends * Remove incorrect source attribute * doxygen adjustments * doxygen build updates * Correct version * Correct function version - function has been merged to 1.12 * Correct version string for map functions * Cleanup warnings for tools debug builds * TRILAB-227 - fix indexing for h5diff selections * Correct location of pos to index function call * TRILAB-227 Enable test * Quote subset args * Use MATCHES because of AppleClang * if blocks needed for build modes * Update list of DT platforms * VS2019 correctly displays float values * revert VS2019 change * Issue #669 remove version from pkgcfg filename * remove version from h5cc script * Java reference functions updated enabled fortran in cmake gcc action yaml file java reference test changed to correctly test refs jni reference functions that create ids changed to use jni id logging * Correct BYPRODUCTS argument * Correct more genereated files BYPRODUCTS * BYPRODUCTS must have unique locations * Fix typo * Fix fortran configure checks * Rework H5_PAC_C_MAX_REAL_PRECISION setting logic * Add note about fortran configure change * Adds a quick for for some egregious chunk_info badness (#722) * Fixes issue with ccmake that prevents building Fortran (#723) ccmake runs iteratively, and the check_fortran_source_runs macros were clobbering a single output file that did not get updated on further configure iterations * Fix conflicts with merge * Move MAX_PRECISION back to HDF5UseFortran.cmake * Use STREQUAL to test macro argument * Move C language test to ConfigureChecks from HDF5UseFortran * MAX_PRECISION defines must be defined * Organize flags and align autotools and cmake * Fix comment in no-error-general warnings files. Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com> * Committing clang-format changes * Cache code tidying * Constification of cache code * Renaming of static calls in cache * Fixes H5C mess made by git during merge * Committing clang-format changes * Fixes a printf format issue w/ parallel * Revert "Committing clang-format changes" This reverts commit 1265fe87fec6589f029cf3b1de6d586c90168654. * Turns off clang formatter in H5C while syncing with develop * Adds a lot of HD prefixes and cleans up err.h usage * Committing clang-format changes * Brings cache skip list optimization from develop * Brings recent changes from develop Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Brings native H5Fdelete implementation from developDana Robinson2021-05-051-0/+1
|
* Brings many parallel changes from developDana Robinson2021-04-071-12/+16
|
* Brings the rest of the header changes from developDana Robinson2021-04-061-4/+4
|
* Updates license url and history docsDana Robinson2021-03-251-1/+1
|
* Formats the source and updates the gcc warning pragmasDana Robinson2021-03-251-147/+128
|
* Document vfd_swmr_pageno_to_mdf_idx_entry() and rename its parameterDavid Young2020-08-311-3/+15
| | | | nindices -> nentries.
* Wrap a line in a comment at 80 columns.David Young2020-08-311-2/+2
|
* Add/improve some struct/member documentation per John's request.David Young2020-08-311-0/+9
|
* Delete disused routine H5FD_has_conflict().David Young2020-08-201-1/+0
|
* Change H5F_open to perform deduplication of open VFDs in a more flexible way,David Young2020-07-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | using the new routine H5FDdeduplicate(). Simplify H5F_open() a bit, pushing some of the configuration checks into the SMWR VFD. For example, check that page buffering is enabled in H5FD_vfd_swmr_open() instead of in H5F_open(). Compare VFD SWMR configurations in H5FD_vfd_swmr_dedup() instead of in H5F_open(). Clone the default file-access property list at a new file-access property list ID, H5P_FILE_ACCESS_ANY_VFD. The new ID is used to indicate that if the file that's being opened is already open under an existing virtual file, and if that virtual file would not ordinarily be opened with the default FAPL, then it's ok to use that virtual file. Add a new optional method, `dedup`, to H5FD_class_t, and use it to customize a VFD's deduplication. Customize the SWMR VFD's deduplication. Make it honor H5P_FILE_ACCESS_ANY_VFD Embed the VFD SWMR configuration in the H5FD_vfd_swmr_t to facilitate comparison of configuration between new and old SWMR virtual files. In H5F__sfile_search(), match using a pointer comparison instead of H5FD_cmp(), because we will only ever enter H5F__sfile_search() with a deduplicated H5FD_t *.
* Delete whitespace at EOL. NFCI.David Young2020-07-141-1/+1
|
* Delete some dead code.David Young2020-07-141-2/+0
|
* Add all extant virtual files to a list. Add an "exclusive owner" (`exc_owner`)David Young2020-07-101-0/+1
| | | | | | | | | | | | | | member to all virtual files. Add a routine, H5FD_has_conflict(), that returns true if a new virtual file is identical to an existing virtual file that has an exclusive owner. Establish an exclusive owner for a VFD SWMR virtual file's lower virtual file. Rename bsdqueue.h to H5queue.h and install it, since it's used by H5FDpublic.h. This is part of a changeset that helps us avoid creating multiple H5F_shared_t for one file when virtual datasets are used with VFD SWMR. The old code for deduplicating VFD SWMR H5F_shared_t instances did not work correctly with VFD SWMR, so we'd end up with multiple H5F_shared_t all active on the same file.
* On each SWMR VFD open for reading, keep a histogram for the number ofDavid Young2020-06-161-0/+1
| | | | | ticks elapsed during API calls. Write the histogram to the swmr_stats log outlet when the SWMR VFD closes.
* Updated the 'const memory free' changes based on PR feedback.Dana Robinson2020-05-201-1/+1
|
* Update some comments.David Young2020-02-261-3/+3
|
* Where n is the number of page-table/shadow-index entries, avoid spending O(n^2)David Young2020-02-261-4/+5
| | | | | | | | time in H5PB_dest(). While we're in H5PB_dest(), mark deleted shadow-index entries as "garbage" and skip the O(n) shadow index-entries copy. Rename shadow index-entry member `moved_to_hdf5_file` to `moved_to_lower_file` while I'm in here---NFCI.
* Make H5FD_vfd_swmr_dump_status() take an unsigned page number to be consistentDavid Young2020-02-061-1/+1
| | | | with other code.
* Numerous changes supporting a floating shadow index:David Young2020-02-031-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Add to the H5F_shared_t (!) a new member that tells the index in the shadow file where the index should be written. Allocate shadow filespace for the header and the index separately so that the index can float. Update tests to match the expected original location of the index. Introduce vfd_swmr_enlarge_shadow_index(), a routine that allocates space in the shadow file for a new index that has (up to) twice as many entries as the old index, allocates a new in-core index of the same size, and copies the old in-core index to the new. Call vfd_swmr_enlarge_shadow_index() in H5PB_vfd_swmr__update_index() when the in-core index has too few slots. In the comment at the top of H5FD__vfd_swmr_load_hdr_and_idx(), describe the protocol that it follows, now, when it reads the shadow header and index. Delete some dead code in the function and add a bit of diagnostic code. TBD quiet the diagnostic code. In H5F_vfd_swmr_init(), follow the protocol: write the index, first, then the header. Modify property-list checks and tests to reserve no fewer than two pages at the front of the shadow file for the header and index.
* Fix typo. NFCI.David Young2020-01-301-1/+1
|
* Prepare the VFD SWMR reader for a "floating" shadow index by overhaulingDavid Young2020-01-241-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the way that the shadow header and shadow index are loaded. In H5FD__vfd_swmr_load_hdr_and_idx(), adopt a new protocol for reading the shadow file: 0 If the maximum number of retries have been attempted, then exit with an error. 1 Try to read the shadow file *header*. If successful, continue to 2. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 2 If the tick number in the header is less than the tick last read by the VFD, then return an error. 3 If the tick number in the header is equal to the last tick read by the VFD, then exit without doing anything. 4 Try to read the shadow file *index*. If successful, continue to 5. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 5 If a different tick number was read from the index than from the index, then continue at 0. 6 Try to *re-read* the shadow file *header*. If successful, continue to 7. If there is a hard failure, then return an error. If there is a failure that may be transient, then sleep and retry at 0. 7 Compare the header that was read previously with the new header. If the new header is different than the old, then we may not have read the index at the right shadow-file offset, or the index may have been read in an inconsistent state, so sleep and retry at 0. Otherwise, return success. Simplify H5FD__vfd_swmr_header_deserialize() and H5FD__vfd_swmr_index_deserialize(). Remove their retry loops. Make each return TRUE on success, FALSE on an error that may be transient, and FAIL on an irrecoverable error. In H5FD__vfd_swmr_header_deserialize(), do not check the size of the shadow file with fstat(2), since the read(2) will fail if the file is too small. This saves us a system call. Lightly consti-ify H5FD__vfd_swmr_index_deserialize() arguments. In H5FD__vfd_swmr_load_hdr_and_idx(): Consolidate all of the retry-looping. Increase the initial retry delay from 1ns to 1/10s. Delete the disused maximum-retry constants. Use #if 0 to disable some error-checking code that ought to be unnecessary under the new protocol. Don't memset() the header and index header, but make sure they're fully initialized with real content, instead.
* Change the blah_blah_blah_md_header `index_length` member fromDavid Young2020-01-101-1/+1
| | | | | | | | `uint64_t` to `size_t` because it describes the size of an in-core structure as well as an on-disk one, and `size_t` is wide enough to store the size of any in-core structure, while `uint64_t` may be much too wide. Check that `index_length` is no more than SIZE_MAX after we read it.
* Change the unwieldy name vfd_swmr_idx_entry_defer_free() toDavid Young2019-12-191-1/+1
| | | | | shadow_image_defer_free(), which is also a better description of what the routine does.
* Use the "tail queue" from queue(3) for the delayed free list of shadow images.David Young2019-12-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Delete the little-used free-list length, dl_len, and just count up the list entries when diagnostic code needs the length. Extract the code for deferring shadow-image free into a new subroutine, `vfd_swmr_idx_entry_defer_free()`. Rename type `deferred_free_head_t` as `deferred_free_queue_t`. Remove the disused H5F__LL_{REMOVE,PREPEND} macros. Add some diagnostic code and #if 0'd assertions. Change `qsort(ptr, n, sizeof(type), cmp)` to `qsort(ptr, n, sizeof(*ptr), cmp)`. Use a `continue` statement to lower a staircase in H5F_update_vfd_swmr_metadata_file(). Add vfd_swmr_mdf_idx_entry_remove() to delete a shadow index entry and add the image at that entry to a deferred-free list. Call it whenever a page is evicted. Update the comment in H5PB_remove_entry() that asks if we need to remove shadow index entries: now we *do* remove them. Remove shadow index entries in H5PB__evict_entry(). Also mention in the comment that the index-entry removal performed by H5PB__evict_entry() ought to be sufficient.
* Merge all of my changes from merge-back-to-feature-vfd_swmr-attempt-1,David Young2019-12-091-11/+33
| | | | | | | | including the merge of `hdffv/hdf5/develop`, back to the branch that Vailin and I share. Now I need to put this branch on a fork with a less confusing name than vchoi_fork!
* Checkin of fixes for a number of bugs in reads to page 0, and in filemainzer2019-01-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | open in the VFD SWMR reader case. Note that the following failures in testvfdswmr.sh: 1) Unable to find metadata file on VFD SWMR reader open. 2) Occasional sanity check failures in the page buffer on raw data write. 3) Filter failures on raw data read in VFD SWMR readers when compression is enabled. 4) Unexpected data errors in VFD SWMR readers when compression is disabled. Note that I expect that items 3 & 4 two aspects of the same issues -- the fact that we don't guarantee that raw data is consistent with metadata. Item 2) must be addressed, but it is so infrequent that it isn't doesn't affect the conclusion VFD SWMR seems to work, and thus it can wait until phase 2. I am given to understand that Vailin has largely addressed item 1), and will be checking in her solution to this soon. Tested on Charis and Jelly.
* Intermin commit to allow update from Vailin's recent changesmainzer2019-01-151-1/+3
|
* Fixes for issues #1 and #2 listed in John's email dated Dec 7 2018:Vailin Choi2018-12-191-0/+1
| | | | | | | | (1) Assertion failure in the vfd_swmr test (2) Reader error in the vfd swmr concurrent tests Also fixes for: (a) Use H5MV_alloc() to allocate space for md_pages_reserved when creating the metadata file in H5F__vfd_swmr_init() (b) Remove a multi-page (when vfd_swmr_writer is true) from the page buffer in H5MF_xfree()
* local commit of first cut at vfd swmr writer EOT code.mainzer2018-11-191-40/+62
| | | | | | This is necessary to allow access to Vailin's recent changes This version passes non-swmr tests in a serial / debug build on charis.
* (1) Add tests for VFD SWMR writerVailin Choi2018-10-221-15/+18
| | | | (2) Bug fixes
* Third batch of checkin:Vailin Choi2018-09-281-20/+46
| | | | | | | | | | 1) Free space manager for the metadata file 2) Delayed free space release linked list 3) H5F_update_vfd_swmr_metadata_file() 3) VFD SWMR driver: read callback 4) Flushing for VFD SWMR 5) Port one concurrent test from swmr test set 6) Bug fixes and refactoring
* Merge branch 'develop' of ↵Vailin Choi2018-09-261-5/+3
|\ | | | | | | | | | | | | | | | | | | https://bitbucket.hdfgroup.org/scm/~vchoi/my_hdf5_fork into feature/vfd_swmr Conflicts: src/H5AC.c src/H5F.c src/H5Pfapl.c Resolve conflicts for H5AC.c, H5F.c and H5Pfapl.c
| * Normalization with vol_integration (property lists, file drivers,Dana Robinson2018-09-211-5/+3
| | | | | | | | other misc).
* | Bug fixes for the previous checkin.Vailin Choi2018-09-041-4/+4
| |
* | Second batch of checkin:Vailin Choi2018-09-041-3/+80
| | | | | | | | | | | | | | | | 1) Define driver for the VFD SWMR reader 2) Implement VFD SWMR open callback 3) Implement H5FD_vfd_swmr_get_tick_and_idx() 4) Load and decode metadata file header and index 4) Closing for VFD SWMR
* | Preliminary checkins for implementation done so far:Vailin Choi2018-08-141-0/+43
|/ | | | | | | | | | | | 1) Public routines: H5Pget/set_vfd_swmr_config 2) Public routine: H5Fvfd_swmr_end_tick 3) Initialization when opening a file with VFD SWMR writer 4) Tests in test/vfd_swmr.c for VFD SWMR 5) Fix a bug in src/H5Fint.c: when error is encountered after the root group is created Note the following: --This is WORK IN PROGRESS and will subject to change as implementation goes. --There is test failure form enc_dec_plist.c: I will fix this when changes to the property list are settled. --The branch is updated with the latest from develop as of 8/14/2018
* Add API context interface and use it throughout the library.Quincey Koziol2018-03-151-36/+10
|
* * Added new public H5FDdriver_query() API call.Dana Robinson2017-05-241-1/+1
| | | | | | * Changed return type of H5FD_driver_query() to herr_t. * Updated swmr.c to use H5FDdriver_query() instead of the private call. * Added some flags tests to vfd.c.
* * Made H5FD_driver_query() a private function instead of static.Dana Robinson2017-05-241-0/+1
| | | | | | | * Moved H5FD_driver_query() to H5FDint.c. * Minor typo fixes in H5I.c. * Updated swmr.c to check the VFD feature flags and skip the retries test when the VFD is not compatible with H5P_DEFAULT.
* Merge pull request #426 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10 to ↵Larry Knox2017-04-251-6/+4
| | | | | | | | | | hdf5_1_10 * commit '54957d37f5aa73912763dbb6e308555e863c43f4': Commit copyright header change for src/H5PLpkg.c which was added after running script to make changes. Add new files in release_docs to MANIFEST. Cimmit changes to Makefile.in(s) and H5PL.c that resulted from running autogen.sh. Merge pull request #407 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10_1 to hdf5_1_10_1 Change copyright headers to replace url referring to file to be removed and replace it with new url for COPYING file.
* Checkin of fix for CGNS bugmainzer2017-04-061-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (https://jira.hdfgroup.org/browse/HDFFV-10055). Briefly, in H5C_collective_write() in H5Cmpio.c, the metadata cache attempts to perform a collective write of metadata cache entries. This worked fine as long as all processes had at least one entry to write. However, when the process has no entries, the function tries to participate in the collective write by calling MPI_File_set_view(), MPI_File_write_all() and then MPI_File_set_view() again, to match the calls in H5FD_mpio_write(). After pull request 183, the CGNS test benchmark_hdf5 started failing. On investigation, I determined that the failure occurred in the first call to MPI_File_set_view() in the "no data to write" path through H5C_collective_write(). Note that pull request 183 did not create the problem, it only exposed it. The bug can be observed after pull request 182 if one executes the CGNS progam src/ptests/benchmark_hdf5 with 90 processes. The problem appears to have been that the calls to MPI_File_set_view() in H5C_collective_write() and H5FD_mpio_write() were using different values for the info parameter. I patched the problem by adding a MPI specific VFD call allowing me to get the MPI_Info used in H5FD_mpio_write() for use in MPI_File_set_view() calls in H5C_collective_write(). Tested serial & parallel, debug & production on Jelly.
* Final merge of page buffering branch to developQuincey Koziol2017-03-141-2/+3
|
* Bring changes to I/O parameters from page_buffering branch.Quincey Koziol2017-03-131-16/+16
|
* Bring file shutdown code from cache image branchQuincey Koziol2017-01-051-1/+1
|
* Bring SWMR support in to the main development branch. (Finally!) More testsQuincey Koziol2016-12-021-0/+1
| | | | and the tool and API wrappers will be coming in over the weekend.
* Warning and whitespace cleanup.Quincey Koziol2016-11-031-3/+15
|
* [svn-r30113] Description:Quincey Koziol2016-06-281-1/+1
| | | | | | | | Clean up warnings (from 2774 -> 1560, with my standard debug build) Tested on: MacOSX/64 10.11.5 (amazon) w/serial, parallel & production (h5committest forthcoming)
generated by cgit v0.12 at 2024-11-18 03:01:47 (GMT)