| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
* Updated source file copyright headers to remove "Copyright by the Board of Trustees
of the University of Illinois", which is kept in the top-level COPYING file.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Sketch of the H5S life cycle.
* Committing clang-format changes
* Fix H5S_UNLIMITED snafu.
* Updated RM template and RM page.
* Added H5S life cycle.
* Committing clang-format changes
* Added H5T life cycle.
* Committing clang-format changes
* Cleaner layout (?)
* Cleaned the H5F life cycle. Called out unfinished biz.
* Committing clang-format changes
* Remaining life cycle skeletons.
* Committing clang-format changes
* Committing clang-format changes
* Added H5Z life cycle.
* Committing clang-format changes
* Added H5G life cycle.
* Committing clang-format changes
* H5 and H5I life cycle updates.
* Committing clang-format changes
* Added H5PL life cycle.
* Committing clang-format changes
* Added H5L life cycle.
* Committing clang-format changes
* Fix for Chris' comment.
* Add a variable for Doxygen pre-processor definitions.
* Forgot to add the H5M API.
* Clarify the H5Z life cycle.
* Committing clang-format changes
* Add H5Zdevelop.h to Doxygen.in. Added H5I life cycle.
* Committing clang-format changes
* Clarified introduction and fixed missing label declaration.
* Added H5O life cycle.
* Committing clang-format changes
* H5O cleanup, part 1.
* Committing clang-format changes
* Cleaned up some of the endless repetition in H5O.
* Committing clang-format changes
* Cookbook & RFC draft layouts.
* Updated manifest.
* Updated the manifest, the example paths, and sketched the 1st recipe.
* Committing clang-format changes
* Outlined two more recipes.
* Committing clang-format changes
* More recipes and RFCs.
* Committing clang-format changes
* Draft of templatized RFC references.
* Another batch of RFC changes.
* Another batch of RFCs.
* Fixed reference.
* RFCs in reverse chronological order.
* First cut of RFCs.
* Fixed reference.
* Updated recipes.
* Updated recipes.
* More RFCs.
* Updated D*PL comments.
* Added H5P descriptions.
* Committing clang-format changes
* H5R life-cycle snapshot.
* Committing clang-format changes
* H5R life-cycle. Added line numbers to life-cycle examples.
* Committing clang-format changes
* Fixed formatting for H5Dchunk_iter().
* Added comment on collective mode requirement w/ compression.
* Simplified API compat. macro dox.
* More API vers. updates.
* Hide the async macro entrails.
* Latest VFD SWMR RFC.
* Create a tag file for permalinks.
* Added TODOs for metadoc.
* Removed duplication.
* Revised RM landing page.
* Trimmed more duplication.
* Committing clang-format changes
* Revised H5D.
* Committing clang-format changes
* Updated survey link.
* Added Doxygen RM entry template link.
* Added the "Multi-Thread HDF5" RFC.
* Added DOXYGEN_TAG_FILE.
* Added selection I/O RFC.
* Added the VFD Sub-filing RFC.
* Updated meta-documentation and added two old presentations.
* Added a few more RFCs (4).
* Fixed MANIFEST.
* Updated meta-documentation.
* Added Filters technical note.
* Fixed MANIFEST.
* Restore the path stripper.
* Experimental full-text search via Google.
* Better full-text search integration.
* Whoops. Forgot this one.
* Oh boy.
* Make CMake happy.
* Added "Debugging HDF5 Applications" technical note.
* Another batch of RFCs.
* Fixes for #1221.
* Updated overview.
* Fixed image dependencies.
* CMake updates.
* Fixed SET.
* Better?
* Update doxygen/dox/Overview.dox
* Fixed documentation errors. Added missing version info.
* Callback documentation updates.
* Fixed indexing errors in the outline.
* Doxygen-ized the HDF5 glossary.
* Fix a few minor typos
.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Take a stab at using constructors to initialize instead of
function-entry macros. This is a work in progress. It's good enough to
run `many_dsets`.
* Committing clang-format changes
* Add the `many_dsets` benchmark and some scripts I used on jelly for
setting up the build/test environment and for recording/flame-graphing
profiles.
* Committing clang-format changes
* Change my Makefile and environment script to work both on jelly
and on mayll (and probably on Summit).
* Disable clang-format "fix."
* Replace the `if (!H5_TERM_GLOBAL)` test in each FUNC_ENTER_ macro with
`if (true)`.
* Fix bad grammar in a comment.
* Instead of labeling the H5*__init_package routines constructors, fold
each into an initialization routine, H5*_init(), and call each of
the H5*_init() routines. Call most of the H5*_init() routines from
H5_init_library() in an explicit order that I found out earlier by
instrumenting each __init_package routine and running the library
tests. Roll H5FD*__init_package routines into H5FD*_init() routines.
This change ends just-in-time initialization of package dependencies by
package initializers.
Don't track in per-package variables (H5_PKG_INIT_VAR) whether each
package has been initialized. Instead, track in a single library
variable whether the whole library is initialized or not.
Drive the initialization of packages by H5_init_library() with a table
of initializer routines. Also drive the termination of packages by
H5_term_library() with a table.
Perform initialization as needed from FUNC_ENTER_API_INIT(err). This
basically restores the old behavior of that macro.
Delete a bunch of #definitions in H5private.h that have fallen out of
use with these changes.
* Committing clang-format changes
* Undo the bad auto-formatting that appears to have occurred in spite
of my disabling it. Bracket some code in /* clang-format off */ /*
clang-format on */ to prevent a recurrence.
* Remove a diagnostic abort().
* Fix a logic error: print a comma between every package terminator run,
and don't print an initial comma.
* Complete the changes I started in H5_term_library() that undo the bad
auto-formatting.
Stop tracking whether package "tops" were initialized in per-package
variables H5*_top_package_initialize_s. H5_term_library() takes care of
that for them.
Remove H5R_top_term_package() and H5R_term_package(), they don't do
anything.
* Committing clang-format changes
* NFCI. Simplify macro text: replace `if (true) {` with `{`.
* Fix formatting and suppress clang-format on a longer range.
* Quiet some unused label, unused variable complaints that cropped up
after I simplified the FUNC_ENTER_ macros for the sake of performance.
* Committing clang-format changes
* Delete some programs and scripts that don't belong in the pull request.
* Use the right function-entry macro.
* Use a sensible format and disable auto-formatting.
* Stop calling do-nothing initializer H5FS_init(). Delete it.
* Document what changes to make if the default VFD changes.
* While I am here, change an `await_prior` flag on the terminator table
to `true` to match the previous, non-table-driven code that was here.
Found the oversight making the following changes:
NFCI: insert an empty line and copy over slightly-edited comments from
the previous version, where those comments still correctly explained how
library termination operated.
* NFCI: lower a staircase.
* Replace every occurrence of FUNC_ENTER_NOAPI_INIT(...) with H5_PUSH_FUNC
since that is all that that macro does any more.
Quiet a bunch of new warnings by changing FUNC_ENTER_NOAPI(...) to
FUNC_ENTER_NOAPI_NOERR and removing disused `done:` labels.
* NFCI: add curly braces around a multiline statement.
* Quiet a signed/unsigned comparison warning.
* Add some documentation about library initialization and shutdown.
* Make sure that the library is initialized, or else that initialization
is already underway, before performing any VFD's initialization.
* Committing clang-format changes
* Committing clang-format changes
* Reduce differences from `develop` branch.
* Always initialize `tot_init`.
* Committing clang-format changes
* Fix typo: H5SL_init initializes skip lists, not VOL.
* Remove H5_TERM_GLOBAL test in H5T_init. H5T_init was unusual in that
it tested H5_TERM_GLOBAL and exited early if it was set. No other
module initializers did that, and I cannot find any reason that should
be necessary. Tests still pass when I remove it, so away it goes.
* Use HD prefix.
* Add function header comments.
* Drop the intermediate variable, it's only used once.
* Extract subroutine `H5FDperform_init(hid_t (*init)(void))` that
initializes the library, if necessary, before calling its VFD-initializer
argument. Use H5FDperform_init in the definition of the symbols
H5FD_<vfd> (e.g., H5FD_SEC2), which may be evaluated before the library
is initialized, like so:
```
```
I implement H5FDperform_init in its own source file, H5FDperform.c,
and exclude that file from trace processing because the `bin/trace`
cannot deal with the function-pointer type.
* Straggler from last: add new source file src/H5FDperform.c.
* Committing clang-format changes
* Add a missing file to the MANIFEST.
* Switch to FUNC_ENTER_API_NOINIT in H5FDperform_init() and hbool_t in
H5_term_library().
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fixed warnings and started H5Epublic.h.
* Include H5FD* headers to correctly resolve references.
* Doxygen2 (#330)
* H5Eauto_is_v2.
* Added a few more calls.
* Added a few more H5E calls.
* First cut of H5E v2.
* Added the deprecated v1 calls.
* Updated spacing.
* Once more.
* Taking some inspiration from Eigen3.
* Add doxygen for the assigned functions: H5Pregister1,H5Pinsert1,H5Pen… (#352)
* Add doxygen for the assigned functions: H5Pregister1,H5Pinsert1,H5Pencode1, H5Pget_filter_by_id1,H5Pget_version, H5Pset_file_space,H5Pget_file_space. Someone already adds H5Pget_filter1. Also fixs an extra parameter 'close' call back function for HPregister2.
* doxygen work. fixs format by using clang-format.
* doxgen work for H5Pregister1 etc. Addressed Barbara and Gerd's comments.
For Quincey's comments, since we are not supposed to change the source code.
I leave this to future improvements.
* added documentation for H5P APIs (#350)
* add documenation for H5Pget_buffer,H5Pget_data_transform,H5Pget_edc_check,H5Pget_hyper_vector_size,H5Pget_preserve,H5Pget_type_conv_cb,H5Pget_vlen_mem_manager,H5Pset_btree_ratios
* format corrections
* fixed grammer
* fixed herr_t
* Better name.
* A fresh look.
* add doxygen to H5Ppublic.h
* use attention instead of warning
* Add doxygen comments in H5Ppublic.h (#375)
* Add doxygen comments in H5Ppublic.h
* H5Pset_meta_block_size
* H5Pset_metadata_read_attempts
* H5Pset_multi_type
* H5Pset_object_flush_cb
* H5Pset_sieve_buf_size
* H5Pset_small_data_block_size
* H5Pset_all_coll_metadata_ops
* H5Pget_all_coll_metadata_ops
* Add DOXYGEN_EXAMPLES_DIR to src/CMakeLists.txt
* Fix clang-format errors
* Fix filenames in doxygen/examples
* add doxygen to H5Ppublic.h (#378)
* add doxygen to H5Ppublic.h
* use attention instead of warning
Co-authored-by: Kimmy Mu <kmu@hdfgroup.org>
* Revert "add doxygen to H5Ppublic.h (#378)"
This reverts commit 2ee1821b138a5c00b15ea57ce9e950367480f5f2.
* Updated Doxygen variables.
* I forgot to copy two images.
* Enable desktop search by default.
* Add my assigned Doxygen documentation.
* Remove whitespace at EOL. Appease clang-format.
* Addressed Chris' comments.
* Added an alias for asynchronous functions.
* One space is enough for all of us.
* Slightly restructured RM page.
* address some issues
* reformatting
* Style external links.
* reformatting
* reformatting
* Added "Metadata Caching in HDF5" as a technical note example.
* Revise this soon!
* Added specification examples.
* Fixed references.
* Added H5AC cache image stuff and file format study.
* Added older FMT versions. Where did 1.0 go?
* Updated C/C++ note and replaced ambiguous labels.
* Reformat source with clang v10.0.1.
* Added the VFL technical note.
* Added what I believe might be called version 1.0 of the format.
* Added the remaining specs.
* Added H5Z callback documentation and fixed a few mistakes.
* Added dox for deprecated H5G calls and fixed a few snippet blockIDs.
* clang-format happy?
* Ok?
* Bonus track: Deprecated H5D functions.
* Carry over the more detailed group description.
* Added documentation for the missing and deprecated H5R calls.
* Life is easier and less repetitive w/ snippets. Use them!
* Eliminate the snippet block ID artifacts in the HTML rendering.
* Fixed snippet HTML artifacts and added a few missing calls.
* Under 20 H5Ps to go!
* Almost complete!
* "This is a form of pedantry up with which I will not put." (Churchill)
* Let's not waste as much space on bulleted lists!
* First complete (?) draft of the Doxygen-based RM.
* Completeness check and minor fixes along the way.
* Pedantry.
* Adding missing H5FD calls checkpoint.
* Pedantry.
* More pedantry.
* Added H5Pset_fapl_log.
* First draft of H5ES.
* Fixed warnings.
* Prep. for map module.
* First cut of the map module.
* Pedantry.
* Possible H5F introduction.
* Fix the indentation.
* Pedantry.
* Ditto.
* Thanks to the reviewers for their comments.
* Added missing images.
* Line numbers are a distraction here.
* More examples, references, and clean-up. Don't repeat yourself!
* Clang pedantry.
* Ditto.
* More reviewer comments...
* Templatized references and cleaned up \todos.
* Committing clang-format changes
* Fixed MANIFEST.
* Addressed Quincey's comments. (OCPLs)
* Fixed a few more \todo items.
* Fixed more \todo items.
* Added attribute life cycle.
* Forgot the examples file.
* Committing clang-format changes
* Pedantry.
* Live and learn!
* Added a sample H5D life cycle.
* Committing clang-format changes
* Pedantry.
Co-authored-by: kyang2014 <kyang2014@users.noreply.github.com>
Co-authored-by: Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Kimmy Mu <kmu@hdfgroup.org>
Co-authored-by: Christopher Hogan <ChristopherHogan@users.noreply.github.com>
Co-authored-by: jya-kmu <53388330+jya-kmu@users.noreply.github.com>
Co-authored-by: David Young <dyoung@hdfgroup.org>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
|
| |
|
|
|
|
| |
* Modify temporary rpath for testing in java example scripts.
* Update URL in source file Copyright headers for web copy of COPYING
file - src and test directories.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
- HDFS VFD files now ignored in the Autotools when the HDFS VFD is not
being built.
- All VFD init code uses idiomatic H5I_INVALID_HID instead of -1
or FAIL.
- The HDFS VFD now includes H5FDdrvr_module.h in the right place.
- The HDFS tests in test/hdfs.c no longer pass NULL to VERIFY macros, which
raised warnings.
- Minor tweaks to fix const, unused variables, etc. warnings in
several VFDs.
|
| | |
|
| |
|
|
|
|
| |
header file, FUNC_ENTER / LEAVE, etc). Removed remaining personal email
addresses from library source code (still needs cleaned from other directories).
Misc. warning, style, and whitespace cleanup.
|
| |
|
|
|
|
|
|
|
|
| |
hdf5_1_10
* commit '54957d37f5aa73912763dbb6e308555e863c43f4':
Commit copyright header change for src/H5PLpkg.c which was added after running script to make changes.
Add new files in release_docs to MANIFEST. Cimmit changes to Makefile.in(s) and H5PL.c that resulted from running autogen.sh.
Merge pull request #407 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10_1 to hdf5_1_10_1
Change copyright headers to replace url referring to file to be removed and replace it with new url for COPYING file.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove all traces of MPI-POSIX VFD and GPFS detection/code.
Remove remaining traces of stream VFD.
Remove testpar/t_posix_compliant test (it's not actually verifying anything).
Clean up H5D__mpio_opt_possible() further.
Moved environment variable that disables MPI collective operations into
MPI-IO VFD (instead of it being in src/H5S.c).
A few other small code cleanups.
Tested on:
Mac OSX/64 10.9.2 (amazon) w/parallel & serial
|
| |
|
|
|
|
| |
https://jira.hdfgroup.uiuc.edu/browse/HDFFV-8203
tested: h5committest
|
| |
|
|
|
|
|
|
|
| |
Clean up code a bit, move new "terminate" VFD callback to new position
in H5FD_class_t struct.
Tested on:
Mac OS X/32 10.6.7 (amazon) w/debug & production
(h5committest not required on this branch)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring file free space branch changes through r15795 into trunk, which
includes a fair bit of code cleanup & rearrangement along with a couple of
bug fixes also.
Tested on:
Mac OS X/32 10.5.5 (amazon) in debug mode
Mac OS X/32 10.5.5 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
|
| |
|
|
|
|
|
|
|
| |
copyright notice.
Tested platform:
Kagiso only since it is only a comment block change. If it works in one
machine, it should work in all, I hope. Still need to check the parallel
build on copper.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
optimization codes:
1. Provide another option for users to do independent IO with MPI file setview(collectively)
2. With the request of collective IO from users, using Independent IO with MPI file setview if we find collective IO is not good for the applications for IO per chunk(multi-chunk IO) case. Previously we used pure independent IO and that actually performed small IO(IO each row) for this case. The recent performance study suggested the independent IO with file setview can acheieve significantly better performance than collective IO when not many processes participate in the IO.
3. For applications that explicitly choose to do collective IO per chunk case, the library won't do any optimization(gather/broadcast) operations. The library simply passes the collective IO request to MPI-IO.
Tested at copper, kagiso, heping, mir and tungsten(cmpi and mpich)
Kagiso is using LAM, t_mpi test was broken even.
The cchunk10 test failed at heping and mir. I suspected it was an MPICH problem. Will investigate later.
Everything passed at copper.
at tungsten: the old cmpi bug(failed at esetw) is still there. Other tests passed.
Some sequential fheap tests failed at kagiso.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New APIs to add for collective chunk IO
Description:
Three new APIs
H5Pset_dxpl_mpio_chunk_opt_ratio
H5Pset_dxpl_mpio_chunk_opt_num
H5Pset_dxpl_mpio_chunk_opt
for optional optimization choices from users.
Solution:
Haven't added tests yet, won't affect other parts of the library.
Will add tests after urgent investigations of memory leaking problems from NASA Aura team.
Platforms tested:
heping: both parallel and sequential
shanti
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Add destructor to match constructor fr VFLs when they are shut down by the
library.
Solution:
Added H5FD_*_term() routines to "undo" changes made in H5FD_*_init()
routines.
Platforms tested:
IBM p690 (copper)
too minor to require h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix/optimization
Description:
Address slowdown in MPI-I/O file metadata operations that was introduced
mid-stream. We now _require_ a POSIX compliant parallel file system for the
MPI-I/O file driver (as well as for the MPI-POSIX file driver).
Also optimized file open operation when the file is being created by
reducing the number of collective & syncronizing calls.
Additionally, refactor the MPI routines into a common place, eliminating
duplicated code.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix & code cleanups
Description:
Change our use of MPI derived datatypes to not create datatypes with
"0-sized" lengths, which causes the LANL Q machine to hang.
Also, get rid of "prefer MPI derived datatypes" environment variable since
it has no advantage.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o parallel
h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
Updated copyright statement in files which hadn't been updated yet.
Platforms tested:
Linux (Only comment change)
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Purpose:
__DLL__ is a keyword in some platforms and __DLL__ is also defined as a macro for windows DLL applications.
That causes problems.
Description:
Solution:
Use H5_DLL*** to replace __DLL***__ at all header files.
Change the macro defination at H5api_adpt.h.
Platforms tested:
linux2.2.18smp, irix64, solaris 2.7 and windows 2000
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix/Code cleanup/New Feature
Description:
Correct problems with writing fill-values to external storage and allocate
the data storage at the correct times.
Also, mostly straighten out the strange code which allocates and fills
raw data storage for datasets. Things are still a bit odd in that the
fill-values for chunked datasets are written when the space is allocated,
instead of in a separate routine, but there are two reasons for this:
it's inefficient (especially in parallel) to iterate through all the chunks
twice, and (more importantly) the space needed to store compressed chunks
isn't known until we've got a buffer of compressed fill-values ready to
write to the chunk.
Additionally, add in the H5D_SPACE_ALLOC_INCR and H5D_SPACE_ALLOC_DEFAULT
setting for the "space time", which incorporate the previous behavior of
the space allocation for chunked datasets.
The default settings for the different types of dataset storage are now
as follows:
Contiguous - Late
Chunked - Incremental
Compact - Early
This checkin also incorporates a change to the behavior of external data
storage in two ways - fill-values are _never_ written to external storage
(under the assumption that writing fill-values is triggered by allocating
space in an HDF5 file, and since space is not allocated in the file, the
fill-values should not be written) and external data files are now created
if they don't exist when data is written to them. The fill-value will
probably need to be revisited at some time in the future, this just seemed
like the safer course currently.
I think I cleaned up some compiler errors also, before getting bogged down
in the fixes for the space allocation and fill-values.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/serial & parallel. Will be testing on IRIX64
6.5 (modi4) in serial & parallel shortly.
|
| |
|
|
|
|
|
|
|
|
|
| |
Code improvement
Description:
Some small code cleanups and took out the code the was turning off the
metadata cache for parallel I/O (!)
Platforms tested:
IRIX64 6.5 (modi4) w/parallel
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Code optimization
Description:
Avoid creating MPI types (and thus requiring a MPI_File_set_view() call)
when contiguous selections are used for dataset I/O. This should be a
performance improvement for those sorts of selections.
Platforms tested:
Linux 2.2.x (eirene) w/parallel && IRIX64 6.5 (modi4) w/parallel & FORTRAN
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Use dataset transfer property list to hold information about the MPI types
for the current transfer, instead of setting pseudo-global variables in
the file's struct.
Platforms tested:
Linux 2.2.x (eirene) w/parallel & IRIX64 6.5 (modi4) w/parallel & FORTRAN
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Change MPI-I/O code to use the address of the dataset for the displacement,
instead of having a separate displacement value. Removed displacement
parameter from H5FD_mpio_setup parameters.
Platforms tested:
Linux 2.2.x (eirene) w/parallel & IRIX64 6.5 (modi4) w/parallel.
|
| |
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Add 'closing' parameter to H5FDflush and VFL "flush" functions, per
http://hdf.ncsa.uiuc.edu/RFC/VFLFlush/VFLFlush.html
Platforms tested:
IRIX64 6.5 (modi4)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix/Code improvement.
Description:
Currently, the chunk data allocation routine invoked to allocate space for
the entire dataset is inefficient. It writes out each chunk in the dataset,
whether it is already allocated or not. Additionally, this happens not
only when it is created, but also anytime it is opened for writing, or the
dataset is extended. Worse, there's too much parallel I/O syncronization,
which slows things down even more.
Solution:
Only attempt to write out chunks that don't already exist. Additionally,
share the I/O writing between all the nodes, instead of writing everything
with process 0. Then, only block with MPI_Barrier if chunks were actually
created.
Platforms tested:
IRIX64 6.5 (modi4)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Performance enhancement
Description:
Doing an MPI_File_sync() just before a file is closed causing a large
performance loss.
Solution:
Add flag to MPI file driver to avoid performance the MPI_File_sync() when
the flag is set before a call to H5F_flush().
Platforms tested:
IRIX64 6.5 (modi4)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
The parallel I/O file driver is optimized to only write metadata with one
process (and broadcast the results to the other processes). This is
currently enabled by a separate call to H5FD_mpio_tas_allsame() before
each metadata write to the file. This can easily lead to problems where
the prelude function call is omitted before the actual write code or, in
a threaded environment, lead to race condititions where the value set is
reset before being used.
Solution:
Since we only want to write metadata from one process, key off of the 'type'
parameter (which has information about whether the data being written it
metadata or raw data) to H5FD_mpio_write() as the method for determining
whether to only write from one process or not.
Platforms tested:
IRIX64 6.5 (modi4)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix
Description:
This file is not C++ friendly/compliant because the protocols
are not bracketed by the #ifdef __cplusplus macro. This was
discovered by a user attempting to use C++ with enable-parallel.
Solution:
Though we are not supporting C++ in parallel mode yet, it is
simple to add the macro bracket. It would help if C++ is supported
in the future.
Platforms tested:
modi4 (64,n32) and eirene (mpich).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
Changed
#include <hdf_file.h>
construct to
#include "hdf_file.h"
so that the GNU compiler can more easily pick up the dependencies
which it places in the .depend and Dependencies files. Also
regenerated the Dependencies to go along with this.
Platforms tested:
Linux
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix
Description:
When parallel I/O is turned on, there were some macros used in the H5D
routines which poked around in the H5F_t structure. This breaks the
privacy of that structure and ties the H5D code too tightly to the H5F_t
struct.
Solution:
Added a small function to retrieve the the value (driver_id) needed from
the H5F_t function.
Platforms tested:
Eyeballed only, Albert needs this right away...
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix for parallel mode.
Description:
H5FD_mpio_tas_allsame was called for all cases, even when
MPIO is not used for access. That corrupted the internal
file handle structure.
Solution:
Define a macro, IS_H5FD_MPIO(f), for testing if f is opened with
MPIO access. Will call H5FD_mpio_tas_allsame only if this
condition is true.
Platform tested:
O2K, both -64 and -n32 modes.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
----------------------
./src/H5public.h
We undefine a bunch of things that could get redefined in the config
file because some customers have applications that include headers
from multiple packages, all of which might be using autoconf.
Include <stdint.h> for the C9x types.
./test/h5test.h
More flushing of stdout for when testing is redirected down a pipe.
./tools/h5ls.c
Added a `-S' or `--simple' switch which causes the output to be
simplified somewhat for easier parsing by other scripts. For instance,
characters are escaped using a very simple mechanism instead of C's
more complicated backslash notation, data doesn't have `{}' or `[]'
characters interspersed for compound and array types, and data is
printed with exactly one element per line. This switch is now used by
an HDF5-to-HTML CGI script being developed for the DMF people.
./tools/h5tools.c
./tools/h5tools.h
The repeat threshold which controls how strings are printed when a
character repeats a bunch of times is now settable at runtime instead
of compile time. The default is to show all characters, like
"abceeeeeeeeeeeeeeeeeeeeeeeeeeeeeefgh"
But if you set it to something like 5 then any sequence of 5 or more
characters is replaced by something shorter, like:
"abc" 'e'x30 "fgh" or
Added an `str_locale' property which describes how to escape special
characters in strings. The default is C-like escapes but an
alternative is ESCAPE_HTML which replaces all non-alphanumeric
characters with a 3-character HTML escape of the form `%XX'
Fixed a bug where empty strings didn't even have the quote characters
printed. Now empty strings show up as `""' instead of absolutely
nothing.
Added a `per_line' property which controls the maximum number of
elements which will appear per line of output. The default is infinity
but in practice the right margin causes line breaks. By setting the
`per_line' value to one and the right margin to a very large value one
can achieve output with exactly one element per line.
|
| |
|
|
| |
It is working now. Still need to tie up some loose ends.
|
| |
|
|
| |
can see it:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
----------------------
./src/H5D.c
./src/H5F.c
./src/H5FDmpio.c
./src/H5Fistore.c
./src/H5Fprivate.h
./src/H5Smpio.c
The `driver_id' for a file was accidently put in two
structs. I removed it from the H5F_file_t struct since it's
really an attribute of the VFL stuff.
More careful incrementing/decrementing the driver ID to fix a
memory leak.
./src/H5P.c
Rewrote H5Pcreate() in terms of H5P_copy() of a default
property list. This fixes some referencing counting bugs.
|
|
|
----------------------
This extensive change is the virtual file layer implementation. I've
ported and tested the sec2, family, and core drivers and only ported
the mpio driver (Albert will test it). So if you need MPIO I would
recommend sticking with the previous version for a while.
You will get a few compile warnings about split and stdio drivers not
being implemented and possibly tracing information not inserted in
some of the drivers. You can safely ignore them but I plan to fix
them.
I'm still working on the split driver because I just realized that it
needs a part of the VFL that isn't written yet.
Documentation is being updated also because there were some minor
changes (mostly just name changes). It should be available on my web
site later this week.
./MANIFEST
./src/Makefile.in
./src/hdf5.h
./src/H5Flow.c [REMOVED]
./src/H5Fstdio.c [REMOVED]
./src/H5Fsec2.c [REMOVED]
./src/H5Fsplit.c [REMOVED]
./src/H5Fmpio.c [REMOVED]
./src/H5Ffamily.c [REMOVED]
./src/H5Fcore.c [REMOVED]
./src/H5MFpublic.h [REMOVED]
./src/H5FD.c [NEW]
./src/H5FDcore.c [NEW]
./src/H5FDcore.h [NEW]
./src/H5FDfamily.c [NEW]
./src/H5FDfamily.h [NEW]
./src/H5FDmpio.c [NEW]
./src/H5FDmpio.h [NEW]
./src/H5FDprivate.h [NEW]
./src/H5FDpublic.h [NEW]
./src/H5FDsec2.c [NEW]
./src/H5FDsec2.h [NEW]
Removed/added files for virtual file layer.
./bin/trace
./src/H5.c
Removed unused public datatypes and added new VFL public
datatypes.
Changed an error message.
./config/BlankForm
./config/dec-flags
./config/gnu-flags
./config/hpux10.20
./config/hpux9.03
./config/irix5.x
./config/irix6.x
./config/solaris2.x
./config/unicosmk
Removed the H5F_OPT_SEEK and H5F_LOW_DFLT constants from the
configuration since they're no longer applicable. The default
file driver is always the sec2 driver and it always optimizes
calls to lseek() or lseek64().
./config/depend.in
C preprocessor errors generated during automatic dependency
building are sent to /dev/null to prevent them from appearing
twice in the make output.
./src/H5AC.c
./src/H5B.c
./src/H5D.c
./src/H5F.c
./src/H5G.c
./src/H5Gent.c
./src/H5Gnode.c
./src/H5HG.c
./src/H5HL.c
./src/H5O.c
./src/H5Oattr.c
./src/H5Odtype.c
./src/H5Oefl.c
./src/H5Oshared.c
./src/H5T.c
./src/H5detect.c
./test/ohdr.c
Changed H5F_ADDR_UNDEF to HADDR_UNDEF to be more consistent
with the `haddr_t' datatype which is now a public type.
./src/H5D.c
./src/H5P.c
./src/H5Ppublic.h
./src/H5Tconv.c
./test/cmpd_dset.c
./test/dsets.c
./test/overhead.c
./test/tselect.c
./test/tvltypes.c
The H5P_DATASET_XFER constant was changed to H5P_DATA_XFER
because the properties apply to all types of I/O operations,
not just datasets.
./src/H5B.c
./src/H5Bprivate.h
./src/H5D.c
./src/H5Dpublic.h
./src/H5F.c
./src/H5Farray.c
./src/H5Fistore.c
./src/H5Fprivate.h
./src/H5Fpublic.h
./src/H5Gnode.c
./src/H5Gpkg.h
./src/H5HG.c
./src/H5HL.c
./src/H5O.c
./src/H5R.c
./src/H5Sall.c
./src/H5Shyper.c
./src/H5Smpio.c
./src/H5Spoint.c
./src/H5Sprivate.h
./test/big.c
./test/h5test.c
./test/istore.c
./testpar/t_dset.c
./testpar/t_file.c
./tools/h5debug.c
./tools/h5ls.c
Modified to work with the virtual file layer by calling H5FD_*
functions instead of H5F_low_* functions and by passing file
access and data transfer properties by object ID instead of
pointer.
Changed H5D_transfer_t to H5FD_mpio_xfer_t since the
COLLECTIVE vs. INDEPENDENT transfer mode is specific to the
MPIO file driver.
Moved MPIO-specific stuff into the MPIO driver.
./src/H5B.c
./src/H5D.c
./src/H5Fprivate.h
The H5F_mpio_* private functions were renamed and placed in
the H5FDmpio driver except those which appeared in H5Smpio.c.
./src/H5E.c
./src/H5Epublic.h
Added major error number H5E_VFL for virtual file layer
related errors.
./src/H5F.c
./src/H5Fprivate.h
Changed the logic that controls whether the boot block is
written. Instead of assuming that the first call to write the
boot block is only to allocate space, I've added a function
argument which makes this explicit.
Changed the way files are compared so that a driver-defined
comparison function can be called. Files which belong to
different drivers are always considered different.
Removed H5F_driver_t since file drivers are now identified by
object ID instead of a special non-user-extendible datatype.
Removed all the hard-coded low-level file properties which
have been replaced by the various file drivers.
./src/H5I.c
./src/H5Iprivate.h
Added the H5I_inc_ref() which was removed a few months ago
since we finally have a use for it.
./src/H5Ipublic.h
Added the H5I_VFL object ID type to identify file drivers in
the virtual file layer.
./src/H5MF.c
./src/H5MFprivate.h
Moved all the allocation/deallocation code into the virtual
file layer which allows file drivers to override much of it.
./src/H5P.c
./src/H5Ppublic.h
Moved file driver-specific code into the various file driver
files.
The H5Pcopy() and H5Pclose() functions make calls into the
virtual file driver to manage the memory for driver-specific
file access and data transfer properties.
./src/H5private.h
./src/H5public.h
The `haddr_t' type is now public.
./test/tfile.c
Added a few more comments.
|
