| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
| |
Refactored free space manager to use metadata cache for serialized free
space sections. This speeds up the fractal heap test considerably...
Tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4/32 (chicago)
Linux 2.4/64 (mir)
Mac OS X (amazon)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) Added trace file support to the metadata cache. This allows capture
of all metadata cache calls in trace files for purposes of optimization
and debuging.
2) Added an expunge entry function. This allows an entry to be deleteded
from the cache without writing it to disk even if it is dirty.
3) Added a function call to resize pinned entries.
4) Added code to deal with entries that are dirty on load. This is
needed in support of a bug fix which can alter object headers on
load to repair files.
5) Added progress reporting code to the "MDC API smoke check" test in
cache_api.c. To enable the progress reporting, set report_progress
to TRUE in mdc_api_call_smoke_check().
Tested with h5committest, and a parallel test on phoenix (dual athelon
linux box).
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Trim trailing whitespace in Makefile.am and C/C++ source files to make
diffing changes easier.
Platforms tested:
None necessary, whitespace only change
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) Check in potential fix to unreproduceable bug in t_cache observed on
Cobalt
2) Check in code supporting added pinned entry features in the metadata
cache.
Description:
1) Elena encountered a bug in t_cache when running on Cobalt. On
investigation I found a potential race condition in t_cache that
could explain her observation.
2) Quincey requested additions to the metadata cache allowing renaming
of pinned entryies, and marking of pinned or protected entries as
dirty.
Solution:
1) Modified t_cache.c to prevent the potential race condition. Elena
was unable to reproduce the bug on Cobalt, so we don't know if my
fix actually addressed the problem.
2) Added code supporting the requested functions. The changes were
relatively trivial, but required substantial new test code.
Platforms tested:
h5committest, serial test on heping, parallel test on phoenix.
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code checkpoint
Description:
Checkpoint fractal heap improvements, as well as move the free space
manager code that it's using into a separate package.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4/64 (mir) w/C++ & FORTRAN
Linux 2.4/32 (heping)
Solaris 2.9 (shanti)
AIX 5.? (copper) w/FORTRAN & parallel
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup/feature twist
Description:
Adjust recent H5AC routines to take H5F_t pointers instead of H5C_t
pointers, to match the rest of the H5AC routines.
This change propagated into a few of the tests, which also had some
compiler warnings cleaned up...
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Linux 2.4/64 (mir) w/C++ & FORTRAN
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add pinned entry capability to cache.
Description:
For frequently accessed cache entries, the protect/unprotect overhead
is sometimes a bottleneck.
Solution:
Allow entries to be pinned in the cache. Pinned entries can't be
evicted, but can be flushed or modified.
Platforms tested:
h5committested -- minus one small typo in test/cache.c whose fix was
tested on copper and heping only.
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code checkpoint
Description:
The fractal heap code is reasonably stable and is passing all its
current tests, so checkpoint everything with CVS.
Also, add "statistics" module for v2 B-trees code, which is only a
stub right now.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX (amazon)
Linux 2.4 (chicago)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature & code cleanup
Description:
Update fractal heap to be able to insert objects into a direct block
hanging off the header.
Extract "octal dump" code into separate routine so that both the local
heaps & fractal heaps can use it.
Other code cleanups & support to get this far.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Check in initial "fractal heap" code, for supporting the group redesign.
Also, remove some remnants of the segmented heap/block tracker/B+tree code
which I didn't find during the last pass.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX (amazon)
Linux 2.4
|
|
|
|
|
|
|
|
|
|
|
|
| |
Maintenance
Description:
Remove half-built (half-baked? :-) implementation of B+tree, block tracker
and segmented heaps. (In preparation for starting the new heap coding)
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (heping)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) Add parallel test for metadata cache
2) Split serial test for metadata cache into two parts
3) Fix bug in which cache was flushed needlessly when the
cache wasn't full.
4) Performance improvements
5) Update API for parallel cache coherency bug fix.
Description:
See above.
Solution:
See above.
Platforms tested:
h5committest
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Repair synchronization bug in the metadata cache in PHDF5
Also repair numerous other bugs that surfaced in testing the
bug fix.
Description:
While operations modifying metadata must be collective, we allow
independant reads. This allows metadata caches on different processes
to adjust to different sizes, and to place the entries on their dirty
lists in different orders. Since only process 0 actually writes
metadata to disk (all other processes thought they did, but the writes
were discarded on the theory that they had to be collective), this made
it possible for another process to modify metadata, flush it, and then
read it back in in its original form (pre-modification) form. The
possibilities for file corruption should be obvious.
Solution:
Make the policy that only process 0 can write to file explicit, and
visible to the metadata caches. Thus only process 0 may flush dirty
entries -- all other caches must retain dirty entries until they are
informed by process 0 that the entries are clean.
Synchronization is handled by counting the bytes of dirty cache entries
created, and then synching up between the caches whenever the sum
exceeds an (eventually user specified) limit. Dirty metadata creation
is consistent across all processes because all operations modifying
metadata must be collective.
This change uncovered may bugs which are repaired in this checkin.
It also required modification of H5HL and H5O to allocate file space
on insertion rather than on flush from cache.
Platforms tested:
H5committest, heping(parallel & serial)
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Initial checkin of B+ Tree code.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Refactor metadata cache to merge "dirtied" flag in with other flags for
H5AC_unprotect and H5C_unprotect.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Interim checkin of code changes moving management of the is_dirty flag
into the cache code.
Description:
Prior to this checkin, management of the is_dirty flag was handled
above the level of the metadata cache. This can no longer be allowed,
as it introduces a race condition in the proposed fix for a cache
coherency bug in PHDF5.
Solution:
Move management fo the is_dirty flag to the cache code proper.
Entries are now marked as dirty via a flag on the unprotect call.
Platforms tested:
h5committested
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove C99 types from new metadata cache related API calls
Description:
Windows (and perhaps others) don't like int32_t and int64_t. While
we have dealt with the issue internally, it is more of a problem
in API calls.
Solution:
Convert int32_t to int and int64_t to long int in the
H5AC_cache_config_t structure used by the new metadata cache
related API calls. Added explicit type casts to convert
between internal and external representations.
Platforms tested:
h5committested
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add API calls allowing user control of the metadata cache.
Description:
Prior to this update, the metadata cache was not configurable
from outside the library.
Solution:
Add API calls allowing the user to configure the metadata cache
either at file open time, or for any open file. Also added calls
permitting the user to monitor cache size and hit rate. These
latter facilities are needed for "manual" cache size control
Platforms tested:
h5committested
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Add first iteration of "segmented heap" code, which will be used to store
links in groups in a more flexible way than the previous "local heap" mechanism.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Solaris 2.9 (shanti)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Add new "block tracker" data structure to library, for tracking blocks of
bytes in a file. Block trackers will be used to keep track of the blocks
belonging to the soon-to-be-implemented "segmented heap" which is designed to
replace the current local & global heaps (starting with the local heap).
Block trackers will also keep track of the free space in the segmented heap
and someday could be used to track the free space in the entire HDF5 file.
They are implemented as a small header of information to cache the state
of the blocks (max & min sizes of blocks tracked, etc.) and the records of
the blocks themselves are stored in a v2 B-tree.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Solaris 2.9 (shanti)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Expand v2 B-tree code to support splitting the root node when enough
records are inserted and move metadata cache callbacks into their own source
file.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor for h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Add basic code for new B-tree implementation. They don't do much yet,
aren't hooked up to anything yet and the format may change, but I'd like to
start getting them into the daily tests.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) Provide facilities in cache to allow us to avoid a potential cache
consistency bug in the parallel case.
2) Clean up a off by one sanity checking bug.
3) Turn off execution of long running tests in debug mode.
Description:
1) In the parallel case, all writes to metadata must be collective,
but reads may not be. In pricipal, this allows us to different
contents in different caches. This isn't a problem as long as the
correct data is always on disk, but unless we can force certain
writes immediately, that need not be the case.
2) & 3) should need no further explanation.
Solution:
1) Add code allowing us to mark cache entries, and then force
these entries to be flushed at a later time.
Note that to actually avoid the bug, we will have to modify
existing code to use these new features.
2) & 3) should need no further explanation.
Platforms tested:
heping (serial debug and production)
committest (copper, sol, and heping). test failed on heping in the
c++ portion of the build, but at Quincey's siggestion, I am proceeding
with the checkin.
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Clean up minor warnings and align with release branch.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & FPH5
Solaris 2.7 (arabica) w/production mode
Linux 2.4 (heping) w/C++ and FORTRAN
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Description: Cache was running too slowly.
Solution: Added a hash table for indexing. Retained the tree, but
only for dirty entries. As we need to flush dirty entries
in increasing address order, this is sufficient.
Updated statistics collection code for the above.
Converted a number of local functions into macros to avoid
the function call overhead.
Added code to disable the clean and dirty LRU lists in serial
mode.
Updated test code to account for the above changes.
Platforms tested: h5committested + serial, parallel, and fp on Eirene.
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
performance.
Description:
Replaced the old metadata cache with a cache with a modified LRU
replacement policy. This should improve the hit rate.
Solution:
Since we want to flush cache entries in increasing address order, I
used the threaded binary B-tree code to store the cache entries.
There is a fair bit of overhead here, so we may want to consider
other options.
While the code is designed to allow the support of other replacement
algorithms, at present, only a modified version of LRU is supported.
The modified LRU algorithm requires that a user selectable portion
of the cache entries be clean. The clean entries are evicted first
when writes are not permitted. If the pool of clean entries is used
up, the cache grows beyond its user specified maximum size. The
cache can also exceed its maximum size if the combined size of the
protected (or locked) entries exceeds the maximum size of the cache.
Platforms tested:
eirene (serial, parallel, fp), h5committested
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
A lot of modifications for the FPHDF5 stuff:
H5AC.c
H5ACprivate.h - Removed AC_find (it's replaced with AC_protect
and AC_unprotect). Added flushing if it's an FPHDF5 driver and
we're doing an AC_set or AC_unprotect with the dirty flag set.
H5B.c - Split up the B_flush function into different functions
since the one function was doing serialization which is better
left as a separate entity.
H5D.c - Removed some FPHDF5 code that was incorrect
H5F.c - Split up the F_flush function so that it no longer
allocates file space. Created new functions (F_init_superblock,
F_read_superblock, and F_write_superblock) for greater modularity
and so that the FPHDF5 non-captain processes can read the
superblock after the captain process writes it.
H5FD.c - Error message correction.
H5FDfphdf5.c - Removed MPI barrier call that wasn't needed.
H5FPclient.c
H5FPserver.c - Modified so that if a process requests data that
isn't exactly aligned, we can return it if we have the block that
contains the requested address.
H5G.c
H5Gent.c
H5Gnode.c
H5HL.c
H5HLpkg.h
H5HLprivate.h
H5Oefl.c - Removed the H5HL_peek function since it was doing a
(now unsafe) holding of the information in the cache. Replaced
with protect and unprotect calls.
H5TB.c - Error fix. The TB_dless function wasn't working
properly.
H5Gstab.c - Format change.
H5err.txt
H5Edefin.h
H5Einit.h
H5Epubgen.h
H5Eterm.h - Added new error code.
Platforms tested:
Modi4 (paralle, Fortran)
Sol (Fortran)
Linux (C++, Fortran)
Copper (Parallel, Fortran)
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
Added extra parameter to the H5AC_protect() function that indicates
if the cache being asked for is going to be written to or just read
from. Those AC_protect calls that were H5AC_find calls are now
read-only. The rest are writes.
Platforms tested:
Linux (h5committest not needed due to size and small impact)
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Removal of H5AC_find()
Description:
The H5AC_find() function is mostly redundant and with the new
Flexible Parallel HDF5 stuff, we need to do locking on metadata
returned from the H5AC_find() anyway. So, all of the locking stuff
will be placed in the H5AC_{un}protect() functions. The H5AC_find()
is no longer needed.
Solution:
Replaced all H5AC_finds with H5AC_protects and H5AC_unprotects.
Platforms tested:
Linux (Fortran & C++)
Solaris (Fortran)
Irix (Parallel & Fortran)
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix
Description:
The metadata cache 'destroy' callback routines need the file handle in
order for certain callback routines (currently just the H5HG one) to perform
extra cleanups. The recent change to call the 'destroy' callback from the
'clear' callback omitted this parameter.
Solution:
Add the file handle to the metadata cache 'clear' callbacks.
Platforms tested:
FreeBSD 4.8 (sleipnir)
too small to need h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
Added an extra flag to the clear functions that triggers a "destroy"
of the object being cleared if necessary. This is a fix for the
FPHDF5 stuff which had an object sticking around after it was
cleared. (In FPHDF5, some processes are in charge of destroying the
object, but all processes might actually allocate the object.)
Platforms tested:
AIX (Copper: Fortran & C++)
Linux (Verbena: Fortran & C++)
IRIX (Modi4: Parallel & Fortran)
(Sol is down)
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Further code cleanups suggested by lint tool.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Break some of the "debugging" routines into their own module, so they
aren't pulled into every executable, which certainly isn't going to use them.
Platforms tested:
h5committested
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Limit the scope on more function prototypes/macros/typedefs.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest not necessary.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup, bug fix & new feature.
Description:
Moved some "package-scope" information out of the private header file and
into the code module. (code cleanup).
Fixed a few places where the "destroy but don't flush" code wasn't working
correctly, which mostly affects the FPH5 effort. (bug fix).
Added the ability to "unprotect" a metadata object and delete it at the
same time. (new feature).
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
Linux 2.4 (burrwhite) w/FORTRAN
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/parallel & FORTRAN
(h5committest not run due to my ongoing difficulties with C++ on burrwhite).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Finish code cleanup
Description:
Wrap up the conversion of H5F_flush's multiple boolean flags into a single
bitfield of flags by pushing the flags down into the H5AC_flush and
H5F_istore_flush routines.
Also, changed the flags from H5_FLUSH_<foo> to H5F_FLUSH_<foo> to be more
consistent with rest of library.
And reverted the changes to H5FDflush and H5FD_flush routines.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Solaris 5.8 (sol)
IRIX64 6.5 (modi4) w/parallel
Misc. update:
|
|
|
|
|
|
|
| |
Code cleanup (sorta)
Description:
Updated to new copyright information as I browsed these files.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
|
|
|
|
|
|
|
|
|
|
|
|
| |
Purpose:
__DLL__ is a keyword in some platforms and __DLL__ is also defined as a macro for windows DLL applications.
That causes problems.
Description:
Solution:
Use H5_DLL*** to replace __DLL***__ at all header files.
Change the macro defination at H5api_adpt.h.
Platforms tested:
linux2.2.18smp, irix64, solaris 2.7 and windows 2000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix
Description:
It was possible to create corrupted metadata information (either in memory
or in the file or both) with a parallel I/O program because of the way
metadata writes were being handled for writes out of the metadata cache.
Solution:
Added a dataset transfer property called "block before metadata write"
which is used by the MPI-I/O and MPI-posix drivers to sync up all the
processes before attempting a metadata write. This property is currently
only for metadata writes from the metadata cache.
Platforms tested:
IRIX64 6.5 (modi4) w/parallel
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup (sorta)
Description:
When the first versions of the HDF5 library were designed, I remembered
vividly the difficulties of porting code from a 32-bit platform to a 16-bit
platform and asked that people use intn & uintn instead of int & unsigned
int, respectively. However, in hindsight, this was overkill and
unnecessary since we weren't going to be porting the HDF5 library to
16-bit architectures.
Currently, the extra uintn & intn typedefs are causing problems for users
who'd like to include both the HDF5 and HDF4 header files in one source
module (like Kent's h4toh5 library).
Solution:
Changed the uintn & intn's to unsigned and int's respectively.
Platforms tested:
FreeBSD 4.4 (hawkwind)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
Changed
#include <hdf_file.h>
construct to
#include "hdf_file.h"
so that the GNU compiler can more easily pick up the dependencies
which it places in the .depend and Dependencies files. Also
regenerated the Dependencies to go along with this.
Platforms tested:
Linux
|
|
|
|
|
|
| |
reduce malloc
abuse.
|
|
|
|
|
|
|
|
| |
objects being
cached instead of in separate structures. This reduces the amount of memory
the hash table uses by about half. This is the initial step along the path of
speeding up the metadata caching.
|
|
|
|
| |
values returned.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
----------------------
./src/H5.c [1.3]
./src/H5AC.c [1.3]
./src/H5ACprivate.h [1.3]
./src/H5B.c [1.3]
./src/H5Bprivate.h [1.3]
./src/H5D.c [1.3]
./src/H5F.c [1.3]
./src/H5Farray.c [1.3]
./src/H5Fcore.c [1.3]
./src/H5Ffamily.c [1.3]
./src/H5Fistore.c [1.3]
./src/H5Flow.c [1.3]
./src/H5Fmpio.c [1.3]
./src/H5Fprivate.h [1.3]
./src/H5Fsec2.c [1.3]
./src/H5Fsplit.c [1.3]
./src/H5Fstdio.c [1.3]
./src/H5G.c [1.3]
./src/H5Gent.c [1.3]
./src/H5Gnode.c [1.3]
./src/H5Gprivate.h [1.3]
./src/H5Gstab.c [1.3]
./src/H5HG.c [1.3]
./src/H5HGprivate.h [1.3]
./src/H5HL.c [1.3]
./src/H5HLprivate.h [1.3]
./src/H5MF.c [1.3]
./src/H5MFprivate.h [1.3]
./src/H5O.c [1.3]
./src/H5Oattr.c [1.3]
./src/H5Ocont.c [1.3]
./src/H5Odtype.c [1.3]
./src/H5Oefl.c [1.3]
./src/H5Olayout.c [1.3]
./src/H5Oprivate.h [1.3]
./src/H5Oshared.c [1.3]
./src/H5Ostab.c [1.3]
./src/H5P.c [1.3]
./src/H5R.c [1.3]
./src/H5Smpio.c [1.3]
./src/H5T.c [1.3]
./src/H5Tvlen.c [1.3]
./src/H5private.h [1.3]
./test/dtypes.c [1.3]
./test/gheap.c [1.3]
./test/istore.c [1.3]
./test/lheap.c [1.3]
./test/ohdr.c [1.3]
./tools/h5debug.c [1.3]
File addresses (the `haddr_t' type) are passed by value
instead of by reference. The type is no longer a struct. This
is one of the preliminary changes needed for the Virtual File
Layer stuff.
./src/H5Fprivate.h [1.3]
./src/H5Flow.c [1.3]
Some address functions were rewritten as macros.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
----------------------
./src/H5F.c
./src/H5private.h
./src/H5Ipublic.h
./src/H5O.c
Fixed a rather nasty bug with file closing that caused the
file boot block to be updated incorrectly, effectively
truncating the file. The bug I fixed was triggered by:
1. Create a file, F
2. Open an object, X
3. Close file F
4. Reopen file F for read/write.
5. Create and close some objects
6. Close file F
7. Close library (exit).
Step 3 pended the close because object X is still open, but
the file ID was removed from the H5I_FILE ID group. Step 4
created a new file because it didn't see any matching file on
the H5I_FILE ID group. Step 5 extends the file. Step 6 writes
the new file boot block to disk. Step 7 closes object X and
completes the close from step 3, writing the old boot block
information to disk.
The new behavior is that step 3 moves the file from the
H5I_FILE group to the H5I_FILE_CLOSING group. Step 4 searches
both groups and finds the file. Step 5 extends the file using
the same H5F_file_t struct as step 3. Step 6 closes the H5F_t
struct opened in step 3 but not the H5F_file_t struct shared
by steps 1 and 3. Step 7 closes object X which closes the
H5F_file_t from step 1, flushing the boot block which was
shared by all steps.
./src/H5F.c
Added some bulletproofing to file reference counting and
removed comments which no longer apply. Added H5F_flush_all()
and H5F_close_all() which apply to all files.
./src/H5A.c
./src/H5D.c
./src/H5F.c
./src/H5G.c
./src/H5I.c
./src/H5Iprivate.h
./src/H5R.c
./src/H5RA.c
./src/H5S.c
./src/H5T.c
Added the new H5I_free_t data type to describe the function
type to be passed as the `free_func' argument to
H5I_init_group().
./src/H5I.c
Bulletproofed the object removal functions. Removed comments
which no longer apply. Changed global variable names so they
don't violate the naming scheme. Added H5I_debug() that prints
the contents of an ID group. Removed H5I_inc_ref() because it
isn't used. Reindented a couple of functions.
./src/H5.c
./src/H5G.c
./src/H5Ipublic.h
Changed H5I_MAXID to H5I_NGROUPS to better relect the fact
that it's the total number of valid ID groups.
./src/H5Shyper.c
Changed hyperslab offset arrays to signed quantities to get
rid of warnings on DEC cluster.
./src/H5Flow.c
./src/H5Fprivate.h
Changed the objno argument of H5F_addr_pack() to be unsigned
to get rid of warnings on DEC cluster.
|