| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
| |
Refactor function name macros and simplify the FUNC_ENTER macros, to clear
away the cruft and prepare for further cleanups.
Tested on:
Mac OSX/64 10.7.3 (amazon) w/debug, production & parallel
|
|
|
|
|
|
|
|
|
| |
unlimited. This is a follow-up checkin for r20440:
1. I added a test case of extending dataset of zero dimension size and shrinking back to zero dimension size.
2. I updated the Makefile to include the new data file to be cleaned up.
Tested on jam - relatively simple.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit metadata tagging framework.
Description:
This check-in contains a new framework whose goal is to apply
a tag value to each new entry in the metadata cache as it is created.
This tag value is such that it relates each piece of metadata to the
HDF5 object that it belongs to (dataset, group, et cetera).
This changeset includes the framework that applies the tags as
well as a suite of tests to verify correct tag application, though does
not yet make use of the tag values to flush/evict individual objects.
Please refer to the "flush/evict individual objects" RFC for further
discussion of these changes.
Tested:
jam, amani, linew (h5committest)
liberty, abe, blue print
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring changes from Coverity fixing branch to trunk:
r18235:
Fixed coverity 114:
if (NULL ==_dest)
H5MM_free(dest);
r18236:
Close Coverity issue #28 (again :-) by working through the logic of the
routine more thoroughly to eliminate the goto statements. (LK & QK)
r18237:
fixed coverity 133:
if (NULL==_dest && NULL==ret_value && NULL != dest)
H5MM_free(dest);
r18238:
Fix coverity items 421 and 422. Added assertion that the heap's free list is
NULL when entering H5HL_fl_deserialize, guarateeing that the free list will
always be linked in even on failure.
r18239:
Fix coverity item 268. Changed H5MM_xfree(read_buf) to read_buf =
H5MM_xfree(read_buf) so that read_buf isn't manipulated after it's been freed.
r18241:
coverity fix: use correct free functon H5FL_FREE()from the previous fix
r18242:
Coverity fix 139: Free dst correctly in H5O_sdspace_copy().
r18243:
Fix Coverity issue #417 by checking for NULL return value from
setup_cache() (LK & QK)
r18244:
Coverity Fix 132: free dest correctly in H5O_efl_copy()
r18245:
Issue 121: H5S_hyper_make_spans() cannot deal with counts of 0. However,
H5Sselect_hyperslab() API does allow a count of 0.
Therefore, simply throw an error if this function encounters a count of 0.
r18246:
Check file_ptr to address coverity issue # 418
r18247:
Fixed coverity 113:
if (sequence)
H5MM_xfree(sequence);
r18248:
Coverity issue #414 by checking for NULL return from setup_cache() (LK & QK)
r18249:
fixed coverity 274: moved H5FL_FREE(H5A_t, attr); to a line above so that if
attr is null, it will not free it.
r18250:
Fix coverity issue #86.
Check return of malloc function to ensure non-null before continuing.
Tested on:
Mac OS X/32 10.6.2 (amazon) w/debug & production)
(h5committested in daily tests on branch)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring r18169 from 'merge_metadata_journaling' branch back to trunk:
Converge changes on metadata_journaling branch and trunk:
- Remove H5F_t* parameter from H5HL_unprotect()
- Remove H5F_t* parameter from H5HL_dirty()
- Remove H5F_t* parameter from H5O_unprotect()
- Bring changes to metadata caching routines back:
- H5AC_pin_protected_entry()
- H5AC_resize_pinned_entry()
- H5AC_unpin_entry()
- H5AC_mark_pinned_entry_dirty()
- H5AC_mark_pinned_or_protected_entry_dirty()
- Revise internal H5C routines to drop H5F_t* and/or H5C_t* as
parameter where possible
- Revise tests to follow internal routine changes
Also, drop file/cache pointer parameter from create/destroy flush
dependency internal cache calls.
Also, other minor changes to speedup the 'cache' test.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.6.2 (amazon) in debug mode
Mac OS X/32 10.6.2 (amazon) w/C++ & FORTRAN, w/threadsafe,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring r18071 from metadata journaling merge branch:
Refactor local heap routines with changes from metadata journaling
branch, along with other misc. changes as the changes on the metadata journaling
branch are being converged with the current state of the trunk.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.6.2 (amazon) in debug mode
Mac OS X/32 10.6.2 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
|
|
|
|
|
|
|
|
|
|
|
|
| |
Description:
When an attribute was created with a datatype or dataspace that was shared in
the same object header that the attribute was in, the attribute could not be
deleted. Changes made to ensure that the attribute can be deleted both when the
attribute is in the object header and when it is shared in the heap. Object
header message decode routines now take an "open_oh" parameter to enable them to
avoid opening the same object header twice.
Tested: jam, smirom (h5committest)
|
|
|
|
|
|
|
|
| |
Clean up warnings & formatting
Tested on:
Mac OS X/32 10.5.4 (amazon)
More tests forthcoming
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
files with incorrect
datatype versions are encountered.
Description: The library now recognizes some problems with datatype versions in
H5O_decode_helper(), and, if not performing strict format checks, automatically
corrects them. Framework added for other message decode routines to
automatically correct file errors. Datatype version information added to
h5debug.
Tested: kagiso, smirom, linew (h5committest)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Check in "unique, but sharable" optimization to ISOHM code, which
allows object header messages that are only used in one object to remain in
the sole user's header, but migrates messages that are used in more than one
header into the ISOHM heap.
Tested on:
Mac OS X/32 10.4.9 (amazon)
FreeBSD/32 6.2 (duty)
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
|
|
|
|
|
|
|
|
|
|
|
|
| |
Take further advantage of the readers/writer locking in the metadata cache
and push the locking of the local heap up to the routines which call B-tree
routines, which should drastically lower the number of metadata cache protect/
unprotect calls.
Tested on:
Mac OS X/32 10.4.9 (amazon)
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fully enable readers/writer locking in metadata cache and correct errors
in cache clients which were modifying a cache entry while only holding a
read lock on it.
Tested on:
Mac OS X/32 10.4.9 (amazon)
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
FreeBSD/32 6.2 (duty)
FreeBSD/64 6.2 (liberty)
|
|
|
|
|
|
|
|
|
| |
copyright notice.
Tested platform:
Kagiso only since it is only a comment block change. If it works in one
machine, it should work in all, I hope. Still need to check the parallel
build on copper.
|
|
|
|
|
|
|
|
|
| |
Remove the "scaffolding" for shared message method invocation and
simplify the way shared messages are dealt with in general.
Tested on:
FreeBSD/32 6.2 (duty)
Mac OS X/32 10.4.8 (amazon)
|
|
|
|
|
|
|
|
| |
Push code further toward shared message method calling refactor.
Tested on:
FreeBSD/32 6.2 (duty)
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
| |
Get rid of two different types of fill value struct (merge H5O_fill_t
and H5O_fill_new_t) and clean up & simplify dataset initialization code.
(In preparation for shared object header message method call
refactoring).
Tested on:
FreeBSD/32 6.2 (duty)
Mac OS X/32 10.4.8 (amazon)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add "set_crt_index" and "get_crt_index" methods for the object header
message class.
Unify fractal heap definitions for shared messages and attributes, under
"object header" fractal heap definitions.
Initial code for adding creation order index to object header messages.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Added can_share callback for OH messages. This determines whether the
message is allowed to be shared in the heap (committed and immutable datatypes
can't be).
Fixed a bug in the dense attribute storage that tried to open the shared message
heap when it hadn't been created yet.
Made the test to extend shared dataspace messages a bit more robust.
Refactored the code the searches a shared message list index to be a little
more efficient.
A few other minor changes.
Tested on smirom, kagiso, Windows, and juniper.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add first pass of "dense" attribute storage to objects. Lots of parts of
this are stubbed out, but all the tests are passing and I'll work on the corner
cases soon.
Eliminated several unused parameters from object header message callback
routines.
Other, miscellaneous code cleanups, etc. (and probably some things I've
forgotten about... :-)
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
AIX/32 5.? (copper)
|
|
|
|
|
|
|
|
|
| |
Finish refactoring internal link deletion code, to make it possible to
wrap up the H5Ldelete_by_idx() coding.
Tested on:
Linxu/32 2.6 (chicago)
Linxu/64 2.6 (chicago2)
|
|
|
|
|
|
|
|
|
|
|
|
| |
This feature is still in progress; Shared Object Header Messages are not
complete as a feature and are not thoroughly tested. There are still
"TODO" comments in the code (comments with the word "JAMES" in them,
so as not to be confused with other TODO comments).
Hopefully this checkin will reduce the liklihood of conflicts as I finish
implementing this feature.
All current tests pass on juniper, copper (parallel), heping, kagiso, and mir.
|
|
|
|
|
|
|
|
|
| |
Clean up some of the warnings on 64-bit Linux...
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Trim trailing whitespace in Makefile.am and C/C++ source files to make
diffing changes easier.
Platforms tested:
None necessary, whitespace only change
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Check in Peter's code to add support for "shallow copy", "create
intermediate groups", "no attributes" and "expand soft links" support.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago) w/ & w/o group-revision enabled
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New/expanded features
Description:
Check in Peter's changed for the object copy code:
- Allow/fix copying datasets using named variable-length datatypes
- Start adding framework for property list to control how object
copying occurs.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Add in a combination of Peter's & my code to support copying
variable-length data from one file to another, although currently only
supported with contiguous data storage.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Check in baseline for compact group revisions, which radically revises the
source code for managing groups and object headers.
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
This initiates the "unstable" phase of the 1.7.x branch, leading up
to the 1.8.0 release. Please test this code, but do _NOT_ keep files created
with it - the format will change again before the release and you will not
be able to read your old files!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
Solution:
There's too many changes to really describe them all, but some of them
include:
- Stop abusing the H5G_entry_t structure and split it into two separate
structures for non-symbol table node use within the library: H5O_loc_t
for object locations in a file and H5G_name_t to store the path to
an opened object. H5G_entry_t is now only used for storing symbol
table entries on disk.
- Retire H5G_namei() in favor of a more general mechanism for traversing
group paths and issuing callbacks on objects located. This gets us out
of the business of hacking H5G_namei() for new features, generally.
- Revised H5O* routines to take a H5O_loc_t instead of H5G_entry_t
- Lots more...
Platforms tested:
h5committested and maybe another dozen configurations.... :-)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature
Description:
Add in baseline "object copy" code from Peter [in the form of a new API
routine: H5Gcopy()]. There's still some work to do (like handling variable-
length datatypes and possibly support for references) and it hasn't been tested
on mounted files yet, but the core functionality is there and working
correctly.
I've also got a set of patches to update the 1.6 branch with tweaks to
keep the branches mostly in sync, but Elena will kill me if I import them
before the 1.6.5 release is out... :-)
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committested
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Refactor metadata cache to merge "dirtied" flag in with other flags for
H5AC_unprotect and H5C_unprotect.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Interim checkin of code changes moving management of the is_dirty flag
into the cache code.
Description:
Prior to this checkin, management of the is_dirty flag was handled
above the level of the metadata cache. This can no longer be allowed,
as it introduces a race condition in the proposed fix for a cache
coherency bug in PHDF5.
Solution:
Move management fo the is_dirty flag to the cache code proper.
Entries are now marked as dirty via a flag on the unprotect call.
Platforms tested:
h5committested
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Description: Removed PABLO from the source
Solution:
Platforms tested: arabica with 64-bit, copper with parallel,
heping with GNU C and C++ and PGI fortran (but
I disabled hl, there is some weird problem only
on heping: F9XMODFLAG is not
propagated to the Makefile files
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup & optimization
Description:
Improve ADF/CGNS benchmark by reducing the number of internal attribute
copies made during creations, opens and writes.
Added new H5O_iterate() routine for iterating through messages of a certain
type in the object header (attributes are the only message currently that can
have multiple instances in the object header).
Cross-pollinated various minor code cleanups to reduce diffs between
branches.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
Too minor to require h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix & code cleanup
Description:
More dataset cleanups to get to a point where we can fix the chunked I/O
bug.
Also fix a couple of errors in the recent file object resurrection changes
which should hopefully address the recent daily test failres (H5T.c)
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix/code cleanup
Description:
Clean up raw data I/O code to bundle the I/O parameters (dataset, DXPL ID,
etc) into a single struct to pass around through the dataset I/O routines,
since they are always passed together, until very near the bottom of the I/O
stack.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
IRIX64 6.5 (modi4)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup & minor optimization
Description:
Re-work the way interface initialization routines are specified in the
library to avoid the overhead of checking for them in routines where there is
no interface initialization routine. This cleans up warnings with gcc 3.4,
reduces the library binary size a bit (about 2-3%) and should speedup the
library's execution slightly.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/gcc34
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix
Description:
When two property lists are compared, the H5Pequal routine was just
comparing the raw information for the property values. This causes problems
when the raw information contains pointers to other information.
Solution:
Allow a 'compare' callback to be registered for properties, so that a user
application get perform the comparison itself, allowing for "deep" compares of
the property value.
This was exported to the H5Pregister & H5Pinsert routines in the development
branch, but not the release branch.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fixes and code cleanup
Description:
Lots of changes here:
- Fixed bug #691 - when shared datatypes are used in attributes they
are incorrectly copied into the attribute instead of referring
the the named datatype in the file. This required bumping the
version of the attribute message. The new version of the attribute
message is only written out when a shared datatype is used in
the attribute. [Also, this format change made the size of the
attribute smaller.]
- Added information to attribute debugging routine so that shared
datatypes are displayed correctly with the h5debug tool.
- Refactored the H5O* routines to extract code that was common to
several routines into subroutines to call.
- Added 'link' method for H5O message sub-classes, which increments
the link count on shared objects when a message is created which
shares them.
- Corrected [unreported] bug where the link count was not being
decremented on the shared object when a object header message
with a reference to that object was deleted from the file.
- Reduced size of shared message from 49 bytes (which was incorrect
anyway and should have been 48 bytes) to 10 bytes, which required
bumping the version of "shared" messages.
- Refactored some of the shared datatype routines to allow for easier
queries of "committedness" internally to the library and also
added routine to easily increment/decrement the reference count of
a shared datatype.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update
Description:
A lot of modifications for the FPHDF5 stuff:
H5AC.c
H5ACprivate.h - Removed AC_find (it's replaced with AC_protect
and AC_unprotect). Added flushing if it's an FPHDF5 driver and
we're doing an AC_set or AC_unprotect with the dirty flag set.
H5B.c - Split up the B_flush function into different functions
since the one function was doing serialization which is better
left as a separate entity.
H5D.c - Removed some FPHDF5 code that was incorrect
H5F.c - Split up the F_flush function so that it no longer
allocates file space. Created new functions (F_init_superblock,
F_read_superblock, and F_write_superblock) for greater modularity
and so that the FPHDF5 non-captain processes can read the
superblock after the captain process writes it.
H5FD.c - Error message correction.
H5FDfphdf5.c - Removed MPI barrier call that wasn't needed.
H5FPclient.c
H5FPserver.c - Modified so that if a process requests data that
isn't exactly aligned, we can return it if we have the block that
contains the requested address.
H5G.c
H5Gent.c
H5Gnode.c
H5HL.c
H5HLpkg.h
H5HLprivate.h
H5Oefl.c - Removed the H5HL_peek function since it was doing a
(now unsafe) holding of the information in the cache. Replaced
with protect and unprotect calls.
H5TB.c - Error fix. The TB_dless function wasn't working
properly.
H5Gstab.c - Format change.
H5err.txt
H5Edefin.h
H5Einit.h
H5Epubgen.h
H5Eterm.h - Added new error code.
Platforms tested:
Modi4 (paralle, Fortran)
Sol (Fortran)
Linux (C++, Fortran)
Copper (Parallel, Fortran)
Misc. update:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix
Description:
New external file double-vectorized I/O routines were checking an assertion
on the wrong variable (essentially a typo :-).
Solution:
Check correct variable.
Platforms tested:
FreeBSD 4.8 (sleipnir)
Triple check not needed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature/enhancement
Description:
Chunked datasets are handled poorly in several circumstances involving
certain selections and chunks that are too large for the chunk cache and/or
chunks with filters, causing the chunk to be read from disk multiple times.
Solution:
Rearrange raw data I/O infrastructure to handle chunked datasets in a much
more friendly way by creating a selection in memory and on disk for each chunk
in a chunked dataset and performing all of the I/O on that chunk at one time.
There are still some scalability (the current code attempts to
create a selection for all the chunks in the dataset, instead of just the
chunks that are accessed, requiring portions of the istore.c and fillval.c
tests to be commented out) and performance issues, but checking this in will
allow the changes to be tested by a much wider audience while I address the
remaining issues.
Platforms tested:
h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature, sorta
Description:
Track changes to various internal APIs
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
Linux 2.4 (burrwhite) w/FORTRAN
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/parallel & FORTRAN
(h5committest not run due to my ongoing difficulties with C++ on burrwhite).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Clean up miscellaneous warnings which have crept into the code.
Fix "_POSIX_C_SOURCE not defined" warning on FreeBSD.
Adjust gcc compiler flags to be more concise for production mode.
Refactor the H5O code so that there is a stronger boundary between code
in the H5O package and code in the library which just calls H5O routines.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel and gcc 2.95.4 & gcc 3.2.2
Misc. update:
Update MANIFEST if you add or remove any file.
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Clean up some compiler warnings
Platforms tested:
FreeBSD 4.7 (sleipnir)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup/new feature.
Description:
Split FUNC_LEAVE into API and non-API specific versions. This allows a
solution to compiling this branch with C++, as well as reducing the size
of the binaries produced.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix/Code cleanup/New Feature
Description:
Correct problems with writing fill-values to external storage and allocate
the data storage at the correct times.
Also, mostly straighten out the strange code which allocates and fills
raw data storage for datasets. Things are still a bit odd in that the
fill-values for chunked datasets are written when the space is allocated,
instead of in a separate routine, but there are two reasons for this:
it's inefficient (especially in parallel) to iterate through all the chunks
twice, and (more importantly) the space needed to store compressed chunks
isn't known until we've got a buffer of compressed fill-values ready to
write to the chunk.
Additionally, add in the H5D_SPACE_ALLOC_INCR and H5D_SPACE_ALLOC_DEFAULT
setting for the "space time", which incorporate the previous behavior of
the space allocation for chunked datasets.
The default settings for the different types of dataset storage are now
as follows:
Contiguous - Late
Chunked - Incremental
Compact - Early
This checkin also incorporates a change to the behavior of external data
storage in two ways - fill-values are _never_ written to external storage
(under the assumption that writing fill-values is triggered by allocating
space in an HDF5 file, and since space is not allocated in the file, the
fill-values should not be written) and external data files are now created
if they don't exist when data is written to them. The fill-value will
probably need to be revisited at some time in the future, this just seemed
like the safer course currently.
I think I cleaned up some compiler errors also, before getting bogged down
in the fixes for the space allocation and fill-values.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/serial & parallel. Will be testing on IRIX64
6.5 (modi4) in serial & parallel shortly.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Changed the last HRETURN* statements in the FUNC_ENTER macros into HGOTO*
macros, which reduces the size of the library binary in certain
configurations by another 10%
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel, IRIX64 6.5 (modi4) serial &
parallel
|