| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Fix logic error in previous checkin and also finish refactoring I/O
initialization, including simplifying all the collective & parallel cases
into a more unified mechanism.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/ & w/o parallel
Linux 2.4 (mir)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup & bug fix
Description:
Refactor recent collective chunk changes to simplify them. Also, compute
the number of chunks by checking the number of objects in the chunk skip list
instead of getting the boundaries of the selection and counting the chunks that
it overlaps, which is faster and computes correct number for certain selections.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor to require h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Refactor, simplify and cleanup Kent's recent checking for collective
chunk I/O. There's a bug that I need to talk to Kent about and some more
cleanups still, but this is reasonable for an interim point.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor for h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
bug fix for collective chunk IO, phase 1
Optimization hasn't been done yet, the collective chunk IO bug should be fixed.
Description:
In chunking storage, memory space and file space will be remapped, So to check
whether file space and memory space are regular in order to use optimized MPI derived
datatype for collective call one has to check per-chunk wise instead of per hyperslab wise.
Even a regular memory space will be stored in span-tree and will be irregular before chunk IO.
Solution:
1. Check file space and memory space per chunk wise instead of per hyperslab wise.
2. For collective IO mode, number of chunks covered by hyperslab may be different. Since we are
handing per chunk per IO, for the extra chunk IO for some(not all) processors, collective mode will
cause program hanged. So for the extra chunk Io mode independent IO has to be used.
3. On some platforms, Complex MPI derived datatype is not working, so we have to use independent IO for collective IO mode if the selection is irregular. However, when the selection is regular, we do want to use collective IO since that will improve performance. Special cares have to be added for this case.
Platforms tested:
copper(AIX 5.1) Linux(heping mpich 1.2.6), Teragrid machine, Cobalt(altix), modi4
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adding codes for the general MPI derived datatype in order to
better incorporate new fixes of HDF5 library.
Description:
Note: These codes have not been tested for general use.
Don't call these functions in your developments of the HDF5 library.
Also these codes are stand-alone codes, they should not affect
other library codes.
Solution:
Platforms tested:
Heping(C and Parallel linux 2.4, mpich 1.2.6)
Arabica(C,C++,Fortran, Solaris 2.7)
Copper(C,c++,Fortran, AIX 5.1, NOTE: c++ FAILED, seems not due to the recent check-in)
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix
Description:
Relax restrictions on parallel I/O to allow compressed, chunked datasets
to be read in parallel (collective access will be degraded to independent
access, but will retrieve the information still).
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
IRIX64 6.5 (modi4)
h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix & code cleanup
Description:
More dataset cleanups to get to a point where we can fix the chunked I/O
bug.
Also fix a couple of errors in the recent file object resurrection changes
which should hopefully address the recent daily test failres (H5T.c)
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix/code cleanup
Description:
Clean up raw data I/O code to bundle the I/O parameters (dataset, DXPL ID,
etc) into a single struct to pass around through the dataset I/O routines,
since they are always passed together, until very near the bottom of the I/O
stack.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
IRIX64 6.5 (modi4)
h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Purpose:
Feature
Description:
Datatypes and groups now use H5FO "file object" code that was previously
only used by datasets. These objects will hold a file open if the file
is closed but they have not yet been closed. If these objects are unlinked
then relinked, they will not be destroyed. If they are opened twice (even
by two different names), both IDs will "see" changes made to the object
using the other ID.
When an object is opened using two different names (e.g., if a dataset was
opened under one name, then mounted and opened under its new name), calling
H5Iget_name() on a given hid_t will return the name used to open that hid_t,
not the current name of the object (this is a feature, and a change from the
previous behavior of datasets).
Solution:
Used H5FO code that was already in place for datasets. Broke H5D_t's, H5T_t's,
and H5G_t's into a "shared" struct and a private struct. The shared structs
(H5D_shared_t, etc.) hold the object's information and are used by all IDs
that point to a given object in the file. The private structs are pointed
to by the hid_t and contain the object's group entry information (including its
name) and a pointer to the shared struct for that object.
This changed the naming of structs throughout the library (e.g., datatype->size
is now datatype->shared->size). I added an updated H5Tinit.c to windows.zip.
Platforms tested:
Visual Studio 7, sleipnir, arabica, verbena
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bug fix
Description:
1 - Dataset contiguous storage cache information had a bug where it was
possible to try to access invalid cache information if the cache wasn't filled
the first time it attempted to loop through the list of offset/length vectors.
2 - Additionally, the contiguous storage cache information was being used
in certain circumstances from the chunked dataset I/O code path, which was
generally fatal since the chunk storage and contiguous storage information
were stored together in a union.
Solution:
1 - Avoid special case of first trip through loop over offset/length
I/O vectors and always check for the contiguous storage sieve buffer buffer
being NULL.
2 - Change the union containing the chunk and contiguous storage cache
information into a struct, allowing both to be used at the same time.
Platforms tested:
FreeBSD 4.10 (sleipnir)
h5committested
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Clean up collective chunking code a bit.
Also, add '--enable-instrument' configure flag to have a mechanism for
determining that optimized operations happened correctly in the library (instead
of just the "normal" way) by allowing 'flag' properties to be set outside the
library and set when the "right" thing happens. This is mainly for debugging
and regression checks, so we make certain we don't break optimized I/O by
accident. It's enabled by default when --enable-debug is on (which is on by
default in the development branch and off by default in the release branch),
but can also be independently controlled with its own configure flag.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
IBM p690 (copper) w/parallel
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code optimization
Description:
Set up datatype ID for dataset's datatype on disk. This allows us to avoid
repeatedly copying the datatype when an ID is needed.
Also, clean up a few warnings in various other places.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.10 (sleipnir) w/parallel
Too minor to require h5committest
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Purpose:
HDF5 now supports SZIP with no encoder.
Description:
SZIP can be configured to have both encoder and decoder or just to have the decoder. HDF5 can now query the configuration of any filter, and will throw errors if users try to write using a filter with encoding disabled.
Solution:
Added H5Zget_filter_info function, changed API for H5Pget_filter and H5P_get_filter_by_id. See SZIP RFC.
Platforms tested:
Copper (fortran, C++, parallel), Sleipnir (C++), Arabica (fortran, C++), Verbena (fortran, C++)
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Refactor code
Description:
Move chunk and contiguous cached raw data from file information to dataset
information. This simplifies a number of internal interfaces, aligns the
code with it's purpose better and should allow more optimizations to the
chunked data I/O performance.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.10 (sleipnir)
h5committest
Misc. update:
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code optimization & bug fix
Description:
When dimension information is being stored in the storage layout message
on disk, it is stored as 32-bit quantities, possibly truncating the dimension
information, if a dimension is greater than 32-bits in size.
Solution:
Fix the storage layout message problem by revising file format to not store
dimension information, since it is already available in the dataspace.
Also revise the storage layout data structures to be more compartmentalized
for the information for contiguous, chunked and compact storage.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
Solaris 2.7 (arabica)
h5committest
|
| |
|
|
|
|
|
|
|
|
|
| |
Code optimization
Description:
Query the dataset creation and transfer properties less often.
Platforms tested:
Solaris 2.7 (arabica)
h5committested
|
| |
|
|
|
|
|
|
|
|
|
| |
Code cleanup
Description:
Limit the scope on more function prototypes/macros/typedefs.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest not necessary.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Code cleanup.
Description:
Move many package or internal function prototypes and macro definitions
into tighter scope according to their current use.
Added more comments where appropriate.
Eliminate ancient, unused functions.
Added a couple "accessor" functions to get parts of data structures which
were moved out of scope.
Platforms tested:
h5committested
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New feature/enhancement
Description:
Chunked datasets are handled poorly in several circumstances involving
certain selections and chunks that are too large for the chunk cache and/or
chunks with filters, causing the chunk to be read from disk multiple times.
Solution:
Rearrange raw data I/O infrastructure to handle chunked datasets in a much
more friendly way by creating a selection in memory and on disk for each chunk
in a chunked dataset and performing all of the I/O on that chunk at one time.
There are still some scalability (the current code attempts to
create a selection for all the chunks in the dataset, instead of just the
chunks that are accessed, requiring portions of the istore.c and fillval.c
tests to be commented out) and performance issues, but checking this in will
allow the changes to be tested by a much wider audience while I address the
remaining issues.
Platforms tested:
h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
./src/H5D.c
./src/H5Dprivate.h
./src/H5Dpkg.h [REMOVED]
./src/H5Tpkg.h
./src/H5Tprivate.h
Rearranged some header stuff so H5A doesn't have to include
the package-private header files that don't belong to it.
./src/H5E.c
Added the H5E_ATTR message to the major error struct.
|
| |
|
