| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
Journaling feature.
|
| |
|
|
|
|
|
|
|
| |
Moved them from
$bin_PROGRAMS which are installed to $check_PROGRAMS which are built always
but not necessary tested automatically.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
/bin/tar: Argument list too long.
After some digging, I found that the tar arguments list generated from
MANIFEST was over 138940 bytes long. This was due to both more number
of files (2153) in the manifest to be distributed and the version name
(hdf5-1.9.8-metadata_journaling_a1) was a bit long. I changed the version
name to hdf5-1.9.8-MDJ_a1 which is 16 bytes shorter, resulting the argument
list to be 34448 bytes shorter. That seems acceptable to kagiso as the
tar command ran successfully.
So, I abbreviated the version name. This is a temporary fix. A real fix
would be to use consecutive tar commands, each with a reasonably shorter
list of arguments to generate the final tar file.
|
| |
|
|
|
|
|
|
| |
counter value
and continue adding more rows from there.
|
|
|
|
|
|
| |
prepare for
an Alpha1 release.
|
|
|
|
|
|
|
|
|
|
|
|
| |
a separated function for easier identification.
Removed the redundent call of H5Pset_libver_bounds().
Changed return(-1) to return(1) since main can't return negative
value, at least not in Unix systems.
-This line, and those below, will be ignored--
M enable_journaling.c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) Removed references to H5AC2__CURR_JNL_CONFIG_VER from H5Fget_jnl_config()
and H5Pget_jnl_config(), and also references to
H5AC__CURR_CACHE_CONFIG_VERSION from H5Fget_mdc_config() and
H5Pget_mdc_config().
2) Removed H5Pset_journal() from H5C2journal.c, and modified test
code to use H5F/Pget/set_mdj_config() instead.
3) Implemented support for callbacks on metadata journaling status change
along with the associated registration / deregistration calls and
associated test code.
4) Fixed bug in journaling shutdown exposed by 3 above.
Tested and passed on Phoenix (serial), Linew (serial), and Kagiso (parallel).
However, while I was testing there were a couple of checkins, forcing
an update and second round of testing.
On the second round, tested and passed on Phoenix (serial) and
Kagiso (parallel), but failed on Linew (serial).
As best I can tell, this was caused by Mike M's checkin -- which
broke the smoke checks in cache2_journal on Linew but not Phoenix
or Kagiso. A typical delta in the architype files follows:
linew.hdfgroup.uiuc.edu% diff -ctw cache2_journal_sc00_000.jnl tmp/cache2_journal_sc00_000.jnl
*** cache2_journal_sc00_000.jnl Fri Aug 22 08:28:49 2008
--- tmp/cache2_journal_sc00_000.jnl Fri Aug 22 05:08:41 2008
***************
*** 1,5 ****
! 0 ver_num 1 target_file_name cache_journal_test.h5 creation_date Fri Aug 22 human_readable 1
! E eoa_value 0x0
C comment Begin transaction on transaction 1.0.
1 bgn_trans 1
2 trans_num 1 length 1 base_addr 0x401 body 01
--- 1,5 ----
! 0 ver_num 1 target_file_name cache_journal_test.h5 creation_date Wed Aug 20 human_readable 1
! E eoa_value 0x772a9c01
C comment Begin transaction on transaction 1.0.
1 bgn_trans 1
2 trans_num 1 length 1 base_addr 0x401 body 01
As you can see, it looks like garbage is getting into the first
eoa write on Linew.
I'm checking in anyway, as Quincey needs my changes, and I will not
have time to work on this for several days.
Mike: Let me know if you are tackling this one -- if not, I'll deal with it.
|
|
|
|
|
|
|
|
| |
flush to
allow manual abort of the application, thus simulate a crash. If program
is NOT aborted, it will complete successfully. Previously, it would
abort itself for -w but that is removed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- EOA logging update
Description:
- EOA values will now be written to the journal
file in their own transaction when the EOA
changes.
- The EOA will be udpated in the HDF5 file's
superblock before the recovery process begins.
This should prevent some loss of raw data as the
file won't be getting truncated upon file open as
it will read the correct EOA value from the
superblock.
- Removed storing of EOA in journal entry messages
since they're in their own transaction.
- Updated tests to reflect change of transaction
formats. Regenerated smoke test files to account
for new entry types, and tweaked transaction number
tests to reflect change in size of journal entries.
- Large testfiles (in test/testfiles) should now
unzip when ./configure is run.
- When journal file is supplied but contains no
complete transactions, instead of reporting
an error, h5recover now informs the user of said
nonexistant transactions, and opens/closes the
hdf5 file with the journal recovered flag set.
- Other various organizational changes to h5recover,
included a bit more added to verbose output.
Tested:
- kagiso, smirom
|
|
|
|
| |
Renamed the reopen mode (-r) to write mode (-w) which is more intutive.
|
|
|
|
|
|
| |
parameters.
They does not affect the results and are not needed.
|
|
|
|
|
|
|
| |
Description: converted the fractal heap header and indirect block metadata
cache clients over to use the new journaling cache callbacks.
Tested: kagiso, smirom
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
from the H5AC2_cache_config_t structure and the associated
H5P/H5Fget/set_mdc_config() API calls, and into the newly created
H5AC2_jnl_config_t structure and H5P/H5Fget/set_jnl_config() calls.
Updated test code accordingly.
Updated the trace file test code for journaling.
Also folded in a fix to an assertion bug in H5C2pkh.h
Tested serial on Phoenix and Linew, and parallel (with and without the
trace file enabled) on kagiso.
|
|
|
|
|
|
|
|
|
|
| |
addresses
for easier octo dump of data file.
Moved latest lib version to apply all the time.
Tested:
kagiso.
|
|
|
|
|
|
| |
Added -z option to turn OFF all caching.
Tested: kagiso.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Bug fixing and cleanup
Description:
- Fixed the problem where the recovery wasn't "sticking" on the first pass
of using the tool. Now should only need to run the tool once to recover
a file.
- Uncommented out some lines generating more verbose output, so now
entry size and addresses are visible with -v (--verbose) option.
- Fixed off-by-one memory allocation error.
- Cleaned up comments to allow for easier reading.
Tested:
- kagiso, smirom
|
|
|
|
|
|
|
|
| |
Description: converted the shared object header message and index stored
as a list metadata cache clients over to use the new journaling
cache callbacks.
Tested: kagiso, smirom
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Convert object header cache client to use the new metadata journaling
cache, which included adding a new client for handling continuation chunks.
Added "real" protect calls around modifying chunks in object headers.
Switched a few more metadata cache library API routines to drop the
file pointer, when it is not needed (pinning/unpinning entries, etc.)
Fixed bug in journaling cache handling of 'image_len' callbacks and
also changed cache to retry deserializing entries when the entry's size is
larger than the speculative size initially tried.
Retrying for 'image_len' callbacks has problems with the 'multi' VFD,
so the h5dump and FORTRAN 'multi' tests are commented out, until the changes to
the 'multi' VFD from the file free space branch are brought back into the
trunk.
Currently, the 'h5recover' tool has a bug which requires it to be run
twice before replaying the journal "sticks". However, this is from an earlier
checkin, since the code in the branch already has this behavior... :-(
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.4 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
|
| |
journaling,
along with test code that uncovered the problem and associated additions &
changes to the gziped test files in test/testfiles
Tested serial on Phoenix and parallel on kagiso
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
which
I broke in the last checkin.
Also added test to verify that the journal_recovered flag in fact
causes a file that is marked as having journaling in progress will
cause the file to be unmarked.
Tested serial on Phoenix and parallel on Kagiso. h5recover is still
failing, but everything else seems to pass.
|
|
|
|
|
|
|
|
| |
identify
this is a feature branch.
H5committested.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
specifically:
1) Fix for failure to detect journaling in progress on HDF5 files which
were not closed correctly. Also associated test code.
Note that this required addition of code to test for journaling in
progress and enable journaling at the end of H5F_flush(). In passing,
I was able to get rid of the wacky code that queued journaling setup
at cache creation time.
2) Test code for startup and shutdown of journaling on an open file.
3) Updates to start checking journal output against architype files
instead of just generating architypes at test time.
Note that per Quincey's request, I have checked in gziped versions
of the architype files. At some point, we will have to add code
to automatically unzip these files, but for the time being you will
have to go to test/testfiles and "gunzip *.gz". The journal tests
will still pass if you don't, but you will get a warning about
missing test files.
4) Fixed bug in journal entry logging code that allowed a comment to
appear in the journal file before the journal file header. (Mike M,:
Please review my fix to verify that I haven't clobbered anything.)
5) Additional test code.
Note that more test code would be a good idea, but this set of bug
fixes should be enough to get us through the basic demo -- at least
as far as the metadata cache is concerned.
Tested serial on Phoenix, and parallel on Kagiso. Also, tested serial
on Linew just prior to some last minute minor edits.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Disable dataset caching when journaling is enabled.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.2 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring revisions 15210:15289 from the trunk into the metadata journaling branch.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Merge revisions 15130:15210 from trunk into metadata journaling branch
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
| |
Description: converted the v2 b-tree header, internal node, and leaf node
metadata cache clients over to use the new journaling cache
callbacks.
Tested: kagiso, smirom
|
|
|
|
|
|
| |
Added -p for patch mode that is needed temporary.
Tested: kagiso
|
|
|
|
|
|
| |
view the h5dump output.
Tested: kagiso.
|
|
|
|
|
|
|
|
|
|
| |
because the journal
file is already existing.
Remove the journal file before starting the journaling tests.
Tested: h5cmmittested.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Merge revisions 15037:15130 from trunk into metadata journaling branch
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.2 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
| |
Description: Removing a broken assertion. In my last checkin I moved the
location where bodydata (in H5C2_jb__journal_entry) gets
initialized, and forget to update the assertion appropriately.
Tested: kagiso
|
|
|
|
|
|
|
|
|
| |
section info
metadata cache clients to use the new journaling cache
callbacks.
Tested on: kagiso, smirom
|
|
|
|
|
|
|
|
|
| |
Description: Journal entry data was not getting copied correctly before being
converted into hex, and thus journal entries were getting
corrupted. H5C2_jb__journal_entry now uses HDmemcpy, which should
fix the issue.
Tested: kagiso
|
|
|
|
|
|
|
|
| |
that there
will be at least 3 completed transactions.
Tested: kagiso.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Merge revisions 14900:15037 from trunk into metadata journaling branch
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.2 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
| |
Tested: kagiso.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Testing mdj smoke check 00 and Testing mdj smoke check 01 tests would
fail if testfiles/ did not exist. This was because cache2_journal.c
putting its temporary file in testfiles/ which does not always exist.
Fix:
Removed the testfiles/ part from all test file names.
Also, added the temporary file names to be cleaned during "check-clean"
request.
Tested: kagiso.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring revisions 14800:14900 from trunk into metadata journaling branch
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.2 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring back revisions 14700:14800 from the trunk
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.2 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Merge revisions 14525:14700 from trunk into metadata journaling branch
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.2 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
| |
|
|
|
|
|
|
|
|
| |
Added a -c option to do create explicitly.
Turn off data sieving in -r mode.
Fixed an error in writedata() to write data correctly now.
tested: kagison.
|
|
|
|
|
|
|
| |
Other file recovery or Journaling related documents can be kept in this file
for now. Will get them more organized later.
Tested: eyeballed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bring changes from trunk from the time the branch was created (r14280)
up to the 1.8.0 release (r14525) back into the metadata journaling branch.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
value into
journal entries to be used by the recovery tool.
This value is only really neded once per transaction, and only when
the EOA changes, so rather than putting it into each journal entry,
this should be moved into its own transaction type. However, in order
to speed testing along, this quick fix has been implemented for the
time being.
Modified h5recover tool to use eoa value as well as journaling tests
accordingly.
Tested: kagiso
|
|
|
|
|
|
|
|
|
|
|
|
| |
* How to use this:
* ./enable_journaling # create JournalEG.h5 file
* ./enable_journaling -r # reopen JournalEG.h5 with Journaling on and
* # add more rows, then crash.
* ./h5recover -j JournalEG.h5.jnl JournalEG.h5 # to recover the file.
* ./enable_journaling -p # patch it with metadata of the added rows.
* Then JournalEG.h5 should have all the expected written rows and data.
Tested: kagiso. (-r failed with a library assertion error.)
|
|
|
|
|
|
|
|
| |
H5Ppublic.h.
Added H5AC2public.h to hdf5.h.
Tested: kagiso.
|
|
|
|
|
|
|
| |
Correct 'serialize' callback to add file pointer.
Tested on:
Linux/32 2.6 (kagiso) w/parallel
|