summaryrefslogtreecommitdiffstats
path: root/src/H5Dmpio.c
Commit message (Collapse)AuthorAgeFilesLines
* [svn-r21925] Description:Quincey Koziol2012-02-101-20/+20
| | | | | | | | | | | | | | Bring r21919-21924 from trunk to 1.8 branch: cleanup FUNC_ENTER macros, etc. Also removed Subversion mergeinfo tags from a few directories and files. These are artifacts from non-root merges using pre-1.6 Subversion clients. (This should fix the problem of unchanged directories looking "changed" during checkins.) Tested on: Mac OSX/64 10.7.3 (amazon) w/debug & production (daily tested on trunk)
* [svn-r21797] Description:Quincey Koziol2011-12-021-33/+30
| | | | | | | Back out r21782 while I figure out what the problem is with the change. Tested on: Daily tests... :-/
* [svn-r21784] Description:Quincey Koziol2011-11-301-30/+33
| | | | | | | | | | | | Bring r21782 from trunk to 1.8 branch: Rearrange checks for reasons why we break collective I/O back to independent I/O into "global" and "local" sections. We should try to minimize the checks in the "local" section... Tested on: Mac OS X/32 10.7.2 (amazon) w/parallel (too minor to require h5committest)
* [svn-r21533] Description:Quincey Koziol2011-10-121-0/+77
| | | | | | | | | Bring r21442 and r21529 from trunk to 1.8 branch: Add new API routines for querying if collective raw data I/O was actually performed collectively. Tested: Linux/32 2.6 (jam) (h5committested on trunk)
* [svn-r21479] Merge revision 21101 from trunk to 1.8 branch:Mike McGreevy2011-10-051-342/+135
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | HDFFV-7639 Purpose: Remove H5_MPI_SPECIAL_COLLECTIVE_IO_WORKS and H5_MPI_COMPLEX_DERIVED_DATATYPE_WORKS #defines from source. Description: Two advanced parallel functionalities, special collective IO and complex derived datatypes, are not supported by older implementations of mpi, and thus our code limits the use of these features with #ifdefs and has checks in configure to set them (or not). Unfortunately, configure can't actually run a parallel check to see if these features are working (nor not) so it resorts to looking in the config files where they are explicity enabled or disabled based on versions of mpi, sytems being built on, or for no documented reason at all (i.e. just set to on or off as some 'default'). Overriding these settings is easy if need be, provided it is known that it needs to be done to get improved performance, and oftentimes it is not. Most new MPI implementations successfully handle the functionality requested when these #defines are set, and many of the "turn these features off" cases in the config files are for old (> 5 years) versions of MPI and retired systems (such as NCSA's tungsten). Therefore, the decision has been made to remove the support for these old versions of MPI and systems that cannot handle these behaviors. The #ifdefs and supporting setup in the config/ files and configure script has been removed, and the code executed when these options were not set removed from the source. In passing, this commit also cleans up some whitespace issues in both t_mpi.c and H5Dmpio.c. Furthermore, in t_mpi.c, the special collective IO test was not getting regularly run due to it being written to work only with four processes (we regularly test with six, previously with three), and thus it failed when actually run due to an out of bounds data buffer assignment. It has been modified to run at any number of processes greater than four, and the memory problem has been fixed so the test passes. Tested: Jam, h5committest, Ember
* [svn-r19093] Description:Quincey Koziol2010-07-191-58/+66
| | | | | | | | | | | | | | | | | | | | Bring r19092 from trunk to 1.8 branch: Bring "shape same" changes from LBL branch to trunk. These changes allow shapes that are the same, but projected into dataspaces with different ranks to be detected correctly, and also contains code to project a dataspace into greater/lesser number of dimensions, so the I/O can proceed in a faster way. These changes also contain several bug fixes and _lots_ of code cleanups to the MPI datatype creation code. Many other misc. code cleanup are included as well... Tested on: FreeBSD/32 6.3 (duty) in debug mode FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode (h5committested on trunk)
* [svn-r19007] Purpose: Improve performance of the the chunk cacheNeil Fortner2010-06-151-10/+12
| | | | | | | | | | Description: Previously, the chunk cache would, when looking for a chunk, alwaylook for the chunk in the b-tree before checking the cache. Reworked the functions that look up chunks to always check the cache first and avoid the b-tree lookup if the chunk is in cache. Tested: jam, linew, amani (h5committest)
* [svn-r18566] Description:Quincey Koziol2010-04-151-6/+5
| | | | | | | | | Bring r18564 & r18565 from trunk to 1.8 branch: clean up compiler warnings and whitespace issues. Tested on: Mac OS X/32 10.6.3 (amazon) (too minor to require h5committest)
* [svn-r17150] Description:Quincey Koziol2009-07-021-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | Bring r17148:17149 from trunk into 1.8 branch: Refactor how chunked dataset information is computed, moving it earlier and avoiding more recomputation. Tested on: FreeBSD/32 6.3 (duty) in debug mode FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (jam) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode Mac OS X/32 10.5.7 (amazon) in debug mode Mac OS X/32 10.5.7 (amazon) w/C++ & FORTRAN, w/threadsafe, in production mode
* [svn-r16920] Description:Quincey Koziol2009-05-051-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Merge these trunk revisions which occurred during the 1.8.3 release code freeze back to the 1.8 branch: From Quincey: 16845 16847 16849 16851 16858 16869 16897 From Ray: 16859 16880 From Allen: 16863 Tested on: FreeBSD/32 6.3 (duty) in debug mode FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (jam) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode Mac OS X/32 10.5.6 (amazon) in debug mode Mac OS X/32 10.5.6 (amazon) w/C++ & FORTRAN, w/threadsafe, in production mode
* [svn-r16566] Description:Quincey Koziol2009-03-111-14/+15
| | | | | | | | Bring r16560 back from trunk. Tested on: Mac OS X/32 10.5.6 (amazon) debug & production (Following up with tests on more platforms)
* [svn-r16536] Description:Quincey Koziol2009-03-021-2/+2
| | | | | | | | | | | Bring v16532 back from trunk to v1.8 branch: Pass the chunk "user data" to H5D_chunk_unlock(), so that chunks with an address already aren't reallocated. Tested on: FreeBSD/32 6.3 (duty) in debug mode (h5committest not needed here, performed on trunk)
* [svn-r16524] Purpose: Fix bugs related to H5Dset_extent and fill valuesNeil Fortner2009-02-271-5/+13
| | | | | | | | | Description: In some situations it was possible for the fill value to not be written to parts of a chunked dataset, particularly when extending and/or shrinking. Prior to the fix for the chunk cache (1015) these bugs would have been exceedingly rare. Tested: jam, smirom, linew (h5committest)
* [svn-r16467] Performance Improvement(bug #1450). When a chunk is bigger ↵Raymond Lu2009-02-111-2/+2
| | | | | | | | | | than the cache size and isn't allocated on disk, the library still loaded it in the cache, which is redundant. I changed it to bypass the cache and added a test in dsets.c. Tested on jam and smirom.
* [svn-r15801] Description:Quincey Koziol2008-10-071-12/+16
| | | | | | | | | | | | | | | | | | | | | | | Bring r15800 back from trunk: - File free space branch changes through r15794 Tested on: Mac OS X/32 10.5.5 (amazon) in debug mode Mac OS X/32 10.5.5 (amazon) w/C++ & FORTRAN, w/threadsafe, in production mode FreeBSD/32 6.3 (duty) in debug mode FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
* [svn-r15629] Description:Quincey Koziol2008-09-161-72/+72
| | | | | | | | | | | | | Remove trailing whitespace from C/C++ source files, with the following script: foreach f (*.[ch] *.cpp) sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f end Tested on: Mac OS X/32 10.5.5 (amazon) No need for h5committest, just whitespace changes...
* [svn-r15624] This check-in fixes a bug inside parallel HDF5 testsuite.MuQun Yang2008-09-151-2/+28
| | | | | | When enable-debug is turned on, a special macro block H5_HAVE_INSTRUMENTED_LIBRARY inside HDF5 will be executed to check if some collective chunk IO test cases are being run with the correct settings(one link, multiple chunk etc.). However,when complicated derived datatype in some mpi-io packages are not supported, the library has to switch one link IO with/without the optimization to multiple chunk IO with/without the optimization. The current testsuite doesn't know this and generates a false assertion failure message. This check-in fix this problem by providing a second property to avoid the false faiure message.
* [svn-r15132] Description:Quincey Koziol2008-06-031-30/+16
| | | | | | | | | | | | | | Bring back revision 15131 from trunk: Finish omnibus chunked dataset I/O refactoring, to separate general actions on chunked datasets from actions that are specific to using the v1 B-tree index. Cleaned up a few bugs and added some additional tests also. Tested on: Mac OS X/32 10.5.3 (amazon) Linux/32 2.4 (chicago)
* [svn-r15016] Description:Quincey Koziol2008-05-161-4/+8
| | | | | | | | | | | | | | | | | | | Port revision 15015 back to 1.8 branch: > Detect chunks that are >4GB before dataset gets created and return error > to application. > > Tweak lots of internal variables that hold the chunk size/dimensions to > use an 'uint32_t', instead of a 'size_t', so that the integer size is constant. > > Correct a number of our tests which were creating datasets with chunks > that were >4GB and add some specific tests for >4GB chunk size detection. > > Minor whitespace & other code cleanups. Tested on: Mac OS X/32 10.5.2 (amazon) Forthcoming testing on other platforms...
* [svn-r14861] Description:Quincey Koziol2008-04-241-1341/+1317
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Bring revision 14860 back to 1.8 branch, change log for rev 14860 is: Omnibus raw data I/O revisions, with wide-ranging changes and refactoring, in order to prepare for implementing "fast append" feature. These changes remove the majority of the code duplication for raw data I/O which has crept in over the last ten years and introduces a more object- oriented design for operating on different types of dataset storage. Description: Omnibus raw data I/O revisions, with wide-ranging changes and refactoring, in order to prepare for implementing "fast append" feature. These changes remove the majority of the code duplication for raw data I/O which has crept in over the last ten years and introduces a more object- oriented design for operating on different types of dataset storage. Chunked storage no longer has it's own I/O routines, it is now handled as either contiguous (if chunk is not pulled into the cache) or compact (if the chunk is cached in memory). No bug or feature changes, at least intentionally... :-) Tested on: Mac OS X/32 10.5.2 (amazon) w/production
* [svn-r14326] Description:Quincey Koziol2007-12-061-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | - Keep skip list for tracking chunks with dataset (instead of creating/ destroying it for each I/O operation) and just delete the skip list nodes. - Avoid computations for normalizing selection offset when offset not set. - Avoid updating object modification time twice during dataset creation. - Avoid updating dataset layout message (and object modification time) until dataset is closed. Tested on: FreeBSD/32 6.2 (duty) in debug mode FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Mac OS X/32 10.4.10 (amazon) in debug mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
* [svn-r14250] Description:Quincey Koziol2007-11-101-27/+29
| | | | | | | | | | | | | | | | | | | | | - Changed name of chunk mapping structure from "fm_map" to "H5D_chunk_map_t" - Pushed parallel I/O specific fields into ifdef'ed sections, when building chunk map - Took out most calls to IS_H5FD_MPI macro Tested on: FreeBSD/32 6.2 (duty) in debug mode FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe, in debug mode Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN, in production mode Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN, in production mode Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN, w/szip filter, in production mode Mac OS X/32 10.4.10 (amazon) in debug mode Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
* [svn-r14096] There're 3 changes in this checkin as below:Raymond Lu2007-08-201-18/+80
| | | | | | | | | | | | | | | | | | 1. In H5Dwrite and H5Dread, let the data buffer point to a fake address if the application passes in an empty buffer. This is mainly for MPIO programs that some processes may not have any data to write or read but still participate the I/O. This solution solves some MPI's problem like the ChaMPIon on tungsten which doesn't support empty buffer. 2. The ChaMPIon on tungsten doesn't support complex derived MPI data type correctly and collective I/O when some processes don't have any data to write or read correctly. Detect the compiler "cmpicc" in the system-specific config file and set the variables for these two cases to false. The PHDF5 library already has set up a way to switch collective chunked I/O to independent under these two cases. 3. A bug fix - During the work of the optimization for compound data I/O, the case for switching collective chunked I/O to independent I/O was leftout. Fixed it by adding I/O caching to it in H5D_multi_chunk_collective_io in H5Dmpio.c. Tested on tungsten, cobalt, and kagiso for parallel; on linew and smirom for serial.
* [svn-r13997] Moved H5D_istore_lock and H5D_istore_unlock from ↵Raymond Lu2007-07-211-10/+8
| | | | | | | | | H5D_istore_writevv and H5D_istore_readvv to H5D_chunk_write and H5D_chunk_read to avoid frequent lock and unlock and to improve some performance. Tested with h5committest on THG machines.
* [svn-r13730] Remove debug information inside H5Dmpio.c when enable-debug is ↵MuQun Yang2007-05-031-48/+71
| | | | | | | | turned on. More code will be cleaned up tomorrow. Since only debugging information is removed, I tested this only at kagiso(linux mpich). Kent
* [svn-r13253] Updated all C and C++ style source code files with the THG ↵Albert Cheng2007-02-071-2/+3
| | | | | | | | | copyright notice. Tested platform: Kagiso only since it is only a comment block change. If it works in one machine, it should work in all, I hope. Still need to check the parallel build on copper.
* [svn-r12727] 1. Cleaning up the comments of the source code.MuQun Yang2006-10-061-92/+85
| | | | | | | 2. Using H5Dmpio_Debug macro to reflect the debugging information of new parallel IO supports. They are very minor changes. Tested at heping. t_pflush2 failed. Others passed.
* [svn-r12593] Description:Quincey Koziol2006-08-171-339/+161
| | | | | | | | | Clean up some of the parallel chunk I/O compiler warnings and other minor code rearrangement. Tested on: FreeBSD 4.11 (sleipnir) w/paralllel h5committest
* [svn-r12555] 1. Erased a strange character added to this codeMuQun Yang2006-08-091-8/+35
| | | | | | | 2. Used to workaround solution to fix the cchunk10 failure caused by the MPICH bug Machine test: heping, mir
* [svn-r12553] This check-in includes the following part of parallel ↵MuQun Yang2006-08-091-200/+385
| | | | | | | | | | | | | | | optimization codes: 1. Provide another option for users to do independent IO with MPI file setview(collectively) 2. With the request of collective IO from users, using Independent IO with MPI file setview if we find collective IO is not good for the applications for IO per chunk(multi-chunk IO) case. Previously we used pure independent IO and that actually performed small IO(IO each row) for this case. The recent performance study suggested the independent IO with file setview can acheieve significantly better performance than collective IO when not many processes participate in the IO. 3. For applications that explicitly choose to do collective IO per chunk case, the library won't do any optimization(gather/broadcast) operations. The library simply passes the collective IO request to MPI-IO. Tested at copper, kagiso, heping, mir and tungsten(cmpi and mpich) Kagiso is using LAM, t_mpi test was broken even. The cchunk10 test failed at heping and mir. I suspected it was an MPICH problem. Will investigate later. Everything passed at copper. at tungsten: the old cmpi bug(failed at esetw) is still there. Other tests passed. Some sequential fheap tests failed at kagiso.
* [svn-r12440] Purpose:Quincey Koziol2006-06-271-138/+138
| | | | | | | | | | | Code cleanup Description: Trim trailing whitespace in Makefile.am and C/C++ source files to make diffing changes easier. Platforms tested: None necessary, whitespace only change
* [svn-r12400] Purpose:MuQun Yang2006-06-031-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some collective chunk IO macro names are confusing, change them to more meaningful names. Description: H5Pset_dxpl_mpio_chunk_opt will set a flag so that the library can do one linked IO or multi-chunk IO with collective in chunking storage directly. That is, the library won't do analyses to determine this. The flags for the enum type we used before are: H5FD_MPIO_OPT_ONE_IO H5FD_MPIO_OPT_MULTI_IO They are not good names because of the following two reasons: 1. It doesn't reflect chunking storage 2. OPT is kind of redundant and misleading, Solution: We change the names to H5FD_MPIO_CHUNK_ONE_IO H5FD_MPIO_CHUNK_MULTI_IO Platforms tested: Since only macro names are changed, no need to test with h5committest. heping(mpich 1.2.6) Misc. update:
* [svn-r12343] Purpose:MuQun Yang2006-05-111-16/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Bug fix for collective chunk IO Description: Several options have been provided inside HDF5 library for obtaining chunk addresses when doing collective chunk IO. One option is to obtain chunk addresses for one process and broadcast to other processes. This option needs all processes to participate. If using link-chunked IO without any optimizations, sometimes this is not true due to the random initialization for one variable at mpich 1.2.7. This is a bug inside the collective chunk IO code. Solution: 1.Initalize all the variables to some safe numbers, 2. Avoid using MPI broadcast to obtain the chunk address if possible until more performance studies have been done. 3. Seems okay to obtain chunk addresses individually for each processor. This option may cover most cases. Platforms tested: h5committest(copper is not usable) NCSA teragrid (mpich 1.2.5) mir 64-bit linux (mpich 1.2.6) Misc. update:
* [svn-r12173] Purpose:MuQun Yang2006-03-291-1/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | Adding parallel tests for optional collective chunk APIs Description: Three new APIs "H5Pset_dxpl_mpio_chunk_opt_ratio H5Pset_dxpl_mpio_chunk_opt_num H5Pset_dxpl_mpio_chunk_opt" for optional optimization choices from users have been added to the libraries. This check-in adds six tests to verify the funcationality and correctedness of these APIs. These tests need to be verified with 3 or more processors and with MPI-IO driver only. Solution: Using H5Pinsert, H5Pget, H5Pset to verify that the library indeed goes into the branch we hope for. Using H5_HAVE_INSTRUMENT macro to isolate these changes so that it won't affect or be misused by the application. Platforms tested: h5committest(shanti still refused to be connected) Parallel tests on heping somehow are skipped. Manually testing at heping. Have checked 1,2,3,4,5 processes. Misc. update:
* [svn-r12123] Purpose:MuQun Yang2006-03-201-12/+14
| | | | | | | | | | | | | | Add more comments Description: Add more comments to H5Dmpio.c, which describes a little bit more about collective IO management. Solution: Platforms tested: Only test at heping since only comments were added. Misc. update:
* [svn-r12117] Purpose:MuQun Yang2006-03-181-38/+25
| | | | | | | | | | | | | | | | | | | | | Enhancing the optimiziation of collective IO per chunk Description: When the user does one of the following two things: 1. to do collective IO per chunk without using our optimization code 2. or the user passes the percent of number of process per chunk to be 0 when choosing to do collective IO per chunk It is not necessary that the library uses MPI-IO collective calls to do any optimization. Solution: Modify the code so that no MPI communication-involved analyses will be done for the above cases. Chunk addresses are obtained globally and IO modes are assigned to collective always. Platforms tested: h5committest Misc. update:
* [svn-r12111] Purpose:MuQun Yang2006-03-171-11/+44
| | | | | | | | | | | | | minor change for collective code Description: Solution: Platforms tested: mir Misc. update:
* [svn-r12090] Purpose:MuQun Yang2006-03-141-24/+30
| | | | | | | | | | | | | | | | | | | New APIs to add for collective chunk IO Description: Three new APIs H5Pset_dxpl_mpio_chunk_opt_ratio H5Pset_dxpl_mpio_chunk_opt_num H5Pset_dxpl_mpio_chunk_opt for optional optimization choices from users. Solution: Haven't added tests yet, won't affect other parts of the library. Will add tests after urgent investigations of memory leaking problems from NASA Aura team. Platforms tested: heping: both parallel and sequential shanti Misc. update:
* [svn-r11964] Purpose:MuQun Yang2006-02-231-6/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | bug fix Description: ret_value is not set to SUCCEED in H5D_mpio_select_write That causes the 64-bit intel compiler unhappy. It will give a non-zero number and cause the testing fake failure. Another one is picked up by cmpi again. For one optimization case, another variable is not initialized properly and compiler set an unhappy number, cause the test failed. Solution: Properly initialize those variables. Platforms tested: teragrid:parallel mir:parallel heping: parallel and sequential tungsten:parallel Misc. update: h5committest doesn't finish due to no space left on device. parallel tests still failed at tungsten with cmpi. It looks like it was a bug from cmpi.
* [svn-r11960] Purpose:MuQun Yang2006-02-211-17/+15
| | | | | | | | | | | | | | code clean up Description: Clean up some warnings in collective chunk IO code Solution: Platforms tested: heping Misc. update:
* [svn-r11955] MuQun Yang2006-02-171-1/+1
| | | | | | | | | | | | | | Purpose: Erase one printf line accidently inserted in the code. Description: Solution: Platforms tested: No need to test. Misc. update:
* [svn-r11950] Purpose:MuQun Yang2006-02-161-129/+1517
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enhanced collective chunk IO supports Description: 1. When using collective IO with chunking storage without any tuning, performance may become worse under some circumstances. 2. Current HDF5 handles raw-data IO per chunk. So for many small chunks, many small IOs will be passed into MPI-IO. That may cause bad performance. 3. For one IO per chunk case, sometimes performance with collective is worse than performance with independent. An obvious case is when only one process is doing IO and all other process are not doing IO, the collective IO will only add overheads for communication. We want to avoid this case. Some management inside our library needs to be done. Solution: - Added managements of collective IO supports for chunking storage inside parallel HDF5 1) Implemented One IO with collective mode for all chunks in the application by building one MPI derived datatype accross all chunks. 2) Implemented the decision-making support to do collective IO inside MPI-IO per chunk. 3) Added the decision-making support to do one IO accross all chunks or to do multiple IOs with each IO per chunk. 4) Added the support to handle the case some processes won't do any IOs in collectively. 5) Some MPI-IO package(mpich 1.2.6 or lower, e.g.) cannot handle collective IO correctly for the case when some processes have no contributions to IOs, a special macro is added to change collective IO mode to independent IO mode inside HDF5 library. Platforms tested: Parallel: IBM AIX 5.2(copper) Linux (heping) mpich-1.2.6 SDSC Teragrid mpich-1.2.5 Linux(Tungsten) mpich-1.2.6 Altix(NCSA cobalt) Seq: Linux(heping) Misc. update:
* [svn-r11712] Purpose:Quincey Koziol2005-11-151-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | New feature Description: Check in baseline for compact group revisions, which radically revises the source code for managing groups and object headers. WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! This initiates the "unstable" phase of the 1.7.x branch, leading up to the 1.8.0 release. Please test this code, but do _NOT_ keep files created with it - the format will change again before the release and you will not be able to read your old files!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! Solution: There's too many changes to really describe them all, but some of them include: - Stop abusing the H5G_entry_t structure and split it into two separate structures for non-symbol table node use within the library: H5O_loc_t for object locations in a file and H5G_name_t to store the path to an opened object. H5G_entry_t is now only used for storing symbol table entries on disk. - Retire H5G_namei() in favor of a more general mechanism for traversing group paths and issuing callbacks on objects located. This gets us out of the business of hacking H5G_namei() for new features, generally. - Revised H5O* routines to take a H5O_loc_t instead of H5G_entry_t - Lots more... Platforms tested: h5committested and maybe another dozen configurations.... :-)
* [svn-r11593] Purpose:Quincey Koziol2005-10-211-27/+27
| | | | | | | | | | | | Code cleanup Description: Clean up & standardize a bit in preparation for coding standards discussion. Platforms tested: FreeBSD 4.11 (sleipnir) Too minor to require h5committest
* [svn-r11245] Purpose:Quincey Koziol2005-08-131-2/+2
| | | | | | | | | | | | | | | | | | | | Code cleanup Description: Trim trailing whitespace, which is making 'diff'ing the two branches difficult. Solution: Ran this script in each directory: foreach f (*.[ch] *.cpp) sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f end Platforms tested: FreeBSD 4.11 (sleipnir) Too minor to require h5committest
* [svn-r11241] Purpose:Quincey Koziol2005-08-131-22/+60
| | | | | | | | | | | | | Code cleanup Description: Fix logic error in previous checkin and also finish refactoring I/O initialization, including simplifying all the collective & parallel cases into a more unified mechanism. Platforms tested: FreeBSD 4.11 (sleipnir) w/ & w/o parallel Linux 2.4 (mir)
* [svn-r11235] Purpose:Quincey Koziol2005-08-121-17/+11
| | | | | | | | | | | | | Code cleanup Description: Refactor, simplify and cleanup Kent's recent checking for collective chunk I/O. There's a bug that I need to talk to Kent about and some more cleanups still, but this is reasonable for an interim point. Platforms tested: FreeBSD 4.11 (sleipnir) w/parallel Too minor for h5committest
* [svn-r11231] Purpose:MuQun Yang2005-08-111-359/+40
| | | | | | | | | | | | | | | | | | | | | | | bug fix for collective chunk IO, phase 1 Optimization hasn't been done yet, the collective chunk IO bug should be fixed. Description: In chunking storage, memory space and file space will be remapped, So to check whether file space and memory space are regular in order to use optimized MPI derived datatype for collective call one has to check per-chunk wise instead of per hyperslab wise. Even a regular memory space will be stored in span-tree and will be irregular before chunk IO. Solution: 1. Check file space and memory space per chunk wise instead of per hyperslab wise. 2. For collective IO mode, number of chunks covered by hyperslab may be different. Since we are handing per chunk per IO, for the extra chunk IO for some(not all) processors, collective mode will cause program hanged. So for the extra chunk Io mode independent IO has to be used. 3. On some platforms, Complex MPI derived datatype is not working, so we have to use independent IO for collective IO mode if the selection is irregular. However, when the selection is regular, we do want to use collective IO since that will improve performance. Special cares have to be added for this case. Platforms tested: copper(AIX 5.1) Linux(heping mpich 1.2.6), Teragrid machine, Cobalt(altix), modi4 Misc. update:
* [svn-r10628] Purpose:Quincey Koziol2005-04-181-3/+1
| | | | | | | | | | | Code cleanup Description: Clean up various warnings reported by the Windows team. Platforms tested: FreeBSD 4.11 (sleipnir) Too minor to require h5committest
* [svn-r10545] Purpose:MuQun Yang2005-04-051-73/+72
| | | | | | | | | | | | | | | | | | | | | | | | Activating collective IO supports for irregular selction inside HDF5 dataset. This support doesn't include to build the final complicated MPI derived datatype support for chunking storage. Description: Support collective chunk IO for both contiguous and chunking storage for irregular selection( using H5S_SELECT_OR for multiple hyperslab selection) Solution: Using MPI derived datatype to realize this feature. Problems still need to be investigated: Big size irregular hyperslab selection test may cause MPI hang or abnormalexit with MPICH family on various platforms. This is really hard to debug since sometimes it can work and sometimes it cannot work. We will continue investigating those cases. This may not be parallel HDF5 bugs since with the recent version of poe at IBM platforms, all tests passed. Platforms tested: 1. Linux heping 2.4.21-27.0.1 with mpich 2. AIX 5.1 copper with mpcc_r 3. Altix cobalt SGI linux 2.4.21-sgi304rp05031014_10149 with icc -lmpi 4. Linux Cluster SDSC TG, intel 8-r2 with mpich 1.2.5 5. NCSA Linux Cluster Tungsten, MPICH-TCP-1.2.5.2, Intel 8.0 under lustre 6. NCSA Linux Cluster Tungsten, MPICH-LAM-INTEL711, sometimes not working 7. NCSA Linux CLuster Tungsten, champion-pro-1.2.0-3, not working for other collective IO tests, but work for irregular selection collective IO test. Misc. update:
generated by cgit v0.12 at 2024-11-05 16:37:59 (GMT)