summaryrefslogtreecommitdiffstats
path: root/src/H5Dcontig.c
Commit message (Collapse)AuthorAgeFilesLines
* [svn-r6825] Purpose:Quincey Koziol2003-05-071-774/+497
| | | | | | | | | | | | | | | | | | | | | | | | | New feature/enhancement Description: Chunked datasets are handled poorly in several circumstances involving certain selections and chunks that are too large for the chunk cache and/or chunks with filters, causing the chunk to be read from disk multiple times. Solution: Rearrange raw data I/O infrastructure to handle chunked datasets in a much more friendly way by creating a selection in memory and on disk for each chunk in a chunked dataset and performing all of the I/O on that chunk at one time. There are still some scalability (the current code attempts to create a selection for all the chunks in the dataset, instead of just the chunks that are accessed, requiring portions of the istore.c and fillval.c tests to be commented out) and performance issues, but checking this in will allow the changes to be tested by a much wider audience while I address the remaining issues. Platforms tested: h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
* [svn-r6823] Purpose:Bill Wendling2003-05-071-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Code Improvements/Bug Fixes Description: Comparison of equality of a double/float variable to 0.0 is not guaranteed to work and is bad practice. In H5Fcontig.c, a warning was given by a statement like: x = (++x) % y; This could be confusing to a compiler I suppose. In H5RS.c, a typedef of a structure was being tagged by the compiler as "useless" because it had the form: typedef struct foo { int var1; /* ... */ }; /* <--- note no name for this typedef */ The statement "typedef struct foo foo" is already in the header file. Solution: Test that the absolute value of the variable is < a very small positive number. Changed "x = (++x) % y" to "++x; x %= y;" instead. Removed the "typedef" from the structure in the H5RS.c file. Platforms tested: Modi4 (Parallel & Fortran) Verbena (C++ & Fortran) Arabica (Fortran) Misc. update:
* [svn-r6653] Purpose:Quincey Koziol2003-04-141-0/+46
| | | | | | | | | | | | | | | | | New feature. Description: Added ability to free space used for storing raw data for both contiguous and chunked datasets. Platforms tested: FreeBSD 4.8 (sleipnir) w/C++ Linux 2.4 (burrwhite) w/FORTRAN Solaris 2.7 (arabica) w/FORTRAN IRIX64 6.5 (modi4) w/parallel & FORTRAN (h5committest not run due to my ongoing difficulties with C++ on burrwhite).
* [svn-r6426] Purpose:Bill Wendling2003-02-211-24/+54
| | | | | | | | | | | | Update Description: Final support addition for FPHDF5. Wherever there was an equivalent MPIO or MPIO/POSIX call/check, I placed an FPHDF5 call/check there as well, with the appropriate #ifdefs in place. The hdf5.h is updated with the H5FDfphdf5.h driver #include. Platforms tested: Linux & Modi4
* [svn-r6266] Purpose:Quincey Koziol2003-01-101-5/+5
| | | | | | | | | | | | Code cleanup/new feature. Description: Split FUNC_LEAVE into API and non-API specific versions. This allows a solution to compiling this branch with C++, as well as reducing the size of the binaries produced. Platforms tested: FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
* [svn-r6252] Purpose:Quincey Koziol2003-01-091-44/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lots of performance improvements & a couple new internal API interfaces. Description: Performance Improvements: - Cached file offset & length sizes in shared file struct, to avoid constantly looking them up in the FCPL. - Generic property improvements: - Added "revision" number to generic property classes to speed up comparisons. - Changed method of storing properties from using a hash-table to the TBBT routines in the library. - Share the propery names between classes and the lists derived from them. - Removed redundant 'def_value' buffer from each property. - Switching code to use a "copy on write" strategy for properties in each list, where the properties in each list are shared with the properties in the class, until a property's value is changed in a list. - Fixed error in layout code which was allocating too many buffers. - Redefined public macros of the form (H5open()/H5check, <variable>) internally to only be (<variable>), avoiding innumerable useless calls to H5open() and H5check_version(). - Reuse already zeroed buffers in H5F_contig_fill instead of constantly re-zeroing them. - Don't write fill values if writing entire dataset. - Use gettimeofday() system call instead of time() system when checking the modification time of a dataset. - Added reference counted string API and use it for tracking the names of objects opening in a file (for the ID->name code). - Removed redundant H5P_get() calls in B-tree routines. - Redefine H5T datatype macros internally to the library, to avoid calling H5check redundantly. - Keep dataspace information for dataset locally instead of reading from disk each time. Added new module to track open objects in a file, to allow this (which will be useful eventually for some FPH5 metadata caching issues). - Remove H5AC_find macro which was inlining metadata cache lookups, and call function instead. - Remove redundant memset() calls from H5G_namei() routine. - Remove redundant checking of object type when locating objects in metadata cache and rely on the address only. - Create default dataset object to use when default dataset creation property list is used to create datasets, bypassing querying for all the property list values. - Use default I/O vector size when performing raw data with the default dataset transfer property list, instead of querying for I/O vector size. - Remove H5P_DEFAULT internally to the library, replacing it with more specific default property list based on the type of property list needed. - Remove redundant memset() calls in object header message (H5O*) routines. - Remove redunant memset() calls in data I/O routines. - Split free-list allocation routines into malloc() and calloc()- like routines, instead of one combined routine. - Remove lots of indirection in H5O*() routines. - Simplify metadata cache entry comparison routine (used when flushing entire cache out). - Only enable metadata cache statistics when H5AC_DEBUG is turned on, instead of always tracking them. - Simplify address comparison macro (H5F_addr_eq). - Remove redundant metadata cache entry protections during dataset creation by protecting the object header once and making all the modifications necessary for the dataset creation before unprotecting it. - Reduce # of "number of element in extent" computations performed by computing and storing the value during dataspace creation. - Simplify checking for group location's file information, when file has not been involving in file-mounting operations. - Use binary encoding for modification time, instead of ASCII. - Hoist H5HL_peek calls (to get information in a local heap) out of loops in many group routine. - Use static variable for iterators of selections, instead of dynamically allocation them each time. - Lookup & insert new entries in one step, avoiding traversing group's B-tree twice. - Fixed memory leak in H5Gget_objname_idx() routine (tangential to performance improvements, but fixed along the way). - Use free-list for reference counted strings. - Don't bother copying object names into cached group entries, since they are re-created when an object is opened. The benchmark I used to measure these results created several thousand small (2K) datasets in a file and wrote out the data for them. This is Elena's "regular.c" benchmark. These changes resulted in approximately ~4.3x speedup of the development branch when compared to the previous code in the development branch and ~1.4x speedup compared to the release branch. Additionally, these changes reduce the total memory used (code and data) by the development branch by ~800KB, bringing the development branch back into the same ballpark as the release branch. I'll send out a more detailed description of the benchmark results as a followup note. New internal API routines: Added "reference counted strings" API for tracking strings that get used by multiple owners without duplicating the strings. Added "ternary search tree" API for text->object mappings. Platforms tested: Tested h5committest {arabica (fortran), eirene (fortran, C++) modi4 (parallel, fortran)} Other platforms/configurations tested? FreeBSD 4.7 (sleipnir) serial & parallel Solaris 2.6 (baldric) serial
* [svn-r5894] Purpose:Quincey Koziol2002-08-271-2/+191
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Bug fix/Code cleanup/New Feature Description: Correct problems with writing fill-values to external storage and allocate the data storage at the correct times. Also, mostly straighten out the strange code which allocates and fills raw data storage for datasets. Things are still a bit odd in that the fill-values for chunked datasets are written when the space is allocated, instead of in a separate routine, but there are two reasons for this: it's inefficient (especially in parallel) to iterate through all the chunks twice, and (more importantly) the space needed to store compressed chunks isn't known until we've got a buffer of compressed fill-values ready to write to the chunk. Additionally, add in the H5D_SPACE_ALLOC_INCR and H5D_SPACE_ALLOC_DEFAULT setting for the "space time", which incorporate the previous behavior of the space allocation for chunked datasets. The default settings for the different types of dataset storage are now as follows: Contiguous - Late Chunked - Incremental Compact - Early This checkin also incorporates a change to the behavior of external data storage in two ways - fill-values are _never_ written to external storage (under the assumption that writing fill-values is triggered by allocating space in an HDF5 file, and since space is not allocated in the file, the fill-values should not be written) and external data files are now created if they don't exist when data is written to them. The fill-value will probably need to be revisited at some time in the future, this just seemed like the safer course currently. I think I cleaned up some compiler errors also, before getting bogged down in the fixes for the space allocation and fill-values. Platforms tested: FreeBSD 4.6 (sleipnir) w/serial & parallel. Will be testing on IRIX64 6.5 (modi4) in serial & parallel shortly.
* [svn-r5842] Purpose:Quincey Koziol2002-08-081-86/+54
| | | | | | | | | | | | | | | | Code cleanup Description: Change most (all?) HRETURN_ERROR macros to HGOTO_ERROR macros, along with HRETURN macros to HGOTO_DONE macros. This unifies the error return path from functions and reduces the size of the library by up to 10% on some platforms. Additionally, I improved a lot of the error cleanup code in many routines. Platforms tested: FreeBSD 4.6 (sleipnir) serial & parallel and IRIX64 6.5 (modi4) serial & parallel.
* [svn-r5828] Purpose:Quincey Koziol2002-07-231-3/+6
| | | | | | | | | | Code cleanup Description: Make the sieve buffer managed by the free-list code. Platforms tested: FreeBSD 4.6 (sleipnir)
* [svn-r5471] Purpose:Quincey Koziol2002-05-291-4/+4
| | | | | | | | | | | | | | | | Code cleanup Description: Broke the FUNC_ENTER macro into several macros, with more specialized uses (which followup mail will describe). This was designed to move most/all of the checks which could be done at compile time to that point, instead of needlessly performing them (over & over :-) at run-time. This reduces the library's size (and thus staticly linked binaries) and has a minor speedup effect also. Platforms tested: IRIX64 6.5 (modi4) with parallel & FORTRAN enabled, and additional testing on FreeBSD and Solaris immediately after the checkin.
* [svn-r5439] Purpose:Quincey Koziol2002-05-201-4/+4
| | | | | | | | | | | Code cleanup Description: Change variable casts to use H5_ASSIGN_OVERFLOW, or check for overflow before the cast with H5_CHECK_OVERFLOW. Platforms tested: FreeBSD 4.5 (sleipnir)
* [svn-r5437] Pedro Vicente Nunes2002-05-181-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Purpose: eliminate warnings on win32 Description: some conversion warnings were being issued Solution: added some type casts Platforms tested: w2000, linux changed the HDmemset to include a cast in win32; this was just to eliminate a compiler warning. probably the macro can also be used in unix #ifdef WIN32 #define HDmemset(X,C,Z) memset((void*)(X),C,Z) #else /* WIN32 */ #define HDmemset(X,C,Z) memset(X,C,Z) #endif /* WIN32 */ the list of previous warnings was D:\disk_w\hdf5\src\H5FDstdio.c(659) : warning C4244: 'initializing' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Fcontig.c(435) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Fcontig.c(497) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Fcontig.c(915) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Fcontig.c(982) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(912) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(995) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(1936) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(2019) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(2862) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(2864) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(2948) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(3690) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(3692) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(3776) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Shyper.c(5167) : warning C4244: '+=' : conversion from '__int64 ' to 'unsigned int ', possible loss of data D:\disk_w\hdf5\src\H5Tvlen.c(371) : warning C4244: '=' : conversion from 'unsigned __int64 ' to 'unsigned int ', possible loss of data
* [svn-r4355] Purpose:Quincey Koziol2001-08-141-5/+5
| | | | | | | | | | | | | | | | | | | | | | Code cleanup (sorta) Description: When the first versions of the HDF5 library were designed, I remembered vividly the difficulties of porting code from a 32-bit platform to a 16-bit platform and asked that people use intn & uintn instead of int & unsigned int, respectively. However, in hindsight, this was overkill and unnecessary since we weren't going to be porting the HDF5 library to 16-bit architectures. Currently, the extra uintn & intn typedefs are causing problems for users who'd like to include both the HDF5 and HDF4 header files in one source module (like Kent's h4toh5 library). Solution: Changed the uintn & intn's to unsigned and int's respectively. Platforms tested: FreeBSD 4.4 (hawkwind)
* [svn-r4181] Purpose:Quincey Koziol2001-07-101-272/+870
| | | | | | | | | Bug Fix, Code Cleanup, Code Optimization, etc. Description: Fold in the hyperslab speedups, clean up compile warnings and change a few things from using 'unsigned' or 'hsize_t' to use 'size_t' instead. Platforms tested: FreeBSD 4.3 (hawkwind), Solaris 2.7 (arabica), Irix64 6.5 (modi4)
* [svn-r3885] Purpose:Quincey Koziol2001-05-021-46/+84
| | | | | | | | | | | | | | | | | | | | | | | | Document bug fix Description: IMPORTANT! IMPORTANT! IMPORTANT! A case where metadata in a file could get corrupted in certain unusual sitations was detected and fixed. In certain circumstances, metadata could get cached in the raw data cache, and if that particular piece of metadata was updated on disk while incorrectly cached, the new metadata would get overwritten with the stale metadata from the raw data cache when it was flushed out. Additionally, I've patched up the raw data cache to be smarter about how much it caches and how much I/O it triggers, leading to some speedups. Solution: Changed the raw data I/O routines which perform caching to require a parameter with the size of the dataset being accessed and limited the cache to no more than that many bytes. Platforms tested: FreeBSD 4.3 (hawkwind)
* [svn-r3832] Purpose:Quincey Koziol2001-04-231-11/+33
| | | | | | | | | | | | Bug Fix. Description: Setting a non-zero userblock size was causing raw data caching code to break. Solution: Changed from using absolute end-of-address-space offsets in cache size calculations to relative offsets. Platforms tested: FreeBSD 4.3 (hawkwind)
* [svn-r3781] Purpose:Bill Wendling2001-04-051-7/+7
| | | | | | | | | | | | | | | | | | Update Description: Changed #include <hdf_file.h> construct to #include "hdf_file.h" so that the GNU compiler can more easily pick up the dependencies which it places in the .depend and Dependencies files. Also regenerated the Dependencies to go along with this. Platforms tested: Linux
* [svn-r3252] Purpose:Quincey Koziol2001-01-091-2/+4
| | | | | | | | | | | | | Code cleanup. Description: Fixed _lots_ (I mean _tons_) of warnings spit out by the gcc with the extra warnings. Including a few show-stoppers for compression on IRIX machines. Solution: Changed lots of variables' types to more sensible and consistent types, more range-checking, more variable typecasts, etc. Platforms tested: FreeBSD 4.2 (hawkwind), IRIX64-64 (modi4)
* [svn-r3236] Purpose:Quincey Koziol2001-01-041-6/+12
| | | | | | | | Clean up code. Description: Cleaned up various compiler warnings. Platforms tested: FreeBSD 4.2 (hawkwind)
* [svn-r2945] Purpose:Quincey Koziol2000-11-161-2/+1
| | | | | | | | Small code optimization Description: Removed some unnecessary buffer assignments. Platforms tested: Solaris 2.6 (baldric)
* [svn-r2793] Purpose:Bill Wendling2000-11-021-6/+6
| | | | | | | | | | | Bug Fix Description: Some of the HRETURN types were incorrect. They were NULL when they should have been FAIL. Solution: Changed the NULLs to FAILs. Platforms tested: Linux
* [svn-r2736] Purpose:Quincey Koziol2000-10-251-14/+32
| | | | | | | | | | Code optimization Description: Minor tweaks throughout the optimized regular hyperslab code to increase speed. This set of improvements increase the benchmark time from taking ~5.46 seconds to ~4.50 seconds, or around a 20% further speedup. Platforms tested: Solaris 2.6 (baldric)
* [svn-r2722] Purpose:Quincey Koziol2000-10-241-8/+8
| | | | | | | | | | Feature symmetry Description: A while ago I needed to get the 'type' of data being accessed during writes to the VFL driver, so I put in code to get the information down there. Albert asked for the same information during reads, so I've added that in. Tested: FreeBSD 4.1.1 (hawkwind)
* [svn-r2652] Purpose:Quincey Koziol2000-10-101-5/+8
| | | | | | | | | | | | | | | Maintainance & performance enhancements Description: Re-arranged header files to protect private symbols better. Changed optimized regular hyperslab I/O to compute the offsets more efficiently from previous method of using matrix operations. Added sequential I/O operations at a more abstract level (at the same level as H5F_arr_read/write), to support the optimized hyperslab I/O. Platforms tested: Solaris 2.6 (baldric) & FreeBSD 4.1.1 (hawkwind)
* [svn-r2633] Purpose:Quincey Koziol2000-10-041-10/+10
| | | | | | | | | | | | Bug Fix Description: Use H5FD_get_eoa instead of H5FD_get_eof to check for reading off the end of the allocated file space. Using H5FD_get_eof was causing the Stream VFD to fail. Solution: Switched from using H5FD_get_eof to H5FD_get_eoa Platforms tested: FreeBSD 4.1.1 (hawkwind)
* [svn-r2611] Purpose:Quincey Koziol2000-09-281-0/+340
Rearrange code Description: The data sieve buffering code for contiguously stored datasets was wedged in the H5F_arr_read/H5F_arr_write routines. Solution: Created a new H5Fcontig.c to hold I/O routines for contiguously stored datasets (like H5Fistore.c for chunked dataset I/O routines) and moved data sieving code into those routines. Platforms tested: Solaris 2.6 (i.e. baldric)
generated by cgit v0.12 at 2024-11-17 07:32:03 (GMT)