summaryrefslogtreecommitdiffstats
path: root/src/H5Fcontig.c
diff options
context:
space:
mode:
authorQuincey Koziol <koziol@hdfgroup.org>2003-01-09 17:20:03 (GMT)
committerQuincey Koziol <koziol@hdfgroup.org>2003-01-09 17:20:03 (GMT)
commit9a433b99a56dc575f1c0b11f95b744de61859dbb (patch)
treed8c766537cb9adc364c902bd45477d97f67a4a9f /src/H5Fcontig.c
parent7fd449cb7987772a2881a5ced2ae7ad5231f1fa3 (diff)
downloadhdf5-9a433b99a56dc575f1c0b11f95b744de61859dbb.zip
hdf5-9a433b99a56dc575f1c0b11f95b744de61859dbb.tar.gz
hdf5-9a433b99a56dc575f1c0b11f95b744de61859dbb.tar.bz2
[svn-r6252] Purpose:
Lots of performance improvements & a couple new internal API interfaces. Description: Performance Improvements: - Cached file offset & length sizes in shared file struct, to avoid constantly looking them up in the FCPL. - Generic property improvements: - Added "revision" number to generic property classes to speed up comparisons. - Changed method of storing properties from using a hash-table to the TBBT routines in the library. - Share the propery names between classes and the lists derived from them. - Removed redundant 'def_value' buffer from each property. - Switching code to use a "copy on write" strategy for properties in each list, where the properties in each list are shared with the properties in the class, until a property's value is changed in a list. - Fixed error in layout code which was allocating too many buffers. - Redefined public macros of the form (H5open()/H5check, <variable>) internally to only be (<variable>), avoiding innumerable useless calls to H5open() and H5check_version(). - Reuse already zeroed buffers in H5F_contig_fill instead of constantly re-zeroing them. - Don't write fill values if writing entire dataset. - Use gettimeofday() system call instead of time() system when checking the modification time of a dataset. - Added reference counted string API and use it for tracking the names of objects opening in a file (for the ID->name code). - Removed redundant H5P_get() calls in B-tree routines. - Redefine H5T datatype macros internally to the library, to avoid calling H5check redundantly. - Keep dataspace information for dataset locally instead of reading from disk each time. Added new module to track open objects in a file, to allow this (which will be useful eventually for some FPH5 metadata caching issues). - Remove H5AC_find macro which was inlining metadata cache lookups, and call function instead. - Remove redundant memset() calls from H5G_namei() routine. - Remove redundant checking of object type when locating objects in metadata cache and rely on the address only. - Create default dataset object to use when default dataset creation property list is used to create datasets, bypassing querying for all the property list values. - Use default I/O vector size when performing raw data with the default dataset transfer property list, instead of querying for I/O vector size. - Remove H5P_DEFAULT internally to the library, replacing it with more specific default property list based on the type of property list needed. - Remove redundant memset() calls in object header message (H5O*) routines. - Remove redunant memset() calls in data I/O routines. - Split free-list allocation routines into malloc() and calloc()- like routines, instead of one combined routine. - Remove lots of indirection in H5O*() routines. - Simplify metadata cache entry comparison routine (used when flushing entire cache out). - Only enable metadata cache statistics when H5AC_DEBUG is turned on, instead of always tracking them. - Simplify address comparison macro (H5F_addr_eq). - Remove redundant metadata cache entry protections during dataset creation by protecting the object header once and making all the modifications necessary for the dataset creation before unprotecting it. - Reduce # of "number of element in extent" computations performed by computing and storing the value during dataspace creation. - Simplify checking for group location's file information, when file has not been involving in file-mounting operations. - Use binary encoding for modification time, instead of ASCII. - Hoist H5HL_peek calls (to get information in a local heap) out of loops in many group routine. - Use static variable for iterators of selections, instead of dynamically allocation them each time. - Lookup & insert new entries in one step, avoiding traversing group's B-tree twice. - Fixed memory leak in H5Gget_objname_idx() routine (tangential to performance improvements, but fixed along the way). - Use free-list for reference counted strings. - Don't bother copying object names into cached group entries, since they are re-created when an object is opened. The benchmark I used to measure these results created several thousand small (2K) datasets in a file and wrote out the data for them. This is Elena's "regular.c" benchmark. These changes resulted in approximately ~4.3x speedup of the development branch when compared to the previous code in the development branch and ~1.4x speedup compared to the release branch. Additionally, these changes reduce the total memory used (code and data) by the development branch by ~800KB, bringing the development branch back into the same ballpark as the release branch. I'll send out a more detailed description of the benchmark results as a followup note. New internal API routines: Added "reference counted strings" API for tracking strings that get used by multiple owners without duplicating the strings. Added "ternary search tree" API for text->object mappings. Platforms tested: Tested h5committest {arabica (fortran), eirene (fortran, C++) modi4 (parallel, fortran)} Other platforms/configurations tested? FreeBSD 4.7 (sleipnir) serial & parallel Solaris 2.6 (baldric) serial
Diffstat (limited to 'src/H5Fcontig.c')
-rw-r--r--src/H5Fcontig.c106
1 files changed, 62 insertions, 44 deletions
diff --git a/src/H5Fcontig.c b/src/H5Fcontig.c
index fce1ba9..a63a9ce 100644
--- a/src/H5Fcontig.c
+++ b/src/H5Fcontig.c
@@ -36,8 +36,11 @@ static int interface_initialize_g = 0;
/* Declare a PQ free list to manage the sieve buffer information */
H5FL_BLK_DEFINE(sieve_buf);
-/* Extern the free list to manage blocks of type conversion data */
-H5FL_BLK_EXTERN(type_conv);
+/* Declare the free list to manage blocks of non-zero fill-value data */
+H5FL_BLK_DEFINE_STATIC(non_zero_fill);
+
+/* Declare the free list to manage blocks of zero fill-value data */
+H5FL_BLK_DEFINE_STATIC(zero_fill);
/*-------------------------------------------------------------------------
@@ -56,11 +59,10 @@ H5FL_BLK_EXTERN(type_conv);
*/
herr_t
H5F_contig_fill(H5F_t *f, hid_t dxpl_id, struct H5O_layout_t *layout,
- struct H5P_genplist_t *dc_plist, const struct H5S_t *space,
- size_t elmt_size)
+ struct H5P_genplist_t *dc_plist, const struct H5O_efl_t *efl,
+ const struct H5S_t *space,
+ const struct H5O_fill_t *fill, size_t elmt_size)
{
- H5O_fill_t fill; /* Fill value information */
- H5O_efl_t efl; /* External File List info */
hssize_t snpoints; /* Number of points in space (for error checking) */
size_t npoints; /* Number of points in space */
size_t ptsperbuf; /* Maximum # of points which fit in the buffer */
@@ -77,6 +79,7 @@ H5F_contig_fill(H5F_t *f, hid_t dxpl_id, struct H5O_layout_t *layout,
unsigned blocks_written=0; /* Flag to indicate that chunk was actually written */
unsigned using_mpi=0; /* Flag to indicate that the file is being accessed with an MPI-capable file driver */
#endif /* H5_HAVE_PARALLEL */
+ int non_zero_fill_f=(-1); /* Indicate that a non-zero fill-value was used */
herr_t ret_value=SUCCEED; /* Return value */
FUNC_ENTER_NOAPI(H5F_contig_fill, FAIL);
@@ -91,12 +94,6 @@ H5F_contig_fill(H5F_t *f, hid_t dxpl_id, struct H5O_layout_t *layout,
assert(space);
assert(elmt_size>0);
- /* Get necessary properties from dataset creation property list */
- if(H5P_get(dc_plist, H5D_CRT_FILL_VALUE_NAME, &fill) < 0)
- HGOTO_ERROR(H5E_STORAGE, H5E_CANTGET, FAIL, "can't get fill value");
- if(H5P_get(dc_plist, H5D_CRT_EXT_FILE_LIST_NAME, &efl) < 0)
- HGOTO_ERROR(H5E_DATASET, H5E_CANTGET, FAIL, "can't retrieve external file list");
-
#ifdef H5_HAVE_PARALLEL
/* Retrieve up MPI parameters */
if(IS_H5FD_MPIO(f)) {
@@ -136,46 +133,62 @@ H5F_contig_fill(H5F_t *f, hid_t dxpl_id, struct H5O_layout_t *layout,
assert(snpoints>=0);
H5_ASSIGN_OVERFLOW(npoints,snpoints,hssize_t,size_t);
- /* Don't write default fill-values to external files */
- if(efl.nused>0 && !fill.buf)
- HGOTO_DONE(SUCCEED);
-
- /* If fill value is library default, use the element size */
- if(!fill.buf)
- fill.size=elmt_size;
+ /* If fill value is not library default, use it to set the element size */
+ if(fill->buf)
+ elmt_size=fill->size;
/*
* Fill the entire current extent with the fill value. We can do
* this quite efficiently by making sure we copy the fill value
* in relatively large pieces.
*/
- ptsperbuf = MAX(1, bufsize/fill.size);
- bufsize = ptsperbuf*fill.size;
-
- /* Allocate temporary buffer */
- if ((buf=H5FL_BLK_ALLOC(type_conv,bufsize,0))==NULL)
- HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed for fill buffer");
-
- /* Fill the buffer with the user's fill value */
- if(fill.buf)
- H5V_array_fill(buf, fill.buf, fill.size, ptsperbuf);
- else /* Fill the buffer with the default fill value */
- HDmemset(buf,0,bufsize);
+ ptsperbuf = MAX(1, bufsize/elmt_size);
+ bufsize = ptsperbuf*elmt_size;
+
+ /* Fill the buffer with the user's fill value */
+ if(fill->buf) {
+ /* Allocate temporary buffer */
+ if ((buf=H5FL_BLK_MALLOC(non_zero_fill,bufsize))==NULL)
+ HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed for fill buffer");
+
+ H5V_array_fill(buf, fill->buf, elmt_size, ptsperbuf);
+
+ /* Indicate that a non-zero fill buffer was used */
+ non_zero_fill_f=1;
+ } /* end if */
+ else { /* Fill the buffer with the default fill value */
+ htri_t buf_avail;
+
+ /* Check if there is an already zeroed out buffer available */
+ buf_avail=H5FL_BLK_AVAIL(zero_fill,bufsize);
+ assert(buf_avail!=FAIL);
+
+ /* Allocate temporary buffer (zeroing it if no buffer is available) */
+ if(!buf_avail)
+ buf=H5FL_BLK_CALLOC(zero_fill,bufsize);
+ else
+ buf=H5FL_BLK_MALLOC(zero_fill,bufsize);
+ if(buf==NULL)
+ HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed for fill buffer");
+
+ /* Indicate that a zero fill buffer was used */
+ non_zero_fill_f=0;
+ } /* end else */
- /* Start at the beginning of the dataset */
- addr = 0;
+ /* Start at the beginning of the dataset */
+ addr = 0;
- /* Loop through writing the fill value to the dataset */
- while (npoints>0) {
- size = MIN(ptsperbuf, npoints) * fill.size;
+ /* Loop through writing the fill value to the dataset */
+ while (npoints>0) {
+ size = MIN(ptsperbuf, npoints) * elmt_size;
#ifdef H5_HAVE_PARALLEL
/* Check if this file is accessed with an MPI-capable file driver */
if(using_mpi) {
/* Round-robin write the chunks out from only one process */
if(mpi_round==mpi_rank) {
- if (H5F_seq_write(f, dxpl_id, layout, dc_plist, space,
- fill.size, size, addr, buf)<0)
+ if (H5F_seq_write(f, dxpl_id, layout, dc_plist, efl, space,
+ elmt_size, size, addr, buf)<0)
HGOTO_ERROR(H5E_DATASET, H5E_CANTINIT, FAIL, "unable to write fill value to dataset");
} /* end if */
mpi_round=(++mpi_round)%mpi_size;
@@ -185,8 +198,8 @@ H5F_contig_fill(H5F_t *f, hid_t dxpl_id, struct H5O_layout_t *layout,
} /* end if */
else {
#endif /* H5_HAVE_PARALLEL */
- if (H5F_seq_write(f, dxpl_id, layout, dc_plist, space,
- fill.size, size, addr, buf)<0)
+ if (H5F_seq_write(f, dxpl_id, layout, dc_plist, efl, space,
+ elmt_size, size, addr, buf)<0)
HGOTO_ERROR(H5E_DATASET, H5E_CANTINIT, FAIL, "unable to write fill value to dataset");
#ifdef H5_HAVE_PARALLEL
} /* end else */
@@ -211,8 +224,13 @@ H5F_contig_fill(H5F_t *f, hid_t dxpl_id, struct H5O_layout_t *layout,
done:
/* Free the buffer for fill values */
- if (buf)
- H5FL_BLK_FREE(type_conv,buf);
+ if (buf) {
+ assert(non_zero_fill_f>=0);
+ if(non_zero_fill_f)
+ H5FL_BLK_FREE(non_zero_fill,buf);
+ else
+ H5FL_BLK_FREE(zero_fill,buf);
+ } /* end if */
FUNC_LEAVE(ret_value);
}
@@ -661,7 +679,7 @@ H5F_contig_readv(H5F_t *f, hsize_t _max_data, H5FD_mem_t type, haddr_t _addr,
} /* end if */
else {
/* Allocate room for the data sieve buffer */
- if (NULL==(f->shared->sieve_buf=H5FL_BLK_ALLOC(sieve_buf,f->shared->sieve_buf_size,0)))
+ if (NULL==(f->shared->sieve_buf=H5FL_BLK_MALLOC(sieve_buf,f->shared->sieve_buf_size)))
HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed");
/* Determine the new sieve buffer size & location */
@@ -1128,7 +1146,7 @@ H5F_contig_writev(H5F_t *f, hsize_t _max_data, H5FD_mem_t type, haddr_t _addr,
} /* end if */
else {
/* Allocate room for the data sieve buffer */
- if (NULL==(f->shared->sieve_buf=H5FL_BLK_ALLOC(sieve_buf,f->shared->sieve_buf_size,0)))
+ if (NULL==(f->shared->sieve_buf=H5FL_BLK_MALLOC(sieve_buf,f->shared->sieve_buf_size)))
HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed");
/* Determine the new sieve buffer size & location */