[svn-r6252] Purpose:

    Lots of performance improvements & a couple new internal API interfaces.

Description:
    Performance Improvements:
        - Cached file offset & length sizes in shared file struct, to avoid
            constantly looking them up in the FCPL.
        - Generic property improvements:
            - Added "revision" number to generic property classes to speed
                up comparisons.
            - Changed method of storing properties from using a hash-table
                to the TBBT routines in the library.
            - Share the propery names between classes and the lists derived
                from them.
            - Removed redundant 'def_value' buffer from each property.
            - Switching code to use a "copy on write" strategy for
                properties in each list, where the properties in each list
                are shared with the properties in the class, until a
                property's value is changed in a list.
        - Fixed error in layout code which was allocating too many buffers.
        - Redefined public macros of the form (H5open()/H5check, <variable>)
            internally to only be (<variable>), avoiding innumerable useless
            calls to H5open() and H5check_version().
        - Reuse already zeroed buffers in H5F_contig_fill instead of
            constantly re-zeroing them.
        - Don't write fill values if writing entire dataset.
        - Use gettimeofday() system call instead of time() system when
            checking the modification time of a dataset.
        - Added reference counted string API and use it for tracking the
            names of objects opening in a file (for the ID->name code).
        - Removed redundant H5P_get() calls in B-tree routines.
        - Redefine H5T datatype macros internally to the library, to avoid
            calling H5check redundantly.
        - Keep dataspace information for dataset locally instead of reading
            from disk each time.  Added new module to track open objects
            in a file, to allow this (which will be useful eventually for
            some FPH5 metadata caching issues).
        - Remove H5AC_find macro which was inlining metadata cache lookups,
            and call function instead.
        - Remove redundant memset() calls from H5G_namei() routine.
        - Remove redundant checking of object type when locating objects
            in metadata cache and rely on the address only.
        - Create default dataset object to use when default dataset creation
            property list is used to create datasets, bypassing querying
            for all the property list values.
        - Use default I/O vector size when performing raw data with the
            default dataset transfer property list, instead of querying for
            I/O vector size.
        - Remove H5P_DEFAULT internally to the library, replacing it with
            more specific default property list based on the type of
            property list needed.
        - Remove redundant memset() calls in object header message (H5O*)
            routines.
        - Remove redunant memset() calls in data I/O routines.
        - Split free-list allocation routines into malloc() and calloc()-
            like routines, instead of one combined routine.
        - Remove lots of indirection in H5O*() routines.
        - Simplify metadata cache entry comparison routine (used when
            flushing entire cache out).
        - Only enable metadata cache statistics when H5AC_DEBUG is turned
            on, instead of always tracking them.
        - Simplify address comparison macro (H5F_addr_eq).
        - Remove redundant metadata cache entry protections during dataset
            creation by protecting the object header once and making all
            the modifications necessary for the dataset creation before
            unprotecting it.
        - Reduce # of "number of element in extent" computations performed
            by computing and storing the value during dataspace creation.
        - Simplify checking for group location's file information, when file
            has not been involving in file-mounting operations.
        - Use binary encoding for modification time, instead of ASCII.
        - Hoist H5HL_peek calls (to get information in a local heap)
            out of loops in many group routine.
        - Use static variable for iterators of selections, instead of
            dynamically allocation them each time.
        - Lookup & insert new entries in one step, avoiding traversing
            group's B-tree twice.
        - Fixed memory leak in H5Gget_objname_idx() routine (tangential to
            performance improvements, but fixed along the way).
        - Use free-list for reference counted strings.
        - Don't bother copying object names into cached group entries,
            since they are re-created when an object is opened.

        The benchmark I used to measure these results created several thousand
        small (2K) datasets in a file and wrote out the data for them.  This is
        Elena's "regular.c" benchmark.

        These changes resulted in approximately ~4.3x speedup of the
        development branch when compared to the previous code in the
        development branch and ~1.4x speedup compared to the release
        branch.

        Additionally, these changes reduce the total memory used (code and
        data) by the development branch by ~800KB, bringing the development
        branch back into the same ballpark as the release branch.

        I'll send out a more detailed description of the benchmark results
        as a followup note.

    New internal API routines:
        Added "reference counted strings" API for tracking strings that get
            used by multiple owners without duplicating the strings.
        Added "ternary search tree" API for text->object mappings.

Platforms tested:
    Tested h5committest {arabica (fortran), eirene (fortran, C++)
	modi4 (parallel, fortran)}
    Other platforms/configurations tested?
        FreeBSD 4.7 (sleipnir) serial & parallel
        Solaris 2.6 (baldric) serial

1 files changed, 493 insertions, 0 deletions

diff --git a/src/H5Opline.c b/src/H5Opline.c
new file mode 100644
index 0000000..eb42f5b
--- /dev/null
+++ b/src/H5Opline.c
@@ -0,0 +1,493 @@
+/*
+ * Copyright (C) 1998-2001 NCSA
+ *		           All rights reserved.
+ *
+ * Programmer:	Robb Matzke <matzke@llnl.gov>
+ *		Wednesday, April 15, 1998
+ *
+ * Purpose:	Data filter pipeline message.
+ */
+#include "H5private.h"
+#include "H5Eprivate.h"
+#include "H5FLprivate.h"	/*Free Lists	  */
+#include "H5MMprivate.h"
+#include "H5Oprivate.h"
+
+/* Interface initialization */
+#define PABLO_MASK	H5O_pline_mask
+static int		interface_initialize_g = 0;
+#define INTERFACE_INIT	NULL
+
+#define H5O_PLINE_VERSION	1
+
+static herr_t H5O_pline_encode (H5F_t *f, uint8_t *p, const void *mesg);
+static void *H5O_pline_decode (H5F_t *f, const uint8_t *p, H5O_shared_t *sh);
+static void *H5O_pline_copy (const void *_mesg, void *_dest);
+static size_t H5O_pline_size (H5F_t *f, const void *_mesg);
+static herr_t H5O_pline_reset (void *_mesg);
+static herr_t H5O_pline_free (void *_mesg);
+static herr_t H5O_pline_debug (H5F_t *f, const void *_mesg,
+			       FILE * stream, int indent, int fwidth);
+
+/* This message derives from H5O */
+const H5O_class_t H5O_PLINE[1] = {{
+    H5O_PLINE_ID,		/* message id number		*/
+    "filter pipeline",		/* message name for debugging	*/
+    sizeof(H5O_pline_t),	/* native message size		*/
+    H5O_pline_decode,		/* decode message		*/
+    H5O_pline_encode,		/* encode message		*/
+    H5O_pline_copy,		/* copy the native value	*/
+    H5O_pline_size,		/* size of raw message		*/
+    H5O_pline_reset,		/* reset method			*/
+    H5O_pline_free,		/* free method			*/
+    NULL,			/* get share method		*/
+    NULL, 			/* set share method		*/
+    H5O_pline_debug,		/* debug the message		*/
+}};
+
+
+/* Declare a free list to manage the H5O_pline_t struct */
+H5FL_DEFINE(H5O_pline_t);
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_decode
+ *
+ * Purpose:	Decodes a filter pipeline message.
+ *
+ * Return:	Success:	Ptr to the native message.
+ *
+ *		Failure:	NULL
+ *
+ * Programmer:	Robb Matzke
+ *              Wednesday, April 15, 1998
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static void *
+H5O_pline_decode(H5F_t UNUSED *f, const uint8_t *p,
+		H5O_shared_t UNUSED *sh)
+{
+    H5O_pline_t		*pline = NULL;
+    void		*ret_value;
+    unsigned		version;
+    size_t		i, j, n, name_length;
+
+    FUNC_ENTER_NOAPI(H5O_pline_decode, NULL);
+
+    /* check args */
+    assert(p);
+
+    /* Decode */
+    if (NULL==(pline = H5FL_CALLOC(H5O_pline_t)))
+	HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed");
+    version = *p++;
+    if (version!=H5O_PLINE_VERSION)
+	HGOTO_ERROR(H5E_PLINE, H5E_CANTLOAD, NULL, "bad version number for filter pipeline message");
+    pline->nfilters = *p++;
+    if (pline->nfilters>32)
+	HGOTO_ERROR(H5E_PLINE, H5E_CANTLOAD, NULL, "filter pipeline message has too many filters");
+    p += 6;	/*reserved*/
+    pline->nalloc = pline->nfilters;
+    pline->filter = H5MM_calloc(pline->nalloc*sizeof(pline->filter[0]));
+    if (NULL==pline->filter)
+	HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed");
+    for (i=0; i<pline->nfilters; i++) {
+	UINT16DECODE(p, pline->filter[i].id);
+	UINT16DECODE(p, name_length);
+	if (name_length % 8)
+	    HGOTO_ERROR(H5E_PLINE, H5E_CANTLOAD, NULL, "filter name length is not a multiple of eight");
+	UINT16DECODE(p, pline->filter[i].flags);
+	UINT16DECODE(p, pline->filter[i].cd_nelmts);
+	if (name_length) {
+	    /*
+	     * Get the name, allocating an extra byte for an extra null
+	     * terminator just in case there isn't one in the file (there
+	     * should be, but to be safe...)
+	     */
+	    pline->filter[i].name = H5MM_malloc(name_length+1);
+	    HDmemcpy(pline->filter[i].name, p, name_length);
+	    pline->filter[i].name[name_length] = '\0';
+	    p += name_length;
+	}
+	if ((n=pline->filter[i].cd_nelmts)) {
+	    /*
+	     * Read the client data values and the padding
+	     */
+	    pline->filter[i].cd_values = H5MM_malloc(n*sizeof(unsigned));
+	    if (NULL==pline->filter[i].cd_values)
+		HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed for client data");
+	    for (j=0; j<pline->filter[i].cd_nelmts; j++)
+		UINT32DECODE(p, pline->filter[i].cd_values[j]);
+	    if (pline->filter[i].cd_nelmts % 2)
+		p += 4; /*padding*/
+	}
+    }
+
+    /* Set return value */
+    ret_value = pline;
+
+done:
+    if (NULL==ret_value && pline) {
+        if (pline->filter) {
+            for (i=0; i<pline->nfilters; i++) {
+                H5MM_xfree(pline->filter[i].name);
+                H5MM_xfree(pline->filter[i].cd_values);
+            }
+            H5MM_xfree(pline->filter);
+        }
+        H5FL_FREE(H5O_pline_t,pline);
+    }
+
+    FUNC_LEAVE(ret_value);
+}
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_encode
+ *
+ * Purpose:	Encodes message MESG into buffer P.
+ *
+ * Return:	Non-negative on success/Negative on failure
+ *
+ * Programmer:	Robb Matzke
+ *              Wednesday, April 15, 1998
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5O_pline_encode (H5F_t UNUSED *f, uint8_t *p/*out*/, const void *mesg)
+{
+    const H5O_pline_t	*pline = (const H5O_pline_t*)mesg;
+    size_t		i, j, name_length;
+    const char		*name=NULL;
+    H5Z_class_t		*cls=NULL;
+    herr_t ret_value=SUCCEED;   /* Return value */
+    
+    FUNC_ENTER_NOAPI(H5O_pline_encode, FAIL);
+
+    /* Check args */
+    assert (p);
+    assert (mesg);
+
+    *p++ = H5O_PLINE_VERSION;
+    *p++ = (uint8_t)(pline->nfilters);
+    *p++ = 0;	/*reserved 1*/
+    *p++ = 0;	/*reserved 2*/
+    *p++ = 0;	/*reserved 3*/
+    *p++ = 0;	/*reserved 4*/
+    *p++ = 0;	/*reserved 5*/
+    *p++ = 0;	/*reserved 6*/
+
+    for (i=0; i<pline->nfilters; i++) {
+	/*
+	 * Get the filter name.  If the pipeline message has a name in it then
+	 * use that one.  Otherwise try to look up the filter and get the name
+	 * as it was registered.
+	 */
+	if (NULL==(name=pline->filter[i].name) &&
+                (cls=H5Z_find(pline->filter[i].id)))
+	    name = cls->name;
+	name_length = name ? HDstrlen(name)+1 : 0;
+
+	/* Encode the filter */
+	UINT16ENCODE(p, pline->filter[i].id);
+	UINT16ENCODE(p, H5O_ALIGN(name_length));
+	UINT16ENCODE(p, pline->filter[i].flags);
+	UINT16ENCODE(p, pline->filter[i].cd_nelmts);
+	if (name_length>0) {
+	    HDmemcpy(p, name, name_length);
+	    p += name_length;
+	    while (name_length++ % 8)
+                *p++ = 0;
+	}
+	for (j=0; j<pline->filter[i].cd_nelmts; j++)
+	    UINT32ENCODE(p, pline->filter[i].cd_values[j]);
+	if (pline->filter[i].cd_nelmts % 2)
+	    UINT32ENCODE(p, 0);
+    }
+
+done:
+    FUNC_LEAVE (ret_value);
+}
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_copy
+ *
+ * Purpose:	Copies a filter pipeline message from SRC to DST allocating
+ *		DST if necessary.  If DST is already allocated then we assume
+ *		that it isn't initialized.
+ *
+ * Return:	Success:	Ptr to DST or allocated result.
+ *
+ *		Failure:	NULL
+ *
+ * Programmer:	Robb Matzke
+ *              Wednesday, April 15, 1998
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static void *
+H5O_pline_copy (const void *_src, void *_dst/*out*/)
+{
+    const H5O_pline_t	*src = (const H5O_pline_t *)_src;
+    H5O_pline_t		*dst = (H5O_pline_t *)_dst;
+    size_t		i;
+    H5O_pline_t		*ret_value = NULL;
+    
+    FUNC_ENTER_NOAPI(H5O_pline_copy, NULL);
+
+    if (!dst && NULL==(dst = H5FL_MALLOC (H5O_pline_t)))
+	HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed");
+
+    *dst = *src;
+    dst->nalloc = dst->nfilters;
+    if (dst->nalloc>0) {
+	dst->filter = H5MM_calloc(dst->nalloc * sizeof(dst->filter[0]));
+	if (NULL==dst->filter)
+	    HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed");
+    } else {
+	dst->filter = NULL;
+    }
+    
+    for (i=0; i<src->nfilters; i++) {
+	dst->filter[i] = src->filter[i];
+	if (src->filter[i].name) {
+	    dst->filter[i].name = H5MM_xstrdup(src->filter[i].name);
+	}
+	if (src->filter[i].cd_nelmts>0) {
+	    dst->filter[i].cd_values = H5MM_malloc(src->filter[i].cd_nelmts*
+						   sizeof(unsigned));
+	    if (NULL==dst->filter[i].cd_values)
+		HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed");
+	    HDmemcpy (dst->filter[i].cd_values, src->filter[i].cd_values,
+		      src->filter[i].cd_nelmts * sizeof(unsigned));
+	}
+    }
+
+    /* Set return value */
+    ret_value = dst;
+
+done:
+    if (!ret_value && dst) {
+	if (dst->filter) {
+	    for (i=0; i<dst->nfilters; i++) {
+		H5MM_xfree(dst->filter[i].name);
+		H5MM_xfree(dst->filter[i].cd_values);
+	    }
+	    H5MM_xfree(dst->filter);
+	}
+	if (!_dst)
+            H5FL_FREE(H5O_pline_t,dst);
+    }
+
+    FUNC_LEAVE (ret_value);
+}
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_size
+ *
+ * Purpose:	Determines the size of a raw filter pipeline message.
+ *
+ * Return:	Success:	Size of message.
+ *
+ *		Failure:	zero
+ *
+ * Programmer:	Robb Matzke
+ *              Wednesday, April 15, 1998
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static size_t
+H5O_pline_size (H5F_t UNUSED *f, const void *mesg)
+{
+    const H5O_pline_t	*pline = (const H5O_pline_t*)mesg;
+    size_t		i, size, name_len;
+    const char		*name = NULL;
+    H5Z_class_t		*cls = NULL;
+    size_t ret_value;           /* Return value */
+        
+    FUNC_ENTER_NOAPI(H5O_pline_size, 0);
+
+    /* Message header */
+    size = 1 +				/*version			*/
+	   1 +				/*number of filters		*/
+	   6;				/*reserved			*/
+
+    for (i=0; i<pline->nfilters; i++) {
+	/* Get the name of the filter, same as done with H5O_pline_encode() */
+	if (NULL==(name=pline->filter[i].name) &&
+                (cls=H5Z_find(pline->filter[i].id)))
+	    name = cls->name;
+	name_len = name ? HDstrlen(name)+1 : 0;
+	
+
+	size += 2 +			/*filter identification number	*/
+		2 +			/*name length			*/
+		2 +			/*flags				*/
+		2 +			/*number of client data values	*/
+		H5O_ALIGN(name_len);	/*length of the filter name	*/
+	
+	size += pline->filter[i].cd_nelmts * 4;
+	if (pline->filter[i].cd_nelmts % 2)
+            size += 4;
+    }
+
+    /* Set return value */
+    ret_value=size;
+
+done:
+    FUNC_LEAVE (ret_value);
+}
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_reset
+ *
+ * Purpose:	Resets a filter pipeline message by clearing all filters.
+ *		The MESG buffer is not freed.
+ *
+ * Return:	Non-negative on success/Negative on failure
+ *
+ * Programmer:	Robb Matzke
+ *              Wednesday, April 15, 1998
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5O_pline_reset (void *mesg)
+{
+    H5O_pline_t	*pline = (H5O_pline_t*)mesg;
+    size_t	i;
+    herr_t ret_value=SUCCEED;   /* Return value */
+    
+    FUNC_ENTER_NOAPI(H5O_pline_reset, FAIL);
+
+    assert (pline);
+
+    for (i=0; i<pline->nfilters; i++) {
+        H5MM_xfree(pline->filter[i].name);
+        H5MM_xfree(pline->filter[i].cd_values);
+    }
+    if(pline->filter)
+        H5MM_xfree(pline->filter);
+    HDmemset(pline, 0, sizeof *pline);
+
+done:
+    FUNC_LEAVE (ret_value);
+}
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_free
+ *
+ * Purpose:	Free's the message
+ *
+ * Return:	Non-negative on success/Negative on failure
+ *
+ * Programmer:	Quincey Koziol
+ *              Saturday, March 11, 2000
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5O_pline_free (void *mesg)
+{
+    herr_t ret_value=SUCCEED;   /* Return value */
+
+    FUNC_ENTER_NOAPI(H5O_pline_free, FAIL);
+
+    assert (mesg);
+
+    H5FL_FREE(H5O_pline_t,mesg);
+
+done:
+    FUNC_LEAVE (ret_value);
+}
+
+
+/*-------------------------------------------------------------------------
+ * Function:	H5O_pline_debug
+ *
+ * Purpose:	Prints debugging information for filter pipeline message MESG
+ *		on output stream STREAM.  Each line is indented INDENT
+ *		characters and the field name takes up FWIDTH characters.
+ *
+ * Return:	Non-negative on success/Negative on failure
+ *
+ * Programmer:	Robb Matzke
+ *              Wednesday, April 15, 1998
+ *
+ * Modifications:
+ *
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5O_pline_debug (H5F_t UNUSED *f, const void *mesg, FILE *stream,
+		int indent, int fwidth)
+{
+    const H5O_pline_t	*pline = (const H5O_pline_t *)mesg;
+    size_t		i, j;
+    herr_t ret_value=SUCCEED;   /* Return value */
+
+    FUNC_ENTER_NOAPI(H5O_pline_debug, FAIL);
+
+    /* check args */
+    assert(f);
+    assert(pline);
+    assert(stream);
+    assert(indent >= 0);
+    assert(fwidth >= 0);
+
+    fprintf(stream, "%*s%-*s %lu/%lu\n", indent, "", fwidth,
+	    "Number of filters:",
+	    (unsigned long)(pline->nfilters),
+	    (unsigned long)(pline->nalloc));
+
+    for (i=0; i<pline->nfilters; i++) {
+	char		name[32];
+	sprintf(name, "Filter at position %lu", (unsigned long)i);
+	fprintf(stream, "%*s%-*s\n", indent, "", fwidth, name);
+	fprintf(stream, "%*s%-*s 0x%04x\n", indent+3, "", MAX(0, fwidth-3),
+		"Filter identification:",
+		(unsigned)(pline->filter[i].id));
+	if (pline->filter[i].name) {
+	    fprintf(stream, "%*s%-*s \"%s\"\n", indent+3, "", MAX(0, fwidth-3),
+		    "Filter name:",
+		    pline->filter[i].name);
+	} else {
+	    fprintf(stream, "%*s%-*s NONE\n", indent+3, "", MAX(0, fwidth-3),
+		    "Filter name:");
+	}
+	fprintf(stream, "%*s%-*s 0x%04x\n", indent+3, "", MAX(0, fwidth-3),
+		"Flags:",
+		(unsigned)(pline->filter[i].flags));
+	fprintf(stream, "%*s%-*s %lu\n", indent+3, "", MAX(0, fwidth-3),
+		"Num CD values:",
+		(unsigned long)(pline->filter[i].cd_nelmts));
+	for (j=0; j<pline->filter[i].cd_nelmts; j++) {
+	    char	field_name[32];
+	    sprintf(field_name, "CD value %lu", (unsigned long)j);
+	    fprintf(stream, "%*s%-*s %lu\n", indent+6, "", MAX(0, fwidth-6),
+		    field_name,
+		    (unsigned long)(pline->filter[i].cd_values[j]));
+	}
+    }
+    
+done:
+    FUNC_LEAVE(ret_value);
+}