Simplify function enter macros for performance benefits (#1024)

* Take a stab at using constructors to initialize instead of
function-entry macros.  This is a work in progress.  It's good enough to
run `many_dsets`.

* Committing clang-format changes

* Add the `many_dsets` benchmark and some scripts I used on jelly for
setting up the build/test environment and for recording/flame-graphing
profiles.

* Committing clang-format changes

* Change my Makefile and environment script to work both on jelly
and on mayll (and probably on Summit).

* Disable clang-format "fix."

* Replace the `if (!H5_TERM_GLOBAL)` test in each FUNC_ENTER_ macro with
`if (true)`.

* Fix bad grammar in a comment.

* Instead of labeling the H5*__init_package routines constructors, fold
each into an initialization routine, H5*_init(), and call each of
the H5*_init() routines.  Call most of the H5*_init() routines from
H5_init_library() in an explicit order that I found out earlier by
instrumenting each __init_package routine and running the library
tests.  Roll H5FD*__init_package routines into H5FD*_init() routines.
This change ends just-in-time initialization of package dependencies by
package initializers.

Don't track in per-package variables (H5_PKG_INIT_VAR) whether each
package has been initialized.  Instead, track in a single library
variable whether the whole library is initialized or not.

Drive the initialization of packages by H5_init_library() with a table
of initializer routines.  Also drive the termination of packages by
H5_term_library() with a table.

Perform initialization as needed from FUNC_ENTER_API_INIT(err).  This
basically restores the old behavior of that macro.

Delete a bunch of #definitions in H5private.h that have fallen out of
use with these changes.

* Committing clang-format changes

* Undo the bad auto-formatting that appears to have occurred in spite
of my disabling it.  Bracket some code in /* clang-format off */ /*
clang-format on */ to prevent a recurrence.

* Remove a diagnostic abort().

* Fix a logic error: print a comma between every package terminator run,
and don't print an initial comma.

* Complete the changes I started in H5_term_library() that undo the bad
auto-formatting.

Stop tracking whether package "tops" were initialized in per-package
variables H5*_top_package_initialize_s.  H5_term_library() takes care of
that for them.

Remove H5R_top_term_package() and H5R_term_package(), they don't do
anything.

* Committing clang-format changes

* NFCI.  Simplify macro text: replace `if (true) {` with `{`.

* Fix formatting and suppress clang-format on a longer range.

* Quiet some unused label, unused variable complaints that cropped up
after I simplified the FUNC_ENTER_ macros for the sake of performance.

* Committing clang-format changes

* Delete some programs and scripts that don't belong in the pull request.

* Use the right function-entry macro.

* Use a sensible format and disable auto-formatting.

* Stop calling do-nothing initializer H5FS_init().  Delete it.

* Document what changes to make if the default VFD changes.

* While I am here, change an `await_prior` flag on the terminator table
to `true` to match the previous, non-table-driven code that was here.
Found the oversight making the following changes:

NFCI: insert an empty line and copy over slightly-edited comments from
the previous version, where those comments still correctly explained how
library termination operated.

* NFCI: lower a staircase.

* Replace every occurrence of FUNC_ENTER_NOAPI_INIT(...) with H5_PUSH_FUNC
since that is all that that macro does any more.

Quiet a bunch of new warnings by changing FUNC_ENTER_NOAPI(...) to
FUNC_ENTER_NOAPI_NOERR and removing disused `done:` labels.

* NFCI: add curly braces around a multiline statement.

* Quiet a signed/unsigned comparison warning.

* Add some documentation about library initialization and shutdown.

* Make sure that the library is initialized, or else that initialization
is already underway, before performing any VFD's initialization.

* Committing clang-format changes

* Committing clang-format changes

* Reduce differences from `develop` branch.

* Always initialize `tot_init`.

* Committing clang-format changes

* Fix typo: H5SL_init initializes skip lists, not VOL.

* Remove H5_TERM_GLOBAL test in H5T_init.  H5T_init was unusual in that
it tested H5_TERM_GLOBAL and exited early if it was set.  No other
module initializers did that, and I cannot find any reason that should
be necessary.  Tests still pass when I remove it, so away it goes.

* Use HD prefix.

* Add function header comments.

* Drop the intermediate variable, it's only used once.

* Extract subroutine `H5FDperform_init(hid_t (*init)(void))` that
initializes the library, if necessary, before calling its VFD-initializer
argument.  Use H5FDperform_init in the definition of the symbols
H5FD_<vfd> (e.g., H5FD_SEC2), which may be evaluated before the library
is initialized, like so:

```
```

I implement H5FDperform_init in its own source file, H5FDperform.c,
and exclude that file from trace processing because the `bin/trace`
cannot deal with the function-pointer type.

* Straggler from last: add new source file src/H5FDperform.c.

* Committing clang-format changes

* Add a missing file to the MANIFEST.

* Switch to FUNC_ENTER_API_NOINIT in H5FDperform_init() and hbool_t in
H5_term_library().

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

1 files changed, 172 insertions, 133 deletions

diff --git a/src/H5.c b/src/H5.c
index 731f15a..27df917 100644
--- a/src/H5.c
+++ b/src/H5.c
@@ -32,6 +32,8 @@
 #include "H5SLprivate.h" /* Skip lists                               */
 #include "H5Tprivate.h"  /* Datatypes                                */
 
+#include "H5FDsec2.h" /* for H5FD_sec2_init() */
+
 /****************/
 /* Local Macros */
 /****************/
@@ -63,9 +65,6 @@ static int H5__mpi_delete_cb(MPI_Comm comm, int keyval, void *attr_val, int *fla
 /* Package Variables */
 /*********************/
 
-/* Package initialization variable */
-hbool_t H5_PKG_INIT_VAR = FALSE;
-
 /*****************************/
 /* Library Private Variables */
 /*****************************/
@@ -101,32 +100,30 @@ static H5_atclose_node_t *H5_atclose_head = NULL;
 /* Declare a free list to manage the H5_atclose_node_t struct */
 H5FL_DEFINE_STATIC(H5_atclose_node_t);
 
-/*--------------------------------------------------------------------------
-NAME
-    H5__init_package -- Initialize interface-specific information
-USAGE
-    herr_t H5__init_package()
-RETURNS
-    Non-negative on success/Negative on failure
-DESCRIPTION
-    Initializes any interface-specific data or routines.
---------------------------------------------------------------------------*/
-herr_t
-H5__init_package(void)
+/*-------------------------------------------------------------------------
+ * Function:    H5_default_vfd_init
+ *
+ * Purpose:     Initialize the default VFD.
+ *
+ * Return:      Success:        non-negative
+ *              Failure:        negative
+ *-------------------------------------------------------------------------
+ */
+static herr_t
+H5_default_vfd_init(void)
 {
-    herr_t ret_value = SUCCEED; /* Return value */
-
-    FUNC_ENTER_NOAPI_NOINIT
-
-    /* Run the library initialization routine, if it hasn't already ran */
-    if (!H5_INIT_GLOBAL && !H5_TERM_GLOBAL) {
-        if (H5_init_library() < 0)
-            HGOTO_ERROR(H5E_LIB, H5E_CANTINIT, FAIL, "unable to initialize library")
-    } /* end if */
+    herr_t ret_value = SUCCEED;
 
+    FUNC_ENTER_NOAPI(FAIL)
+    /* Load the hid_t for the default VFD for the side effect
+     * it has of initializing the default VFD.
+     */
+    if (H5FD_sec2_init() == H5I_INVALID_HID) {
+        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to load default VFD ID")
+    }
 done:
     FUNC_LEAVE_NOAPI(ret_value)
-} /* end H5__init_package() */
+}
 
 /*--------------------------------------------------------------------------
  * NAME
@@ -145,8 +142,13 @@ done:
 herr_t
 H5_init_library(void)
 {
+    size_t i;
     herr_t ret_value = SUCCEED;
 
+    /* Run the library initialization routine, if it hasn't already run */
+    if (H5_INIT_GLOBAL || H5_TERM_GLOBAL)
+        HGOTO_DONE(SUCCEED)
+
     /* Set the 'library initialized' flag as early as possible, to avoid
      * possible re-entrancy.
      */
@@ -257,26 +259,32 @@ H5_init_library(void)
      * The dataspace interface needs to be initialized so that future IDs for
      *   dataspaces work.
      */
-    if (H5E_init() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize error interface")
-    if (H5VL_init_phase1() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize vol interface")
-    if (H5P_init_phase1() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize property list interface")
-    if (H5AC_init() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize metadata caching interface")
-    if (H5L_init() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize link interface")
-    if (H5FS_init() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize FS interface")
-    if (H5S_init() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize dataspace interface")
-
+    /* clang-format off */
+    struct {
+        herr_t (*func)(void);
+        const char *descr;
+    } initializer[] = {
+        {H5E_init, "error"}
+    ,   {H5VL_init_phase1, "VOL"}
+    ,   {H5SL_init, "skip lists"}
+    ,   {H5FD_init, "VFD"}
+    ,   {H5_default_vfd_init, "default VFD"}
+    ,   {H5P_init_phase1, "property list"}
+    ,   {H5AC_init, "metadata caching"}
+    ,   {H5L_init, "link"}
+    ,   {H5S_init, "dataspace"}
     /* Finish initializing interfaces that depend on the interfaces above */
-    if (H5P_init_phase2() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize property list interface")
-    if (H5VL_init_phase2() < 0)
-        HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL, "unable to initialize vol interface")
+    ,   {H5P_init_phase2, "property list"}
+    ,   {H5VL_init_phase2, "VOL"}
+    };
+
+    for (i = 0; i < NELMTS(initializer); i++) {
+        if (initializer[i].func() < 0) {
+            HGOTO_ERROR(H5E_FUNC, H5E_CANTINIT, FAIL,
+                "unable to initialize %s interface", initializer[i].descr)
+        }
+    }
+    /* clang-format on */
 
     /* Debugging? */
     H5__debug_mask("-all");
@@ -300,9 +308,11 @@ done:
 void
 H5_term_library(void)
 {
-    int         pending, ntries = 0, n;
-    size_t      at = 0;
-    char        loop[1024];
+    int         pending, ntries   = 0;
+    char        loop[1024], *next = loop;
+    size_t      i;
+    size_t      nleft = sizeof(loop);
+    int         nprinted;
     H5E_auto2_t func;
 
 #ifdef H5_HAVE_THREADSAFE
@@ -346,108 +356,137 @@ H5_term_library(void)
         H5_atclose_head = NULL;
     } /* end if */
 
+    /* clang-format off */
+
     /*
      * Terminate each interface. The termination functions return a positive
      * value if they do something that might affect some other interface in a
      * way that would necessitate some cleanup work in the other interface.
      */
-#define DOWN(F)                                                                                              \
-    (((n = H5##F##_term_package()) && (at + 8) < sizeof loop)                                                \
-         ? (HDsprintf(loop + at, "%s%s", (at ? "," : ""), #F), at += HDstrlen(loop + at), n)                 \
-         : ((n > 0 && (at + 5) < sizeof loop) ? (HDsprintf(loop + at, "..."), at += HDstrlen(loop + at), n)  \
-                                              : n))
 
-    do {
-        pending = 0;
-
-        /* Try to organize these so the "higher" level components get shut
-         * down before "lower" level components that they might rely on. -QAK
-         */
+#define TERMINATOR(module, wait) {      \
+      .func = H5##module##_term_package \
+    , .name = #module                   \
+    , .completed = false                \
+    , .await_prior = wait               \
+    }
 
+    /*
+     * Termination is ordered by the `terminator` table so the "higher" level
+     * packages are shut down before "lower" level packages that they
+     * rely on:
+     */
+    struct {
+        int (*func)(void);       /* function to terminate the module; returns 0
+                                  * on success, >0 if termination was not
+                                  * completed and we should try to terminate
+                                  * some dependent modules, first.
+                                  */
+        const char *name;        /* name of the module */
+        hbool_t     completed;   /* true iff this terminator was already
+                                  * completed
+                                  */
+        const hbool_t await_prior;  /* true iff all prior terminators in the
+                                     * list must complete before this
+                                     * terminator is attempted
+                                     */
+    } terminator[] = {
         /* Close the event sets first, so that all asynchronous operations
-         *  complete before anything else attempts to shut down.
+         * complete before anything else attempts to shut down.
          */
-        pending += DOWN(ES);
-
-        /* Close down the user-facing interfaces, after the event sets */
-        if (pending == 0) {
-            /* Close the interfaces dependent on others */
-            pending += DOWN(L);
-
-            /* Close the "top" of various interfaces (IDs, etc) but don't shut
-             *  down the whole interface yet, so that the object header messages
-             *  get serialized correctly for entries in the metadata cache and the
-             *  symbol table entry in the superblock gets serialized correctly, etc.
-             *  all of which is performed in the 'F' shutdown.
-             */
-            pending += DOWN(A_top);
-            pending += DOWN(D_top);
-            pending += DOWN(G_top);
-            pending += DOWN(M_top);
-            pending += DOWN(R_top);
-            pending += DOWN(S_top);
-            pending += DOWN(T_top);
-        } /* end if */
-
+        TERMINATOR(ES, false)
+        /* Do not attempt to close down package L until after event sets
+         * have finished closing down.
+         */
+    ,   TERMINATOR(L, true)
+        /* Close the "top" of various interfaces (IDs, etc) but don't shut
+         * down the whole interface yet, so that the object header messages
+         * get serialized correctly for entries in the metadata cache and the
+         * symbol table entry in the superblock gets serialized correctly, etc.
+         * all of which is performed in the 'F' shutdown.
+         *
+         * The tops of packages A, D, G, M, S, T do not need to wait for L
+         * or previous packages to finish closing down.
+         */
+    ,   TERMINATOR(A_top, false)
+    ,   TERMINATOR(D_top, false)
+    ,   TERMINATOR(G_top, false)
+    ,   TERMINATOR(M_top, false)
+    ,   TERMINATOR(S_top, false)
+    ,   TERMINATOR(T_top, false)
         /* Don't shut down the file code until objects in files are shut down */
-        if (pending == 0)
-            pending += DOWN(F);
-
+    ,   TERMINATOR(F, true)
         /* Don't shut down the property list code until all objects that might
-         * use property lists are shut down */
-        if (pending == 0)
-            pending += DOWN(P);
-
+         * use property lists are shut down
+         */
+    ,   TERMINATOR(P, true)
         /* Wait to shut down the "bottom" of various interfaces until the
-         *      files are closed, so pieces of the file can be serialized
-         *      correctly.
+         * files are closed, so pieces of the file can be serialized
+         * correctly.
+         *
+         * Shut down the "bottom" of the attribute, dataset, group,
+         * reference, dataspace, and datatype interfaces, fully closing
+         * out the interfaces now.
          */
-        if (pending == 0) {
-            /* Shut down the "bottom" of the attribute, dataset, group,
-             *  reference, dataspace, and datatype interfaces, fully closing
-             *  out the interfaces now.
-             */
-            pending += DOWN(A);
-            pending += DOWN(D);
-            pending += DOWN(G);
-            pending += DOWN(M);
-            pending += DOWN(R);
-            pending += DOWN(S);
-            pending += DOWN(T);
-        } /* end if */
-
-        /* Don't shut down "low-level" components until "high-level" components
-         * have successfully shut down.  This prevents property lists and IDs
-         * from being closed "out from underneath" of the high-level objects
-         * that depend on them. -QAK
+    ,   TERMINATOR(A, true)
+    ,   TERMINATOR(D, false)
+    ,   TERMINATOR(G, false)
+    ,   TERMINATOR(M, false)
+    ,   TERMINATOR(S, false)
+    ,   TERMINATOR(T, false)
+        /* Wait to shut down low-level packages like AC until after
+         * the preceding high-level packages have shut down.  This prevents
+         * low-level objects from closing "out from underneath" their
+         * reliant high-level objects.
          */
-        if (pending == 0) {
-            pending += DOWN(AC);
-            /* Shut down the "pluggable" interfaces, before the plugin framework */
-            pending += DOWN(Z);
-            pending += DOWN(FD);
-            pending += DOWN(VL);
-            /* Don't shut down the plugin code until all "pluggable" interfaces (Z, FD, PL) are shut down */
-            if (pending == 0)
-                pending += DOWN(PL);
-            /* Don't shut down the error code until other APIs which use it are shut down */
-            if (pending == 0)
-                pending += DOWN(E);
-            /* Don't shut down the ID code until other APIs which use them are shut down */
-            if (pending == 0)
-                pending += DOWN(I);
-            /* Don't shut down the skip list code until everything that uses it is down */
-            if (pending == 0)
-                pending += DOWN(SL);
-            /* Don't shut down the free list code until everything that uses it is down */
-            if (pending == 0)
-                pending += DOWN(FL);
-            /* Don't shut down the API context code until _everything_ else is down */
-            if (pending == 0)
-                pending += DOWN(CX);
-        } /* end if */
+    ,   TERMINATOR(AC, true)
+        /* Shut down the "pluggable" interfaces, before the plugin framework */
+    ,   TERMINATOR(Z, false)
+    ,   TERMINATOR(FD, false)
+    ,   TERMINATOR(VL, false)
+        /* Don't shut down the plugin code until all "pluggable" interfaces
+         * (Z, FD, PL) are shut down
+         */
+    ,   TERMINATOR(PL, true)
+        /* Shut down the following packages in strictly the order given
+         * by the table.
+         */
+    ,   TERMINATOR(E, true)
+    ,   TERMINATOR(I, true)
+    ,   TERMINATOR(SL, true)
+    ,   TERMINATOR(FL, true)
+    ,   TERMINATOR(CX, true)
+    };
+
+    do {
+        pending = 0;
+        for (i = 0; i < NELMTS(terminator); i++) {
+            if (terminator[i].completed)
+                continue;
+            if (pending != 0 && terminator[i].await_prior)
+                break;
+            if (terminator[i].func() == 0) {
+                terminator[i].completed = true;
+                continue;
+            }
+
+            /* log a package when its terminator needs to be retried */
+            pending++;
+            nprinted = HDsnprintf(next, nleft, "%s%s",
+                (next != loop) ? "," : "", terminator[i].name);
+            if (nprinted < 0)
+                continue;
+            if ((size_t)nprinted >= nleft)
+                nprinted = HDsnprintf(next, nleft, "...");
+            if (nprinted < 0 || (size_t)nprinted >= nleft)
+                continue;
+            nleft -= (size_t)nprinted;
+            next += nprinted;
+        }
     } while (pending && ntries++ < 100);
 
+    /* clang-format on */
+
     if (pending) {
         /* Only display the error message if the user is interested in them. */
         if (func) {