Merge doxygen2 into develop (#553)

* Fixed warnings and started H5Epublic.h.

* Include H5FD* headers to correctly resolve references.

* Doxygen2 (#330)

* H5Eauto_is_v2.

* Added a few more calls.

* Added a few more H5E calls.

* First cut of H5E v2.

* Added the deprecated v1 calls.

* Updated spacing.

* Once more.

* Taking some inspiration from Eigen3.

* Add doxygen for the assigned functions: H5Pregister1,H5Pinsert1,H5Pen… (#352)

* Add doxygen for the assigned functions: H5Pregister1,H5Pinsert1,H5Pencode1, H5Pget_filter_by_id1,H5Pget_version, H5Pset_file_space,H5Pget_file_space. Someone already adds H5Pget_filter1. Also fixs an extra parameter 'close' call back function for HPregister2.

* doxygen work. fixs format by using clang-format.

* doxgen work for H5Pregister1 etc. Addressed Barbara and Gerd's comments.
For Quincey's comments, since we are not supposed to change the source code.
I leave this to future improvements.

* added documentation for H5P APIs (#350)

* add documenation for H5Pget_buffer,H5Pget_data_transform,H5Pget_edc_check,H5Pget_hyper_vector_size,H5Pget_preserve,H5Pget_type_conv_cb,H5Pget_vlen_mem_manager,H5Pset_btree_ratios

* format corrections

* fixed grammer

* fixed herr_t

* Better name.

* A fresh look.

* add doxygen to H5Ppublic.h

* use attention instead of warning

* Add doxygen comments in H5Ppublic.h (#375)

* Add doxygen comments in H5Ppublic.h

* H5Pset_meta_block_size
* H5Pset_metadata_read_attempts
* H5Pset_multi_type
* H5Pset_object_flush_cb
* H5Pset_sieve_buf_size
* H5Pset_small_data_block_size
* H5Pset_all_coll_metadata_ops
* H5Pget_all_coll_metadata_ops

* Add DOXYGEN_EXAMPLES_DIR to src/CMakeLists.txt

* Fix clang-format errors

* Fix filenames in doxygen/examples

* add doxygen to H5Ppublic.h (#378)

* add doxygen to H5Ppublic.h

* use attention instead of warning

Co-authored-by: Kimmy Mu <kmu@hdfgroup.org>

* Revert "add doxygen to H5Ppublic.h (#378)"

This reverts commit 2ee1821b138a5c00b15ea57ce9e950367480f5f2.

* Updated Doxygen variables.

* I forgot to copy two images.

* Enable desktop search by default.

* Add my assigned Doxygen documentation.

* Remove whitespace at EOL.  Appease clang-format.

* Addressed Chris' comments.

* Added an alias for asynchronous functions.

* One space is enough for all of us.

* Slightly restructured RM page.

* address some issues

* reformatting

* Style external links.

* reformatting

* reformatting

* Added "Metadata Caching in HDF5" as a technical note example.

* Revise this soon!

* Added specification examples.

* Fixed references.

* Added H5AC cache image stuff and file format study.

* Added older FMT versions. Where did 1.0 go?

* Updated C/C++ note and replaced ambiguous labels.

* Reformat source with clang v10.0.1.

* Added the VFL technical note.

* Added what I believe might be called version 1.0 of the format.

* Added the remaining specs.

* Added H5Z callback documentation and fixed a few mistakes.

* Added dox for deprecated H5G calls and fixed a few snippet blockIDs.

* clang-format happy?

* Ok?

* Bonus track: Deprecated H5D functions.

* Carry over the more detailed group description.

* Added documentation for the missing and deprecated H5R calls.

* Life is easier and less repetitive w/ snippets. Use them!

* Eliminate the snippet block ID artifacts in the HTML rendering.

* Fixed snippet HTML artifacts and added a few missing calls.

* Under 20 H5Ps to go!

* Almost complete!

* "This is a form of pedantry up with which I will not put." (Churchill)

* Let's not waste as much space on bulleted lists!

* First complete (?) draft of the Doxygen-based RM.

* Completeness check and minor fixes along the way.

* Pedantry.

* Adding missing H5FD calls checkpoint.

* Pedantry.

* More pedantry.

* Added H5Pset_fapl_log.

* First draft of H5ES.

* Fixed warnings.

* Prep. for map module.

* First cut of the map module.

* Pedantry.

* Possible H5F introduction.

* Fix the indentation.

* Pedantry.

* Ditto.

* Thanks to the reviewers for their comments.

* Added missing images.

* Line numbers are a distraction here.

* More examples, references, and clean-up. Don't repeat yourself!

* Clang pedantry.

* Ditto.

* More reviewer comments...

* Templatized references and cleaned up \todos.

* Committing clang-format changes

* Fixed MANIFEST.

* Addressed Quincey's comments. (OCPLs)

* Fixed a few more \todo items.

* Fixed more \todo items.

* Added attribute life cycle.

* Forgot the examples file.

* Committing clang-format changes

* Pedantry.

* Live and learn!

* Added a sample H5D life cycle.

* Committing clang-format changes

* Pedantry.

Co-authored-by: kyang2014 <kyang2014@users.noreply.github.com>
Co-authored-by: Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Kimmy Mu <kmu@hdfgroup.org>
Co-authored-by: Christopher Hogan <ChristopherHogan@users.noreply.github.com>
Co-authored-by: jya-kmu <53388330+jya-kmu@users.noreply.github.com>
Co-authored-by: David Young <dyoung@hdfgroup.org>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

1 files changed, 787 insertions, 0 deletions

diff --git a/doxygen/examples/ThreadSafeLibrary.html b/doxygen/examples/ThreadSafeLibrary.html
new file mode 100644
index 0000000..8daf386
--- /dev/null
+++ b/doxygen/examples/ThreadSafeLibrary.html
@@ -0,0 +1,787 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
+		"http://www.w3.org/TR/REC-html40/loose.dtd">
+<html lang="en-US">
+<head>
+  <title>Thread Safe Library</title>
+</head>
+
+<h1>1. Library header files and conditional compilation</h1>
+
+<p>
+The following code is placed at the beginning of H5private.h:
+</p>
+
+<blockquote>
+  <pre>
+  #ifdef H5_HAVE_THREADSAFE
+  #include &lt;pthread.h&gt;
+  #endif
+  </pre>
+</blockquote>
+
+<p>
+<code>H5_HAVE_THREADSAFE</code> is defined when the HDF-5 library is
+compiled with the --enable-threadsafe configuration option. In general,
+code for the non-threadsafe version of HDF-5 library are placed within
+the <code>#else</code> part of the conditional compilation. The exception
+to this rule are the changes to the <code>FUNC_ENTER</code> (in
+H5private.h), <code>HRETURN</code> and <code>HRETURN_ERROR</code> (in
+H5Eprivate.h) macros (see section 3.2).
+</p>
+
+
+<h1>2. Global variables/structures</h1>
+
+<h2>2.1 Global library initialization variable</h2>
+
+<p>
+In the threadsafe implementation, the global library initialization
+variable <code>H5_libinit_g</code> is changed to a global structure
+consisting of the variable with its associated lock (locks are explained
+in section 4.1):
+</p>
+
+<blockquote>
+  <pre>
+    hbool_t  H5_libinit_g = FALSE;
+  </pre>
+</blockquote>
+
+<p>
+becomes
+</p>
+
+<blockquote>
+  <pre>
+    H5_api_t H5_g;
+  </pre>
+</blockquote>
+
+<p>
+where <code>H5_api_t</code> is
+</p>
+
+<blockquote>
+  <pre>
+    typedef struct H5_api_struct {
+      H5_mutex_t init_lock;           /* API entrance mutex */
+      hbool_t H5_libinit_g;
+    } H5_api_t;
+  </pre>
+</blockquote>
+
+<p>
+All former references to <code>H5_libinit_g</code> in the library are now
+made using the macro <code>H5_INIT_GLOBAL</code>. If the threadsafe
+library is to be used, the macro is set to <code>H5_g.H5_libinit_g</code>
+instead.
+</p>
+
+<h2>2.2 Global serialization variable</h2>
+
+<p>
+A new global boolean variable <code>H5_allow_concurrent_g</code> is used
+to determine if multiple threads are allowed to an API call
+simultaneously. This is set to <code>FALSE</code>.
+</p>
+
+<p>
+All APIs that are allowed to do so have their own local variable that
+shadows the global variable and is set to <code>TRUE</code>. In phase 1,
+no such APIs exist.
+</p>
+
+<p>
+It is defined in <code>H5.c</code> as follows:
+</p>
+
+<blockquote>
+  <pre>
+    hbool_t H5_allow_concurrent_g = FALSE;
+  </pre>
+</blockquote>
+
+<h2>2.3 Global thread initialization variable</h2>
+
+<p>
+The global variable <code>H5_first_init_g</code> of type
+<code>pthread_once_t</code> is used to allow only the first thread in the
+application process to call an initialization function using
+<code>pthread_once</code>. All subsequent calls to
+<code>pthread_once</code> by any thread are disregarded.
+</p>
+
+<p>
+The call sets up the mutex in the global structure <code>H5_g</code> (see
+section 3.1) via an initialization function
+<code>H5_first_thread_init</code>. The first thread initialization
+function is described in section 4.2.
+</p>
+
+<p>
+<code>H5_first_init_g</code> is defined in <code>H5.c</code> as follows:
+</p>
+
+<blockquote>
+  <pre>
+    pthread_once_t H5_first_init_g = PTHREAD_ONCE_INIT;
+  </pre>
+</blockquote>
+
+<h2>2.4 Global key for per-thread error stacks</h2>
+
+<p>
+A global pthread-managed key <code>H5_errstk_key_g</code> is used to
+allow pthreads to maintain a separate error stack (of type
+<code>H5E_t</code>) for each thread. This is defined in <code>H5.c</code>
+as:
+</p>
+
+<blockquote>
+  <pre>
+    pthread_key_t H5_errstk_key_g;
+  </pre>
+</blockquote>
+
+<p>
+Error stack management is described in section 4.3.
+</p>
+
+<h2>2.5 Global structure and key for thread cancellation prevention</h2>
+
+<p>
+We need to preserve the thread cancellation status of each thread
+individually by using a key <code>H5_cancel_key_g</code>. The status is
+preserved using a structure (of type <code>H5_cancel_t</code>) which
+maintains the cancellability state of the thread before it entered the
+library and a count (which works very much like the recursive lock
+counter) which keeps track of the number of API calls the thread makes
+within the library.
+</p>
+
+<p>
+The structure is defined in <code>H5private.h</code> as:
+</p>
+
+<blockquote>
+  <pre>
+    /* cancelability structure */
+    typedef struct H5_cancel_struct {
+      int previous_state;
+      unsigned int cancel_count;
+    } H5_cancel_t;
+  </pre>
+</blockquote>
+
+<p>
+Thread cancellation is described in section 4.4.
+</p>
+
+
+<h1>3. Changes to Macro expansions</h1>
+
+<h2>3.1 Changes to FUNC_ENTER</h2>
+
+<p>
+The <code>FUNC_ENTER</code> macro is now extended to include macro calls
+to initialize first threads, disable cancellability and wraps a lock
+operation around the checking of the global initialization flag. It
+should be noted that the cancellability should be disabled before
+acquiring the lock on the library. Doing so otherwise would allow the
+possibility that the thread be cancelled just after it has acquired the
+lock on the library and in that scenario, if the cleanup routines are not
+properly set, the library would be permanently locked out.
+</p>
+
+<p>
+The additional macro code and new macro definitions can be found in
+Appendix E.1 to E.5. The changes are made in <code>H5private.h</code>.
+</p>
+
+<h2>3.2 Changes to HRETURN and HRETURN_ERROR</h2>
+
+<p>
+The <code>HRETURN</code> and <code>HRETURN_ERROR</code> macros are the
+counterparts to the <code>FUNC_ENTER</code> macro described in section
+3.1. <code>FUNC_LEAVE</code> makes a macro call to <code>HRETURN</code>,
+so it is also covered here.
+</p>
+
+<p>
+The basic changes to these two macros involve adding macro calls to call
+an unlock operation and re-enable cancellability if necessary.  It should
+be noted that the cancellability should be re-enabled only after the
+thread has released the lock to the library. The consequence of doing
+otherwise would be similar to that described in section 3.1.
+</p>
+
+<p>
+The additional macro code and new macro definitions can be found in
+Appendix E.9 to E.9. The changes are made in <code>H5Eprivate.h</code>.
+</p>
+
+<h1>4. Implementation of threadsafe functionality</h1>
+
+<h2>4.1 Recursive Locks</h2>
+
+<p>
+A recursive mutex lock m allows a thread t1 to successfully lock m more
+than once without blocking t1. Another thread t2 will block if t2 tries
+to lock m while t1 holds the lock to m. If t1 makes k lock calls on m,
+then it also needs to make k unlock calls on m before it releases the
+lock.
+</p>
+
+<p>
+Our implementation of recursive locks is built on top of a pthread mutex
+lock (which is not recursive). It makes use of a pthread condition
+variable to have unsuccessful threads wait on the mutex.  Waiting threads
+are awaken by a signal from the final unlock call made by the thread
+holding the lock.
+</p>
+
+<p>
+Recursive locks are defined to be the following type
+(<code>H5private.h</code>):
+</p>
+
+<blockquote>
+  <pre>
+    typedef struct H5_mutex_struct {
+      pthread_t owner_thread;         /* current lock owner */
+      pthread_mutex_t atomic_lock;    /* lock for atomicity of new mechanism */
+      pthread_cond_t cond_var;        /* condition variable */
+      unsigned int lock_count;
+    } H5_mutex_t;
+  </pre>
+</blockquote>
+
+<p>
+Detailed implementation code can be found in Appendix A. The
+implementation changes are made in <code>H5TS.c</code>.
+</p>
+
+<h2>4.2 First thread initialization</h2>
+
+<p>
+Because the mutex lock associated with a recursive lock cannot be
+statically initialized, a mechanism is required to initialize the
+recursive lock associated with <code>H5_g</code> so that it can be used
+for the first time.
+</p>
+
+<p>
+The pthreads library allows this through the pthread_once call which as
+described in section 3.3 allows only the first thread accessing the
+library in an application to initialize <code>H5_g</code>.
+</p>
+
+<p>
+In addition to initializing <code>H5_g</code>, it also initializes the
+key (see section 3.4) for use with per-thread error stacks (see section
+4.3).
+</p>
+
+<p>
+The first thread initialization mechanism is implemented as the function
+call <code>H5_first_thread_init()</code> in <code>H5TS.c</code>. This is
+described in appendix B.
+</p>
+
+<h2>4.3 Per-thread error stack management</h2>
+
+<p>
+Pthreads allows individual threads to access dynamic and persistent
+per-thread data through the use of keys. Each key is associated with
+a table that maps threads to data items. Keys can be initialized by
+<code>pthread_key_create()</code> in pthreads (see sections 3.4 and 4.2).
+Per-thread data items are accessed using a key through the
+<code>pthread_getspecific()</code> and <code>pthread_setspecific()</code>
+calls to read and write to the association table respectively.
+</p>
+
+<p>
+Per-thread error stacks are accessed through the key
+<code>H5_errstk_key_g</code> which is initialized by the first thread
+initialization call (see section 4.2).
+</p>
+
+<p>
+In the non-threadsafe version of the library, there is a global stack
+variable <code>H5E_stack_g[1]</code> which is no longer defined in the
+threadsafe version. At the same time, the macro call to gain access to
+the error stack <code>H5E_get_my_stack</code> is changed from:
+</p>
+
+<blockquote>
+  <pre>
+    #define H5E_get_my_stack() (H5E_stack_g+0)
+  </pre>
+</blockquote>
+
+<p>
+to:
+</p>
+
+<blockquote>
+  <pre>
+    #define H5E_get_my_stack() H5E_get_stack()
+  </pre>
+</blockquote>
+
+<p>
+where <code>H5E_get_stack()</code> is a surrogate function that does the
+following operations:
+</p>
+
+<ol>
+  <li>if a thread is attempting to get an error stack for the first
+			time, the error stack is dynamically allocated for the thread and
+			associated with <code>H5_errstk_key_g</code> using
+			<code>pthread_setspecific()</code>. The way we detect if it is the
+			first time is through <code>pthread_getspecific()</code> which
+			returns <code>NULL</code> if no previous value is associated with
+			the thread using the key.</li>
+
+  <li>if <code>pthread_getspecific()</code> returns a non-null value,
+			then that is the pointer to the error stack associated with the
+			thread and the stack can be used as usual.</li>
+</ol>
+
+<p>
+A final change to the error reporting routines is as follows; the current
+implementation reports errors to always be detected at thread 0. In the
+threadsafe implementation, this is changed to report the number returned
+by a call to <code>pthread_self()</code>.
+</p>
+
+<p>
+The change in code (reflected in <code>H5Eprint</code> of file
+<code>H5E.c</code>) is as follows:
+</p>
+
+<blockquote>
+	<pre>
+    #ifdef H5_HAVE_THREADSAFE
+      fprintf (stream, "HDF5-DIAG: Error detected in thread %d."
+               ,pthread_self());
+    #else
+      fprintf (stream, "HDF5-DIAG: Error detected in thread 0.");
+    #endif
+  </pre>
+</blockquote>
+
+<p>
+Code for <code>H5E_get_stack()</code> can be found in Appendix C. All the
+above changes were made in <code>H5E.c</code>.
+</p>
+
+<h2>4.4 Thread Cancellation safety</h2>
+
+<p>
+To prevent thread cancellations from killing a thread while it is in the
+library, we maintain per-thread information about the cancellability
+status of the thread before it entered the library so that we can restore
+that same status when the thread leaves the library.
+</p>
+
+<p>
+By <i>enter</i> and <i>leave</i> the library, we mean the points when a
+thread makes an API call from a user application and the time that API
+call returns. Other API or callback function calls made from within that
+API call are considered <i>within</i> the library.
+</p>
+
+<p>
+Because other API calls may be made from within the first API call, we
+need to maintain a counter to determine which was the first and
+correspondingly the last return.
+</p>
+
+<p>
+When a thread makes an API call, the macro <code>H5_API_SET_CANCEL</code>
+calls the worker function <code>H5_cancel_count_inc()</code> which does
+the following:
+</p>
+
+<ol>
+  <li>if this is the first time the thread has entered the library,
+      a new cancellability structure needs to be assigned to it.</li>
+  <li>if the thread is already within the library when the API call is
+			made, then cancel_count is simply incremented. Otherwise, we set
+			the cancellability state to <code>PTHREAD_CANCEL_DISABLE</code>
+			while storing the previous state into the cancellability structure.
+			<code>cancel_count</code> is also incremented in this case.</li>
+</ol>
+
+<p>
+When a thread leaves an API call, the macro
+<code>H5_API_UNSET_CANCEL</code> calls the worker function
+<code>H5_cancel_count_dec()</code> which does the following:
+</p>
+
+<ol>
+  <li>if <code>cancel_count</code> is greater than 1, indicating that the
+			thread is not yet about to leave the library, then
+			<code>cancel_count</code> is simply decremented.</li>
+  <li>otherwise, we reset the cancellability state back to its original
+			state before it entered the library and decrement the count (back
+			to zero).</li>
+</ol>
+
+<p>
+<code>H5_cancel_count_inc</code> and <code>H5_cancel_count_dec</code> are
+described in Appendix D and may be found in <code>H5TS.c</code>.
+</p>
+
+<h1>5. Test programs</h1>
+
+<p>
+Except where stated, all tests involve 16 simultaneous threads that make
+use of HDF-5 API calls without any explicit synchronization typically
+required in a non-threadsafe environment.
+</p>
+
+<h2>5.1 Data set create and write</h2>
+
+<p>
+The test program sets up 16 threads to simultaneously create 16
+different datasets named from <i>zero</i> to <i>fifteen</i> for a single
+file and then writing an integer value into that dataset equal to the
+dataset's named value.
+</p>
+
+<p>
+The main thread would join with all 16 threads and attempt to match the
+resulting HDF-5 file with expected results - that each dataset contains
+the correct value (0 for <i>zero</i>, 1 for <i>one</i> etc ...) and all
+datasets were correctly created.
+</p>
+
+<p>
+The test is implemented in the file <code>ttsafe_dcreate.c</code>.
+</p>
+
+<h2>5.2 Test on error stack</h2>
+
+<p>
+The error stack test is one in which 16 threads simultaneously try to
+create datasets with the same name. The result, when properly serialized,
+should be equivalent to 16 attempts to create the dataset with the same
+name.
+</p>
+
+<p>
+The error stack implementation runs correctly if it reports 15 instances
+of the dataset name conflict error and finally generates a correct HDF-5
+containing that single dataset. Each thread should report its own stack
+of errors with a thread number associated with it.
+</p>
+
+<p>
+The test is implemented in the file <code>ttsafe_error.c</code>.
+</p>
+
+<h2>5.3 Test on cancellation safety</h2>
+
+<p>
+The main idea in thread cancellation safety is as follows; a child thread
+is spawned to create and write to a dataset. Following that, it makes a
+<code>H5Diterate</code> call on that dataset which activates a callback
+function.
+</p>
+
+<p>
+A deliberate barrier is invoked at the callback function which waits for
+both the main and child thread to arrive at that point. After that
+happens, the main thread proceeds to make a thread cancel call on the
+child thread while the latter sleeps for 3 seconds before proceeding to
+write a new value to the dataset.
+</p>
+
+<p>
+After the iterate call, the child thread logically proceeds to wait
+another 3 seconds before writing another newer value to the dataset.
+</p>
+
+<p>
+The test is correct if the main thread manages to read the second value
+at the end of the test. This means that cancellation did not take place
+until the end of the iteration call despite of the 3 second wait within
+the iteration callback and the extra dataset write operation.
+Furthermore, the cancellation should occur before the child can proceed
+to write the last value into the dataset.
+</p>
+
+<h2>5.4 Test on attribute creation</h2>
+
+<p>
+A main thread makes 16 threaded calls to <code>H5Acreate</code> with a
+generated name for each attribute. Sixteen attributes should be created
+for the single dataset in random (chronological) order and receive values
+depending on its generated attribute name (e.g. <i>attrib010</i> would
+receive the value 10).
+</p>
+
+<p>
+After joining with all child threads, the main thread proceeds to read
+each attribute by generated name to see if the value tallies. Failure is
+detected if the attribute name does not exist (meaning they were never
+created) or if the wrong values were read back.
+</p>
+
+<h1>A. Recursive Lock implementation code</h1>
+
+<blockquote>
+  <pre>
+  void H5_mutex_init(H5_mutex_t *H5_mutex)
+  {
+    H5_mutex-&gt;owner_thread = NULL;
+    pthread_mutex_init(&amp;H5_mutex-&gt;atomic_lock, NULL);
+    pthread_cond_init(&amp;H5_mutex-&gt;cond_var, NULL);
+    H5_mutex-&gt;lock_count = 0;
+  }
+
+  void H5_mutex_lock(H5_mutex_t *H5_mutex)
+  {
+    pthread_mutex_lock(&amp;H5_mutex-&gt;atomic_lock);
+
+    if (pthread_equal(pthread_self(), H5_mutex-&gt;owner_thread)) {
+    	/* already owned by self - increment count */
+    	H5_mutex-&gt;lock_count++;
+    } else {
+    	if (H5_mutex-&gt;owner_thread == NULL) {
+    		/* no one else has locked it - set owner and grab lock */
+    		H5_mutex-&gt;owner_thread = pthread_self();
+    		H5_mutex-&gt;lock_count = 1;
+    	} else {
+    		/* if already locked by someone else */
+    		while (1) {
+    			pthread_cond_wait(&amp;H5_mutex-&gt;cond_var, &amp;H5_mutex-&gt;atomic_lock);
+
+    			if (H5_mutex-&gt;owner_thread == NULL) {
+    				H5_mutex-&gt;owner_thread = pthread_self();
+    				H5_mutex-&gt;lock_count = 1;
+    				break;
+    			} /* else do nothing and loop back to wait on condition*/
+    		}
+    	}
+    }
+
+    pthread_mutex_unlock(&amp;H5_mutex-&gt;atomic_lock);
+  }
+
+  void H5_mutex_unlock(H5_mutex_t *H5_mutex)
+  {
+    pthread_mutex_lock(&amp;H5_mutex-&gt;atomic_lock);
+    H5_mutex-&gt;lock_count--;
+
+    if (H5_mutex-&gt;lock_count == 0) {
+    	H5_mutex-&gt;owner_thread = NULL;
+    	pthread_cond_signal(&amp;H5_mutex-&gt;cond_var);
+    }
+    pthread_mutex_unlock(&amp;H5_mutex-&gt;atomic_lock);
+  }
+  </pre>
+</blockquote>
+
+<h1>B. First thread initialization</h1>
+
+<blockquote>
+  <pre>
+  void H5_first_thread_init(void)
+  {
+    /* initialize global API mutex lock                      */
+    H5_g.H5_libinit_g = FALSE;
+    H5_g.init_lock.owner_thread = NULL;
+    pthread_mutex_init(&amp;H5_g.init_lock.atomic_lock, NULL);
+    pthread_cond_init(&amp;H5_g.init_lock.cond_var, NULL);
+    H5_g.init_lock.lock_count = 0;
+
+    /* initialize key for thread-specific error stacks       */
+    pthread_key_create(&amp;H5_errstk_key_g, NULL);
+
+    /* initialize key for thread cancellability mechanism    */
+    pthread_key_create(&amp;H5_cancel_key_g, NULL);
+  }
+  </pre>
+</blockquote>
+
+
+<h1>C. Per-thread error stack acquisition</h1>
+
+<blockquote>
+  <pre>
+  H5E_t *H5E_get_stack(void)
+  {
+    H5E_t *estack;
+
+    if (estack = pthread_getspecific(H5_errstk_key_g)) {
+    	return estack;
+    } else {
+    	/* no associated value with current thread - create one */
+    	estack = (H5E_t *)malloc(sizeof(H5E_t));
+    	pthread_setspecific(H5_errstk_key_g, (void *)estack);
+    	return estack;
+    }
+  }
+  </pre>
+</blockquote>
+
+<h1>D. Thread cancellation mechanisms</h1>
+
+<blockquote>
+  <pre>
+  void H5_cancel_count_inc(void)
+  {
+    H5_cancel_t *cancel_counter;
+
+    if (cancel_counter = pthread_getspecific(H5_cancel_key_g)) {
+      /* do nothing here */
+    } else {
+      /*
+       * first time thread calls library - create new counter and
+       * associate with key
+       */
+      cancel_counter = (H5_cancel_t *)malloc(sizeof(H5_cancel_t));
+      cancel_counter-&gt;cancel_count = 0;
+      pthread_setspecific(H5_cancel_key_g, (void *)cancel_counter);
+    }
+
+    if (cancel_counter-&gt;cancel_count == 0) {
+      /* thread entering library */
+      pthread_setcancelstate(PTHREAD_CANCEL_DISABLE,
+                             &amp;(cancel_counter-&gt;previous_state));
+    }
+
+    cancel_counter-&gt;cancel_count++;
+  }
+
+  void H5_cancel_count_dec(void)
+  {
+    H5_cancel_t *cancel_counter = pthread_getspecific(H5_cancel_key_g);
+
+    if (cancel_counter-&gt;cancel_count == 1)
+      pthread_setcancelstate(cancel_counter-&gt;previous_state, NULL);
+
+    cancel_counter-&gt;cancel_count--;
+  }
+  </pre>
+</blockquote>
+
+<h1>E. Macro expansion codes</h1>
+
+<h2>E.1 <code>FUNC_ENTER</code></h2>
+
+<blockquote>
+  <pre>
+  /* Initialize the library */                                \
+  H5_FIRST_THREAD_INIT                                        \
+  H5_API_UNSET_CANCEL                                         \
+  H5_API_LOCK_BEGIN                                           \
+    if (!(H5_INIT_GLOBAL)) {                                  \
+      H5_INIT_GLOBAL = TRUE;                                  \
+        if (H5_init_library() &lt; 0) {                          \
+          HRETURN_ERROR (H5E_FUNC, H5E_CANTINIT, err,         \
+                        "library initialization failed");     \
+        }                                                     \
+    }                                                         \
+    H5_API_LOCK_END                                           \
+             :
+             :
+             :
+  </pre>
+</blockquote>
+
+<h2>E.2 <code>H5_FIRST_THREAD_INIT</code></h2>
+
+<blockquote>
+  <pre>
+  /* Macro for first thread initialization */
+  #define H5_FIRST_THREAD_INIT                                \
+    pthread_once(&amp;H5_first_init_g, H5_first_thread_init);
+  </pre>
+</blockquote>
+
+
+<h2>E.3 <code>H5_API_UNSET_CANCEL</code></h2>
+
+<blockquote>
+  <pre>
+  #define H5_API_UNSET_CANCEL                                 \
+    if (H5_IS_API(FUNC)) {                                    \
+      H5_cancel_count_inc();                                  \
+    }
+  </pre>
+</blockquote>
+
+
+<h2>E.4 <code>H5_API_LOCK_BEGIN</code></h2>
+
+<blockquote>
+  <pre>
+  #define H5_API_LOCK_BEGIN                                   \
+     if (H5_IS_API(FUNC)) {                                   \
+       H5_mutex_lock(&amp;H5_g.init_lock);
+  </pre>
+</blockquote>
+
+
+<h2>E.5 <code>H5_API_LOCK_END</code></h2>
+
+<blockquote>
+  <pre>
+  #define H5_API_LOCK_END }
+  </pre>
+</blockquote>
+
+
+<h2>E.6 <code>HRETURN</code> and <code>HRETURN_ERROR</code></h2>
+
+<blockquote>
+  <pre>
+            :
+            :
+    H5_API_UNLOCK_BEGIN                                       \
+    H5_API_UNLOCK_END                                         \
+    H5_API_SET_CANCEL                                         \
+    return ret_val;                                           \
+  }
+  </pre>
+</blockquote>
+
+<h2>E.7 <code>H5_API_UNLOCK_BEGIN</code></h2>
+
+<blockquote>
+  <pre>
+  #define H5_API_UNLOCK_BEGIN                                 \
+    if (H5_IS_API(FUNC)) {                                    \
+      H5_mutex_unlock(&amp;H5_g.init_lock);
+  </pre>
+</blockquote>
+
+<h2>E.8 <code>H5_API_UNLOCK_END</code></h2>
+
+<blockquote>
+  <pre>
+  #define H5_API_UNLOCK_END }
+  </pre>
+</blockquote>
+
+
+<h2>E.9 <code>H5_API_SET_CANCEL</code></h2>
+
+<blockquote>
+  <pre>
+  #define H5_API_SET_CANCEL                                   \
+    if (H5_IS_API(FUNC)) {                                    \
+      H5_cancel_count_dec();                                  \
+    }
+  </pre>
+</blockquote>
+
+<h2>By Chee Wai Lee</h2>
+<h4>By Bill Wendling</h4>
+
+</body>
+</html>