diff options
| author | Bill Wendling <wendling@ncsa.uiuc.edu> | 2000-10-27 15:58:29 (GMT) |
|---|---|---|
| committer | Bill Wendling <wendling@ncsa.uiuc.edu> | 2000-10-27 15:58:29 (GMT) |
| commit | 4dee0b2ace5081306a6e2db4f3e0689dfec96037 (patch) | |
| tree | 7ecd279eaae7bcbe8bee24b9cda8eede59d21221 | |
| parent | 9fe247782e75a45574a30100e5572b504132fc32 (diff) | |
| download | hdf5-4dee0b2ace5081306a6e2db4f3e0689dfec96037.zip hdf5-4dee0b2ace5081306a6e2db4f3e0689dfec96037.tar.gz hdf5-4dee0b2ace5081306a6e2db4f3e0689dfec96037.tar.bz2 | |
[svn-r2740] Purpose:
Documentation of TS Library
Description:
This is the document Chee Wai wrote up about the thread safe
version of the HDF5 library. I just put it in HTML format and
checked it in...
Platforms tested:
Netscrape
| -rw-r--r-- | doc/html/TechNotes/ThreadSafeLibrary.html | 794 |
1 files changed, 794 insertions, 0 deletions
diff --git a/doc/html/TechNotes/ThreadSafeLibrary.html b/doc/html/TechNotes/ThreadSafeLibrary.html new file mode 100644 index 0000000..1266eed --- /dev/null +++ b/doc/html/TechNotes/ThreadSafeLibrary.html @@ -0,0 +1,794 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" + "http://www.w3.org/TR/REC-html40/loose.dtd"> +<html lang="en-US"> +<head> + <title>Thread Safe Library</title> +</head> + +<body bgcolor=#ffffff> + +<center><h1>HDF5 Thread Safe library</h1></center> + +<p> + +<h1>1. Library header files and conditional compilation</h1> + +<p> +The following code is placed at the beginning of H5private.h: +</p> + +<blockquote> + <pre> + #ifdef H5_HAVE_THREADSAFE + #include <pthread.h> + #endif + </pre> +</blockquote> + +<p> +<code>H5_HAVE_THREADSAFE</code> is defined when the HDF-5 library is +compiled with the --enable-threadsafe configuration option. In general, +code for the non-threadsafe version of HDF-5 library are placed within +the <code>#else</code> part of the conditional compilation. The exception +to this rule are the changes to the <code>FUNC_ENTER</code> (in +H5private.h), <code>HRETURN</code> and <code>HRETURN_ERROR</code> (in +H5Eprivate.h) macros (see section 3.2). +</p> + + +<h1>2. Global variables/structures</h1> + +<h2>2.1 Global library initialization variable</h2> + +<p> +In the threadsafe implementation, the global library initialization +variable <code>H5_libinit_g</code> is changed to a global structure +consisting of the variable with its associated lock (locks are explained +in section 4.1): +</p> + +<blockquote> + <pre> + hbool_t H5_libinit_g = FALSE; + </pre> +</blockquote> + +<p> +becomes +</p> + +<blockquote> + <pre> + H5_api_t H5_g; + </pre> +</blockquote> + +<p> +where <code>H5_api_t</code> is +</p> + +<blockquote> + <pre> + typedef struct H5_api_struct { + H5_mutex_t init_lock; /* API entrance mutex */ + hbool_t H5_libinit_g; + } H5_api_t; + </pre> +</blockquote> + +<p> +All former references to <code>H5_libinit_g</code> in the library are now +made using the macro <code>H5_INIT_GLOBAL</code>. If the threadsafe +library is to be used, the macro is set to <code>H5_g.H5_libinit_g</code> +instead. +</p> + +<h2>2.2 Global serialization variable</h2> + +<p> +A new global boolean variable <code>H5_allow_concurrent_g</code> is used +to determine if multiple threads are allowed to an API call +simultaneously. This is set to <code>FALSE</code>. +</p> + +<p> +All APIs that are allowed to do so have their own local variable that +shadows the global variable and is set to <code>TRUE</code>. In phase 1, +no such APIs exist. +</p> + +<p> +It is defined in <code>H5.c</code> as follows: +</p> + +<blockquote> + <pre> + hbool_t H5_allow_concurrent_g = FALSE; + </pre> +</blockquote> + +<h2>2.3 Global thread initialization variable</h2> + +<p> +The global variable <code>H5_first_init_g</code> of type +<code>pthread_once_t</code> is used to allow only the first thread in the +application process to call an initialization function using +<code>pthread_once</code>. All subsequent calls to +<code>pthread_once</code> by any thread are disregarded. +</p> + +<p> +The call sets up the mutex in the global structure <code>H5_g</code> (see +section 3.1) via an initialization function +<code>H5_first_thread_init</code>. The first thread initialization +function is described in section 4.2. +</p> + +<p> +<code>H5_first_init_g</code> is defined in <code>H5.c</code> as follows: +</p> + +<blockquote> + <pre> + pthread_once_t H5_first_init_g = PTHREAD_ONCE_INIT; + </pre> +</blockquote> + +<h2>2.4 Global key for per-thread error stacks</h2> + +<p> +A global pthread-managed key <code>H5_errstk_key_g</code> is used to +allow pthreads to maintain a separate error stack (of type +<code>H5E_t</code>) for each thread. This is defined in <code>H5.c</code> +as: +</p> + +<blockquote> + <pre> + pthread_key_t H5_errstk_key_g; + </pre> +</blockquote> + +<p> +Error stack management is described in section 4.3. +</p> + +<h2>2.5 Global structure and key for thread cancellation prevention</h2> + +<p> +We need to preserve the thread cancellation status of each thread +individually by using a key <code>H5_cancel_key_g</code>. The status is +preserved using a structure (of type <code>H5_cancel_t</code>) which +maintains the cancellability state of the thread before it entered the +library and a count (which works very much like the recursive lock +counter) which keeps track of the number of API calls the thread makes +within the library. +</p> + +<p> +The structure is defined in <code>H5private.h</code> as: +</p> + +<blockquote> + <pre> + /* cancelability structure */ + typedef struct H5_cancel_struct { + int previous_state; + unsigned int cancel_count; + } H5_cancel_t; + </pre> +</blockquote> + +<p> +Thread cancellation is described in section 4.4. +</p> + + +<h1>3. Changes to Macro expansions</h1> + +<h2>3.1 Changes to FUNC_ENTER</h2> + +<p> +The <code>FUNC_ENTER</code> macro is now extended to include macro calls +to initialize first threads, disable cancellability and wraps a lock +operation around the checking of the global initialization flag. It +should be noted that the cancellability should be disabled before +acquiring the lock on the library. Doing so otherwise would allow the +possibility that the thread be cancelled just after it has acquired the +lock on the library and in that scenario, if the cleanup routines are not +properly set, the library would be permanently locked out. +</p> + +<p> +The additional macro code and new macro definitions can be found in +Appendix E.1 to E.5. The changes are made in <code>H5private.h</code>. +</p> + +<h2>3.2 Changes to HRETURN and HRETURN_ERROR</h2> + +<p> +The <code>HRETURN</code> and <code>HRETURN_ERROR</code> macros are the +counterparts to the <code>FUNC_ENTER</code> macro described in section +3.1. <code>FUNC_LEAVE</code> makes a macro call to <code>HRETURN</code>, +so it is also covered here. +</p> + +<p> +The basic changes to these two macros involve adding macro calls to call +an unlock operation and re-enable cancellability if necessary. It should +be noted that the cancellability should be re-enabled only after the +thread has released the lock to the library. The consequence of doing +otherwise would be similar to that described in section 3.1. +</p> + +<p> +The additional macro code and new macro definitions can be found in +Appendix E.9 to E.9. The changes are made in <code>H5Eprivate.h</code>. +</p> + +<h1>4. Implementation of threadsafe functionality</h1> + +<h2>4.1 Recursive Locks</h2> + +<p> +A recursive mutex lock m allows a thread t1 to successfully lock m more +than once without blocking t1. Another thread t2 will block if t2 tries +to lock m while t1 holds the lock to m. If t1 makes k lock calls on m, +then it also needs to make k unlock calls on m before it releases the +lock. +</p> + +<p> +Our implementation of recursive locks is built on top of a pthread mutex +lock (which is not recursive). It makes use of a pthread condition +variable to have unsuccessful threads wait on the mutex. Waiting threads +are awaken by a signal from the final unlock call made by the thread +holding the lock. +</p> + +<p> +Recursive locks are defined to be the following type +(<code>H5private.h</code>): +</p> + +<blockquote> + <pre> + typedef struct H5_mutex_struct { + pthread_t owner_thread; /* current lock owner */ + pthread_mutex_t atomic_lock; /* lock for atomicity of new mechanism */ + pthread_cond_t cond_var; /* condition variable */ + unsigned int lock_count; + } H5_mutex_t; + </pre> +</blockquote> + +<p> +Detailed implementation code can be found in Appendix A. The +implementation changes are made in <code>H5TS.c</code>. +</p> + +<h2>4.2 First thread initialization</h2> + +<p> +Because the mutex lock associated with a recursive lock cannot be +statically initialized, a mechanism is required to initialize the +recursive lock associated with <code>H5_g</code> so that it can be used +for the first time. +</p> + +<p> +The pthreads library allows this through the pthread_once call which as +described in section 3.3 allows only the first thread accessing the +library in an application to initialize <code>H5_g</code>. +</p> + +<p> +In addition to initializing <code>H5_g</code>, it also initializes the +key (see section 3.4) for use with per-thread error stacks (see section +4.3). +</p> + +<p> +The first thread initialization mechanism is implemented as the function +call <code>H5_first_thread_init()</code> in <code>H5TS.c</code>. This is +described in appendix B. +</p> + +<h2>4.3 Per-thread error stack management</h2> + +<p> +Pthreads allows individual threads to access dynamic and persistent +per-thread data through the use of keys. Each key is associated with +a table that maps threads to data items. Keys can be initialized by +<code>pthread_key_create()</code> in pthreads (see sections 3.4 and 4.2). +Per-thread data items are accessed using a key through the +<code>pthread_getspecific()</code> and <code>pthread_setspecific()</code> +calls to read and write to the association table respectively. +</p> + +<p> +Per-thread error stacks are accessed through the key +<code>H5_errstk_key_g</code> which is initialized by the first thread +initialization call (see section 4.2). +</p> + +<p> +In the non-threadsafe version of the library, there is a global stack +variable <code>H5E_stack_g[1]</code> which is no longer defined in the +threadsafe version. At the same time, the macro call to gain access to +the error stack <code>H5E_get_my_stack</code> is changed from: +</p> + +<blockquote> + <pre> + #define H5E_get_my_stack() (H5E_stack_g+0) + </pre> +</blockquote> + +<p> +to: +</p> + +<blockquote> + <pre> + #define H5E_get_my_stack() H5E_get_stack() + </pre> +</blockquote> + +<p> +where <code>H5E_get_stack()</code> is a surrogate function that does the +following operations: +</p> + +<ol> + <li>if a thread is attempting to get an error stack for the first + time, the error stack is dynamically allocated for the thread and + associated with <code>H5_errstk_key_g</code> using + <code>pthread_setspecific()</code>. The way we detect if it is the + first time is through <code>pthread_getspecific()</code> which + returns <code>NULL</code> if no previous value is associated with + the thread using the key.</li> + + <li>if <code>pthread_getspecific()</code> returns a non-null value, + then that is the pointer to the error stack associated with the + thread and the stack can be used as usual.</li> +</ol> + +<p> +A final change to the error reporting routines is as follows; the current +implementation reports errors to always be detected at thread 0. In the +threadsafe implementation, this is changed to report the number returned +by a call to <code>pthread_self()</code>. +</p> + +<p> +The change in code (reflected in <code>H5Eprint</code> of file +<code>H5E.c</code>) is as follows: +</p> + +<blockquote> + <pre> + #ifdef H5_HAVE_THREADSAFE + fprintf (stream, "HDF5-DIAG: Error detected in thread %d." + ,pthread_self()); + #else + fprintf (stream, "HDF5-DIAG: Error detected in thread 0."); + #endif + </pre> +</blockquote> + +<p> +Code for <code>H5E_get_stack()</code> can be found in Appendix C. All the +above changes were made in <code>H5E.c</code>. +</p> + +<h2>4.4 Thread Cancellation safety</h2> + +<p> +To prevent thread cancellations from killing a thread while it is in the +library, we maintain per-thread information about the cancellability +status of the thread before it entered the library so that we can restore +that same status when the thread leaves the library. +</p> + +<p> +By <i>enter</i> and <i>leave</i> the library, we mean the points when a +thread makes an API call from a user application and the time that API +call returns. Other API or callback function calls made from within that +API call are considered <i>within</i> the library. +</p> + +<p> +Because other API calls may be made from within the first API call, we +need to maintain a counter to determine which was the first and +correspondingly the last return. +</p> + +<p> +When a thread makes an API call, the macro <code>H5_API_SET_CANCEL</code> +calls the worker function <code>H5_cancel_count_inc()</code> which does +the following: +</p> + +<ol> + <li>if this is the first time the thread has entered the library, + a new cancellability structure needs to be assigned to it.</li> + <li>if the thread is already within the library when the API call is + made, then cancel_count is simply incremented. Otherwise, we set + the cancellability state to <code>PTHREAD_CANCEL_DISABLE</code> + while storing the previous state into the cancellability structure. + <code>cancel_count</code> is also incremented in this case.</li> +</ol> + +<p> +When a thread leaves an API call, the macro +<code>H5_API_UNSET_CANCEL</code> calls the worker function +<code>H5_cancel_count_dec()</code> which does the following: +</p> + +<ol> + <li>if <code>cancel_count</code> is greater than 1, indicating that the + thread is not yet about to leave the library, then + <code>cancel_count</code> is simply decremented.</li> + <li>otherwise, we reset the cancellability state back to its original + state before it entered the library and decrement the count (back + to zero).</li> +</ol> + +<p> +<code>H5_cancel_count_inc</code> and <code>H5_cancel_count_dec</code> are +described in Appendix D and may be found in <code>H5TS.c</code>. +</p> + +<h1>5. Test programs</h1> + +<p> +Except where stated, all tests involve 16 simultaneous threads that make +use of HDF-5 API calls without any explicit synchronization typically +required in a non-threadsafe environment. +</p> + +<h2>5.1 Data set create and write</h2> + +<p> +The test program sets up 16 threads to simultaneously create 16 +different datasets named from <i>zero</i> to <i>fifteen</i> for a single +file and then writing an integer value into that dataset equal to the +dataset's named value. +</p> + +<p> +The main thread would join with all 16 threads and attempt to match the +resulting HDF-5 file with expected results - that each dataset contains +the correct value (0 for <i>zero</i>, 1 for <i>one</i> etc ...) and all +datasets were correctly created. +</p> + +<p> +The test is implemented in the file <code>ttsafe_dcreate.c</code>. +</p> + +<h2>5.2 Test on error stack</h2> + +<p> +The error stack test is one in which 16 threads simultaneously try to +create datasets with the same name. The result, when properly serialized, +should be equivalent to 16 attempts to create the dataset with the same +name. +</p> + +<p> +The error stack implementation runs correctly if it reports 15 instances +of the dataset name conflict error and finally generates a correct HDF-5 +containing that single dataset. Each thread should report its own stack +of errors with a thread number associated with it. +</p> + +<p> +The test is implemented in the file <code>ttsafe_error.c</code>. +</p> + +<h2>5.3 Test on cancellation safety</h2> + +<p> +The main idea in thread cancellation safety is as follows; a child thread +is spawned to create and write to a dataset. Following that, it makes a +<code>H5Diterate</code> call on that dataset which activates a callback +function. +</p> + +<p> +A deliberate barrier is invoked at the callback function which waits for +both the main and child thread to arrive at that point. After that +happens, the main thread proceeds to make a thread cancel call on the +child thread while the latter sleeps for 3 seconds before proceeding to +write a new value to the dataset. +</p> + +<p> +After the iterate call, the child thread logically proceeds to wait +another 3 seconds before writing another newer value to the dataset. +</p> + +<p> +The test is correct if the main thread manages to read the second value +at the end of the test. This means that cancellation did not take place +until the end of the iteration call despite of the 3 second wait within +the iteration callback and the extra dataset write operation. +Furthermore, the cancellation should occur before the child can proceed +to write the last value into the dataset. +</p> + +<h2>5.4 Test on attribute creation</h2> + +<p> +A main thread makes 16 threaded calls to <code>H5Acreate</code> with a +generated name for each attribute. Sixteen attributes should be created +for the single dataset in random (chronological) order and receive values +depending on its generated attribute name (e.g. <i>attrib010</i> would +receive the value 10). +</p> + +<p> +After joining with all child threads, the main thread proceeds to read +each attribute by generated name to see if the value tallies. Failure is +detected if the attribute name does not exist (meaning they were never +created) or if the wrong values were read back. +</p> + +<h1>A. Recursive Lock implementation code</h1> + +<blockquote> + <pre> + void H5_mutex_init(H5_mutex_t *H5_mutex) + { + H5_mutex->owner_thread = NULL; + pthread_mutex_init(&H5_mutex->atomic_lock, NULL); + pthread_cond_init(&H5_mutex->cond_var, NULL); + H5_mutex->lock_count = 0; + } + + void H5_mutex_lock(H5_mutex_t *H5_mutex) + { + pthread_mutex_lock(&H5_mutex->atomic_lock); + + if (pthread_equal(pthread_self(), H5_mutex->owner_thread)) { + /* already owned by self - increment count */ + H5_mutex->lock_count++; + } else { + if (H5_mutex->owner_thread == NULL) { + /* no one else has locked it - set owner and grab lock */ + H5_mutex->owner_thread = pthread_self(); + H5_mutex->lock_count = 1; + } else { + /* if already locked by someone else */ + while (1) { + pthread_cond_wait(&H5_mutex->cond_var, &H5_mutex->atomic_lock); + + if (H5_mutex->owner_thread == NULL) { + H5_mutex->owner_thread = pthread_self(); + H5_mutex->lock_count = 1; + break; + } /* else do nothing and loop back to wait on condition*/ + } + } + } + + pthread_mutex_unlock(&H5_mutex->atomic_lock); + } + + void H5_mutex_unlock(H5_mutex_t *H5_mutex) + { + pthread_mutex_lock(&H5_mutex->atomic_lock); + H5_mutex->lock_count--; + + if (H5_mutex->lock_count == 0) { + H5_mutex->owner_thread = NULL; + pthread_cond_signal(&H5_mutex->cond_var); + } + pthread_mutex_unlock(&H5_mutex->atomic_lock); + } + </pre> +</blockquote> + +<h1>B. First thread initialization</h1> + +<blockquote> + <pre> + void H5_first_thread_init(void) + { + /* initialize global API mutex lock */ + H5_g.H5_libinit_g = FALSE; + H5_g.init_lock.owner_thread = NULL; + pthread_mutex_init(&H5_g.init_lock.atomic_lock, NULL); + pthread_cond_init(&H5_g.init_lock.cond_var, NULL); + H5_g.init_lock.lock_count = 0; + + /* initialize key for thread-specific error stacks */ + pthread_key_create(&H5_errstk_key_g, NULL); + + /* initialize key for thread cancellability mechanism */ + pthread_key_create(&H5_cancel_key_g, NULL); + } + </pre> +</blockquote> + + +<h1>C. Per-thread error stack acquisition</h1> + +<blockquote> + <pre> + H5E_t *H5E_get_stack(void) + { + H5E_t *estack; + + if (estack = pthread_getspecific(H5_errstk_key_g)) { + return estack; + } else { + /* no associated value with current thread - create one */ + estack = (H5E_t *)malloc(sizeof(H5E_t)); + pthread_setspecific(H5_errstk_key_g, (void *)estack); + return estack; + } + } + </pre> +</blockquote> + +<h1>D. Thread cancellation mechanisms</h1> + +<blockquote> + <pre> + void H5_cancel_count_inc(void) + { + H5_cancel_t *cancel_counter; + + if (cancel_counter = pthread_getspecific(H5_cancel_key_g)) { + /* do nothing here */ + } else { + /* + * first time thread calls library - create new counter and + * associate with key + */ + cancel_counter = (H5_cancel_t *)malloc(sizeof(H5_cancel_t)); + cancel_counter->cancel_count = 0; + pthread_setspecific(H5_cancel_key_g, (void *)cancel_counter); + } + + if (cancel_counter->cancel_count == 0) { + /* thread entering library */ + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, + &(cancel_counter->previous_state)); + } + + cancel_counter->cancel_count++; + } + + void H5_cancel_count_dec(void) + { + H5_cancel_t *cancel_counter = pthread_getspecific(H5_cancel_key_g); + + if (cancel_counter->cancel_count == 1) + pthread_setcancelstate(cancel_counter->previous_state, NULL); + + cancel_counter->cancel_count--; + } + </pre> +</blockquote> + +<h1>E. Macro expansion codes</h1> + +<h2>E.1 <code>FUNC_ENTER</code></h2> + +<blockquote> + <pre> + /* Initialize the library */ \ + H5_FIRST_THREAD_INIT \ + H5_API_UNSET_CANCEL \ + H5_API_LOCK_BEGIN \ + if (!(H5_INIT_GLOBAL)) { \ + H5_INIT_GLOBAL = TRUE; \ + if (H5_init_library() < 0) { \ + HRETURN_ERROR (H5E_FUNC, H5E_CANTINIT, err, \ + "library initialization failed"); \ + } \ + } \ + H5_API_LOCK_END \ + : + : + : + </pre> +</blockquote> + +<h2>E.2 <code>H5_FIRST_THREAD_INIT</code></h2> + +<blockquote> + <pre> + /* Macro for first thread initialization */ + #define H5_FIRST_THREAD_INIT \ + pthread_once(&H5_first_init_g, H5_first_thread_init); + </pre> +</blockquote> + + +<h2>E.3 <code>H5_API_UNSET_CANCEL</code></h2> + +<blockquote> + <pre> + #define H5_API_UNSET_CANCEL \ + if (H5_IS_API(FUNC)) { \ + H5_cancel_count_inc(); \ + } + </pre> +</blockquote> + + +<h2>E.4 <code>H5_API_LOCK_BEGIN</code></h2> + +<blockquote> + <pre> + #define H5_API_LOCK_BEGIN \ + if (H5_IS_API(FUNC)) { \ + H5_mutex_lock(&H5_g.init_lock); + </pre> +</blockquote> + + +<h2>E.5 <code>H5_API_LOCK_END</code></h2> + +<blockquote> + <pre> + #define H5_API_LOCK_END } + </pre> +</blockquote> + + +<h2>E.6 <code>HRETURN</code> and <code>HRETURN_ERROR</code></h2> + +<blockquote> + <pre> + : + : + H5_API_UNLOCK_BEGIN \ + H5_API_UNLOCK_END \ + H5_API_SET_CANCEL \ + return ret_val; \ + } + </pre> +</blockquote> + +<h2>E.7 <code>H5_API_UNLOCK_BEGIN</code></h2> + +<blockquote> + <pre> + #define H5_API_UNLOCK_BEGIN \ + if (H5_IS_API(FUNC)) { \ + H5_mutex_unlock(&H5_g.init_lock); + </pre> +</blockquote> + +<h2>E.8 <code>H5_API_UNLOCK_END</code></h2> + +<blockquote> + <pre> + #define H5_API_UNLOCK_END } + </pre> +</blockquote> + + +<h2>E.9 <code>H5_API_SET_CANCEL</code></h2> + +<blockquote> + <pre> + #define H5_API_SET_CANCEL \ + if (H5_IS_API(FUNC)) { \ + H5_cancel_count_dec(); \ + } + </pre> +</blockquote> + +<h2>By Chee Wai Lee</h2> +<h4>By Bill Wendling</h4> +<h4>27. October 2000</h4> + +</body> +</html> |