From d4a3097ec5d9e44d377c4b91a05b3e0c5f9f1e2c Mon Sep 17 00:00:00 2001 From: Gerd Heber Date: Fri, 20 Nov 2020 23:18:14 -0600 Subject: Full set of current H5F documentation. (#105) * First cut of the H5 public API documentation. * Added H5Z "bonus track." * Applied Quincey's patch. * Added the missing patches from Quincey's original patch. * H5PL (complete) and basic H5VL API documentation. * Added H5I API docs. * Added H5L API docs. * First installment from Elena's H5T batch. * Second installment of Elena's H5T batch. * Final installment of Elena's H5T batch. * Migrated documentation for SWMR functions. * Catching up on MDC functions. * Integrated the H5F MDC function documentation. * Added MDC and parallel H5F functions. * Slightly updated main page. * Added doxygen/dox/H5AC_cache_config_t.dox to MANIFEST. --- MANIFEST | 1 + doxygen/Doxyfile.in | 1 + doxygen/dox/H5AC_cache_config_t.dox | 415 +++++++++++++++++ doxygen/dox/mainpage.dox | 70 +-- src/H5Fpublic.h | 868 +++++++++++++++++++++++++++++++++++- 5 files changed, 1311 insertions(+), 44 deletions(-) create mode 100644 doxygen/dox/H5AC_cache_config_t.dox diff --git a/MANIFEST b/MANIFEST index 3acf466..39e2537 100644 --- a/MANIFEST +++ b/MANIFEST @@ -197,6 +197,7 @@ ./doxygen/aliases ./doxygen/Doxyfile.in ./doxygen/dox/api-compat-macros.dox +./doxygen/dox/H5AC_cache_config_t.dox ./doxygen/dox/H5Acreate.dox ./doxygen/dox/H5Aiterate.dox ./doxygen/dox/H5Fget_info.dox diff --git a/doxygen/Doxyfile.in b/doxygen/Doxyfile.in index 711fa7a..24fc4ba 100644 --- a/doxygen/Doxyfile.in +++ b/doxygen/Doxyfile.in @@ -854,6 +854,7 @@ INPUT_ENCODING = UTF-8 FILE_PATTERNS = H5*public.h \ H5*module.h \ + H5FDmpio.h \ H5VLconnector.h \ H5VLconnector_passthru.h \ H5VLnative.h \ diff --git a/doxygen/dox/H5AC_cache_config_t.dox b/doxygen/dox/H5AC_cache_config_t.dox new file mode 100644 index 0000000..9b9862b --- /dev/null +++ b/doxygen/dox/H5AC_cache_config_t.dox @@ -0,0 +1,415 @@ +/** + * \page H5AC-cache-config-t Metadata Cache Configuration + * \tableofcontents + * + * \section gcf General configuration fields + * + * \par version + * Integer field containing the version number of this version + * of the H5AC_cache_config_t structure. Any instance of + * H5AC_cache_config_t passed to the cache must have a known + * version number, or an error will be flagged. + * + * \par rpt_fcn_enabled + * \parblock + * Boolean field used to enable and disable the default + * reporting function. This function is invoked every time the + * automatic cache resize code is run, and reports on its activities. + * + * This is a debugging function, and should normally be turned off. + * \endparblock + * + * \par open_trace_file + * \parblock + * Boolean field indicating whether the trace_file_name + * field should be used to open a trace file for the cache. + * + * \Emph{*** DEPRECATED ***} Use \Code{H5Fstart/stop} logging functions instead + * + * The trace file is a debuging feature that allow the capture of + * top level metadata cache requests for purposes of debugging and/or + * optimization. This field should normally be set to \c FALSE, as + * trace file collection imposes considerable overhead. + * + * This field should only be set to \c TRUE when the trace_file_name + * contains the full path of the desired trace file, and either + * there is no open trace file on the cache, or the \c close_trace_file + * field is also \c TRUE. + * \endparblock + * + * \par close_trace_file + * \parblock + * Boolean field indicating whether the current trace + * file (if any) should be closed. + * + * \Emph{*** DEPRECATED ***} Use \Code{H5Fstart/stop} logging functions instead + * + * See the above comments on the open_trace_file field. This field + * should be set to \c FALSE unless there is an open trace file on the + * cache that you wish to close. + * \endparblock + * + * \par trace_file_name + * \parblock + * Full path of the trace file to be opened if the + * open_trace_file field is \c TRUE. + * + * \Emph{*** DEPRECATED ***} Use \Code{H5Fstart/stop} logging functions instead + * + * In the parallel case, an ascii representation of the mpi rank of + * the process will be appended to the file name to yield a unique + * trace file name for each process. + * + * The length of the path must not exceed #H5AC__MAX_TRACE_FILE_NAME_LEN + * characters. + * \endparblock + * + * \par evictions_enabled + * \parblock + * Boolean field used to either report the current + * evictions enabled status of the cache, or to set the cache's + * evictions enabled status. + * + * In general, the metadata cache should always be allowed to + * evict entries. However, in some cases it is advantageous to + * disable evictions briefly, and thereby postpone metadata + * writes. However, this must be done with care, as the cache + * can grow quickly. If you do this, re-enable evictions as + * soon as possible and monitor cache size. + * + * At present, evictions can only be disabled if automatic + * cache resizing is also disabled (that is, \Code{(incr_mode == + * H5C_incr__off ) && ( decr_mode == H5C_decr__off )}). There + * is no logical reason why this should be so, but it simplifies + * implementation and testing, and I can't think of any reason + * why it would be desireable. If you can think of one, I'll + * revisit the issue. (JM) + * \endparblock + * + * \par set_initial_size + * Boolean flag indicating whether the size of the + * initial size of the cache is to be set to the value given in + * the initial_size field. If set_initial_size is \c FALSE, the + * initial_size field is ignored. + * + * \par initial_size + * If enabled, this field contain the size the cache is + * to be set to upon receipt of this structure. Needless to say, + * initial_size must lie in the closed interval \Code{[min_size, max_size]}. + * + * \par min_clean_fraction + * \c double in the range 0 to 1 indicating the fraction + * of the cache that is to be kept clean. This field is only used + * in parallel mode. Typical values are 0.1 to 0.5. + * + * \par max_size + * Maximum size to which the cache can be adjusted. The + * supplied value must fall in the closed interval + * \Code{[MIN_MAX_CACHE_SIZE, MAX_MAX_CACHE_SIZE]}. Also, \c max_size must + * be greater than or equal to \c min_size. + * + * \par min_size + * Minimum size to which the cache can be adjusted. The + * supplied value must fall in the closed interval + * \Code{[H5C__MIN_MAX_CACHE_SIZE, H5C__MAX_MAX_CACHE_SIZE]}. Also, \c min_size + * must be less than or equal to \c max_size. + * + * \par epoch_length + * \parblock + * Number of accesses on the cache over which to collect + * hit rate stats before running the automatic cache resize code, + * if it is enabled. + * + * At the end of an epoch, we discard prior hit rate data and start + * collecting afresh. The epoch_length must lie in the closed + * interval \Code{[H5C__MIN_AR_EPOCH_LENGTH, H5C__MAX_AR_EPOCH_LENGTH]}. + * \endparblock + * + * + * \section csicf Cache size increase control fields + * + * \par incr_mode + * Instance of the \c H5C_cache_incr_mode enumerated type whose + * value indicates how we determine whether the cache size should be + * increased. At present there are two possible values: + * \li \c H5C_incr__off: Don't attempt to increase the size of the cache + * automatically.\n + * When this increment mode is selected, the remaining fields + * in the cache size increase section ar ignored. + * \li \c H5C_incr__threshold: Attempt to increase the size of the cache + * whenever the average hit rate over the last epoch drops + * below the value supplied in the \c lower_hr_threshold + * field.\n + * Note that this attempt will fail if the cache is already + * at its maximum size, or if the cache is not already using + * all available space. + * + * Note that you must set \c decr_mode to \c H5C_incr__off if you + * disable metadata cache entry evictions. + * + * \par lower_hr_threshold + * \parblock + * Lower hit rate threshold. If the increment mode + * (\c incr_mode) is \c H5C_incr__threshold and the hit rate drops below the + * value supplied in this field in an epoch, increment the cache size by + * \c size_increment. Note that cache size may not be incremented above + * \c max_size, and that the increment may be further restricted by the + * \c max_increment field if it is enabled. + * + * When enabled, this field must contain a value in the range [0.0, 1.0]. + * Depending on the \c incr_mode selected, it may also have to be less than + * \c upper_hr_threshold. + * \endparblock + * + * \par increment + * \parblock + * Double containing the multiplier used to derive the new + * cache size from the old if a cache size increment is triggered. + * The increment must be greater than 1.0, and should not exceed 2.0. + * + * The new cache size is obtained my multiplying the current max cache + * size by the increment, and then clamping to \c max_size and to stay + * within the \c max_increment as necessary. + * \endparblock + * + * \par apply_max_increment + * Boolean flag indicating whether the \c max_increment + * field should be used to limit the maximum cache size increment. + * + * \par max_increment + * If enabled by the \c apply_max_increment field described + * above, this field contains the maximum number of bytes by which the + * cache size can be increased in a single re-size. + * + * \par flash_incr_mode + * \parblock + * Instance of the \c H5C_cache_flash_incr_mode enumerated + * type whose value indicates whether and by which algorithm we should + * make flash increases in the size of the cache to accommodate insertion + * of large entries and large increases in the size of a single entry. + * + * The addition of the flash increment mode was occasioned by performance + * problems that appear when a local heap is increased to a size in excess + * of the current cache size. While the existing re-size code dealt with + * this eventually, performance was very bad for the remainder of the + * epoch. + * + * At present, there are two possible values for the \c flash_incr_mode: + * + * \li \c H5C_flash_incr__off: Don't perform flash increases in the size of the cache. + * + * \li \c H5C_flash_incr__add_space: Let \c x be either the size of a newly + * newly inserted entry, or the number of bytes by which the + * size of an existing entry has been increased.\n + * If \Code{x > flash_threshold * current max cache size}, + * increase the current maximum cache size by \Code{x * flash_multiple} + * less any free space in the cache, and star a new epoch. For + * now at least, pay no attention to the maximum increment. + * + * In both of the above cases, the flash increment pays no attention to + * the maximum increment (at least in this first incarnation), but DOES + * stay within max_size. + * + * With a little thought, it should be obvious that the above flash + * cache size increase algorithm is not sufficient for all circumstances + * -- for example, suppose the user round robins through + * \Code{(1/flash_threshold) +1} groups, adding one data set to each on each + * pass. Then all will increase in size at about the same time, requiring + * the max cache size to at least double to maintain acceptable + * performance, however the above flash increment algorithm will not be + * triggered. + * + * Hopefully, the add space algorithms detailed above will be sufficient + * for the performance problems encountered to date. However, we should + * expect to revisit the issue. + * \endparblock + * + * \par flash_multiple + * Double containing the multiple described above in the + * \c H5C_flash_incr__add_space section of the discussion of the + * \c flash_incr_mode section. This field is ignored unless \c flash_incr_mode + * is \c H5C_flash_incr__add_space. + * + * \par flash_threshold + * Double containing the factor by which current max cache + * size is multiplied to obtain the size threshold for the add_space flash + * increment algorithm. The field is ignored unless \c flash_incr_mode is + * \c H5C_flash_incr__add_space. + * + * + * \section csdcf Cache size decrease control fields + * + * \par decr_mode + * \parblock + * Instance of the \c H5C_cache_decr_mode enumerated type whose + * value indicates how we determine whether the cache size should be + * decreased. At present there are four possibilities. + * + * \li \c H5C_decr__off: Don't attempt to decrease the size of the cache + * automatically.\n + * When this increment mode is selected, the remaining fields + * in the cache size decrease section are ignored. + * \li \c H5C_decr__threshold: Attempt to decrease the size of the cache + * whenever the average hit rate over the last epoch rises + * above the value supplied in the \c upper_hr_threshold + * field. + * \li \c H5C_decr__age_out: At the end of each epoch, search the cache for + * entries that have not been accessed for at least the number + * of epochs specified in the epochs_before_eviction field, and + * evict these entries. Conceptually, the maximum cache size + * is then decreased to match the new actual cache size. However, + * this reduction may be modified by the \c min_size, the + * \c max_decrement, and/or the \c empty_reserve. + * \li \c H5C_decr__age_out_with_threshold: Same as age_out, but we only + * attempt to reduce the cache size when the hit rate observed + * over the last epoch exceeds the value provided in the + * \c upper_hr_threshold field. + * + * Note that you must set \c decr_mode to \c H5C_decr__off if you + * disable metadata cache entry evictions. + * \endparblock + * + * \par upper_hr_threshold + * \parblock + * Upper hit rate threshold. The use of this field + * varies according to the current \c decr_mode : + * + * \c H5C_decr__off or \c H5C_decr__age_out: The value of this field is + * ignored. + * + * \li \c H5C_decr__threshold: If the hit rate exceeds this threshold in any + * epoch, attempt to decrement the cache size by size_decrement.\n + * Note that cache size may not be decremented below \c min_size.\n + * Note also that if the \c upper_threshold is 1.0, the cache size\n + * will never be reduced. + * + * \li \c H5C_decr__age_out_with_threshold: If the hit rate exceeds this + * threshold in any epoch, attempt to reduce the cache size + * by evicting entries that have not been accessed for more + * than the specified number of epochs. + * \endparblock + * + * \par decrement + * \parblock + * This field is only used when the decr_mode is + * \c H5C_decr__threshold. + * + * The field is a double containing the multiplier used to derive the + * new cache size from the old if a cache size decrement is triggered. + * The decrement must be in the range 0.0 (in which case the cache will + * try to contract to its minimum size) to 1.0 (in which case the + * cache will never shrink). + * \endparblock + * + * \par apply_max_decrement + * Boolean flag used to determine whether decrements + * in cache size are to be limited by the \c max_decrement field. + * + * \par max_decrement + * Maximum number of bytes by which the cache size can be + * decreased in a single re-size. Note that decrements may also be + * restricted by the \c min_size of the cache, and (in age out modes) by + * the \c empty_reserve field. + * + * \par epochs_before_eviction + * \parblock + * Integer field used in \c H5C_decr__age_out and + * \c H5C_decr__age_out_with_threshold decrement modes. + * + * This field contains the number of epochs an entry must remain + * unaccessed before it is evicted in an attempt to reduce the + * cache size. If applicable, this field must lie in the range + * \Code{[1, H5C__MAX_EPOCH_MARKERS]}. + * \endparblock + * + * \par apply_empty_reserve + * Boolean field controlling whether the empty_reserve + * field is to be used in computing the new cache size when the + * decr_mode is H5C_decr__age_out or H5C_decr__age_out_with_threshold. + * + * \par empty_reserve + * \parblock + * To avoid a constant racheting down of cache size by small + * amounts in the \c H5C_decr__age_out and \c H5C_decr__age_out_with_threshold + * modes, this field allows one to require that any cache size + * reductions leave the specified fraction of unused space in the cache. + * + * The value of this field must be in the range [0.0, 1.0]. I would + * expect typical values to be in the range of 0.01 to 0.1. + * \endparblock + * + * + * \section pcf Parallel Configuration Fields + * + * In PHDF5, all operations that modify metadata must be executed collectively. + * + * We used to think that this was enough to ensure consistency across the + * metadata caches, but since we allow processes to read metadata individually, + * the order of dirty entries in the LRU list can vary across processes, + * which can result in inconsistencies between the caches. + * + * PHDF5 uses several strategies to prevent such inconsistencies in metadata, + * all of which use the fact that the same stream of dirty metadata is seen + * by all processes for purposes of synchronization. This is done by + * having each process count the number of bytes of dirty metadata generated, + * and then running a "sync point" whenever this count exceeds a user + * specified threshold (see \c dirty_bytes_threshold below). + * + * The current metadata write strategy is indicated by the + * \c metadata_write_strategy field. The possible values of this field, along + * with the associated metadata write strategies are discussed below. + * + * \par dirty_bytes_threshold + * \parblock + * Threshold of dirty byte creation used to + * synchronize updates between caches. (See above for outline and + * motivation.) + * + * This value MUST be consistent across all processes accessing the + * file. This field is ignored unless HDF5 has been compiled for + * parallel. + * \endparblock + * + * \par metadata_write_strategy + * Integer field containing a code indicating the + * desired metadata write strategy. The valid values of this field + * are enumerated and discussed below: + * + * \li #H5AC_METADATA_WRITE_STRATEGY__PROCESS_0_ONLY\n + * When metadata_write_strategy is set to this value, only process + * zero is allowed to write dirty metadata to disk. All other + * processes must retain dirty metadata until they are informed at + * a sync point that the dirty metadata in question has been written + * to disk.\n + * When the sync point is reached (or when there is a user generated + * flush), process zero flushes sufficient entries to bring it into + * complience with its min clean size (or flushes all dirty entries in + * the case of a user generated flush), broad casts the list of + * entries just cleaned to all the other processes, and then exits + * the sync point.\n + * Upon receipt of the broadcast, the other processes mark the indicated + * entries as clean, and leave the sync point as well. + * + * \li #H5AC_METADATA_WRITE_STRATEGY__DISTRIBUTED\n + * In the distributed metadata write strategy, process zero still makes + * the decisions as to what entries should be flushed, but the actual + * flushes are distributed across the processes in the computation to + * the extent possible.\n + * In this strategy, when a sync point is triggered (either by dirty + * metadata creation or manual flush), all processes enter a barrier.\n + * On the other side of the barrier, process 0 constructs an ordered + * list of the entries to be flushed, and then broadcasts this list + * to the caches in all the processes.\n + * All processes then scan the list of entries to be flushed, flushing + * some, and marking the rest as clean. The algorithm for this purpose + * ensures that each entry in the list is flushed exactly once, and + * all are marked clean in each cache.\n + * Note that in the case of a flush of the cache, no message passing + * is necessary, as all processes have the same list of dirty entries, + * and all of these entries must be flushed. Thus in this case it is + * sufficient for each process to sort its list of dirty entries after + * leaving the initial barrier, and use this list as if it had been + * received from process zero.\n + * To avoid possible messages from the past/future, all caches must + * wait until all caches are done before leaving the sync point. + */ \ No newline at end of file diff --git a/doxygen/dox/mainpage.dox b/doxygen/dox/mainpage.dox index 83fc323..eda967b 100644 --- a/doxygen/dox/mainpage.dox +++ b/doxygen/dox/mainpage.dox @@ -1,36 +1,44 @@ -/*! \mainpage API Documentation for HDF5 Version 1.13 (Draft) +/*! \mainpage HDF5 C-API Reference + * + * The HDF5 C-API provides applications with fine-grained control over all + * aspects HDF5 functionality. This functionality is grouped into the following + * \Emph{modules}: + * \li \ref H5A "Attributes" — Management of HDF5 attributes (\ref H5A) + * \li \ref H5D "Datasets" — Management of HDF5 datasets (\ref H5D) + * \li \ref H5S "Dataspaces" — Management of HDF5 dataspaces which describe the shape of datasets and attributes (\ref H5S) + * \li \ref H5T "Datatypes" — Management of datatypes which describe elements of datasets and attributes (\ref H5T) + * \li \ref H5E "Error Handling" — Functions for handling errors that occur within HDF5 (\ref H5E) + * \li \ref H5F "Files" — Management of HDF5 files (\ref H5F) + * \li \ref H5Z "Filters" — Configuration of filters that process data during I/O operation (\ref H5Z) + * \li \ref H5G "Groups" — Management of groups in HDF5 files (\ref H5G) + * \li \ref H5I "Identifiers" — Management of object identifiers and object names (\ref H5I) + * \li \ref H5 "Library" — General purpose library functions (\ref H5) + * \li \ref H5L "Links" — Management of links in HDF5 groups (\ref H5L) + * \li \ref H5O "Objects" — Management of objects in HDF5 files (\ref H5O) + * \li \ref H5PL "Plugins" — Programmatic control over dynamically loaded plugins (\ref H5PL) + * \li \ref H5P "Property Lists" — Management of property lists to control HDF5 library behavior (\ref H5P) + * \li \ref H5R "References" — Management of references to specific objects and data regions in an HDF5 file (\ref H5R) + * \li \ref H5VL "Virtual Object Layer" — Management of the Virtual Object Layer (\ref H5VL) + * + * Here are a few simple rules to follow: + * + * \li \Bold{Handle discipline:} If you acquire a handle (by creation or coopy), \Emph{you own it!} (..., i.e., you have to close it.) + * \li \Bold{Dynamic memory allocation:} ... + * \li \Bold{Use of locations:} Identifier + name combo + * + * \attention \Bold{C++ Developers using HDF5 C-API functions beware:}\n + * If a C routine that takes a function pointer as an argument is called from + * within C++ code, the C routine should be returned from normally. + * Examples of this kind of routine include callbacks such as H5Pset_elink_cb() + * and H5Pset_type_conv_cb() and functions such as H5Tconvert() and H5Ewalk2().\n + * Exiting the routine in its normal fashion allows the HDF5 C library to clean + * up its work properly. In other words, if the C++ application jumps out of + * the routine back to the C++ \c catch statement, the library is not given the + * opportunity to close any temporary data structures that were set up when the + * routine was called. The C++ application should save some state as the + * routine is started so that any problem that occurs might be diagnosed. * * \todo Fix the search form for server deployments. * \todo Make it mobile-friendly * - * \section intro_sec Introduction - * - * \todo Write an introduction. - * - * \section quick_links Quick Links - * - * - * - * \section using_locations The Use of Locations (Identifier + Name) in the HDF5 API - * - * \todo Make this crystal clear! - * - * \section cpp_note Programming Note for C++ Developers Using C Functions - * - * If a C routine that takes a function pointer as an argument is called from - * within C++ code, the C routine should be returned from normally. - * - * Examples of this kind of routine include callbacks such as H5Pset_elink_cb() - * and H5Pset_type_conv_cb() and functions such as H5Tconvert() and H5Ewalk2(). - * - * Exiting the routine in its normal fashion allows the HDF5 C library to clean - * up its work properly. In other words, if the C++ application jumps out of - * the routine back to the C++ \c catch statement, the library is not given the - * opportunity to close any temporary data structures that were set up when the - * routine was called. The C++ application should save some state as the - * routine is started so that any problem that occurs might be diagnosed. */ \ No newline at end of file diff --git a/src/H5Fpublic.h b/src/H5Fpublic.h index be8eced..82895a8 100644 --- a/src/H5Fpublic.h +++ b/src/H5Fpublic.h @@ -226,6 +226,7 @@ typedef enum H5F_file_space_type_t { H5F_FILE_SPACE_NTYPES /**< Sentinel */ } H5F_file_space_type_t; +//! [H5F_retry_info_t_snip] #define H5F_NUM_METADATA_READ_RETRY_TYPES 21 /** @@ -236,6 +237,7 @@ typedef struct H5F_retry_info_t { unsigned nbins; uint32_t *retries[H5F_NUM_METADATA_READ_RETRY_TYPES]; } H5F_retry_info_t; +//! [H5F_retry_info_t_snip] /** * Callback for H5Pset_object_flush_cb() in a file access property list @@ -953,32 +955,542 @@ H5_DLL ssize_t H5Fget_file_image(hid_t file_id, void *buf_ptr, size_t buf_len); /** * \ingroup MDC * - * \todo Finish this! + * \brief Obtains current metadata cache configuration for target file + * + * \file_id + * \param[in,out] config_ptr Pointer to the H5AC_cache_config_t instance in which + * the current metadata cache configuration is to be + * reported. The fields of this structure are discussed + * \ref H5AC-cache-config-t "here". + * \return \herr_t + * + * \details H5Fget_mdc_config() loads the current metadata cache configuration + * into the instance of H5AC_cache_config_t pointed to by the \p config_ptr + * parameter. + * + * Note that the \c version field of \p config_ptr must be initialized + * --this allows the library to support old versions of the H5AC_cache_config_t + * structure. + * + * \par General configuration section + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
int version IN: Integer field indicating the the version of the H5AC_cache_config_t in use. This field should + * be set to #H5AC__CURR_CACHE_CONFIG_VERSION (defined in H5ACpublic.h).
hbool_t rpt_fcn_enabled

OUT: Boolean flag indicating whether the adaptive cache resize report function is enabled. This + * field should almost always be set to disabled (0). Since resize algorithm activity is + * reported via stdout, it MUST be set to disabled (0) on Windows machines.

The + * report function is not supported code, and can be expected to change between versions of the + * library. Use it at your own risk.

hbool_t open_trace_file OUT: Boolean field indicating whether the trace_file_name field should be used to open + * a trace file for the cache. This field will always be set to 0 in this context.
hbool_t close_trace_file OUT: Boolean field indicating whether the current trace file (if any) should be closed. This field + * will always be set to 0 in this context.
char*trace_file_name OUT: Full path name of the trace file to be opened if the open_trace_file field is set + * to 1. This field will always be set to the empty string in this context.
hbool_t evictions_enabled OUT: Boolean flag indicating whether metadata cache entry evictions are + * enabled.
hbool_t set_initial_size OUT: Boolean flag indicating whether the cache should be created with a user specified initial + * maximum size.

If the configuration is loaded from the cache, this flag will always be set + * to 0.

size_t initial_size OUT: Initial maximum size of the cache in bytes, if applicable.

If the configuration is loaded + * from the cache, this field will contain the cache maximum size as of the time of the + * call.

double min_clean_fraction OUT: Float value specifying the minimum fraction of the cache that must be kept either clean or + * empty when possible.
size_t max_size OUT: Upper bound (in bytes) on the range of values that the adaptive cache resize code can select + * as the maximum cache size.
size_t min_size OUT: Lower bound (in bytes) on the range of values that the adaptive cache resize code can select + * as the maximum cache size.
long int epoch_length OUT: Number of cache accesses between runs of the adaptive cache resize + * code.
+ * + * \par Increment configuration section + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
enum H5C_cache_incr_mode incr_mode OUT: Enumerated value indicating the operational mode of the automatic cache size increase code. At + * present, only the following values are legal:

\c H5C_incr__off: Automatic cache size increase is + * disabled.

\c H5C_incr__threshold: Automatic cache size increase is enabled using the hit rate + * threshold algorithm.

double lower_hr_threshold OUT: Hit rate threshold used in the hit rate threshold cache size increase algorithm.
double increment OUT: The factor by which the current maximum cache size is multiplied to obtain an initial new + * maximum cache size if a size increase is triggered in the hit rate threshold cache size increase + * algorithm.
hbool_t apply_max_increment OUT: Boolean flag indicating whether an upper limit will be applied to the size of cache size + * increases.
size_t max_increment OUT: The maximum number of bytes by which the maximum cache size can be increased in a single step + * -- if applicable.
enum H5C_cache_flash_incr_mode flash_incr_mode OUT: Enumerated value indicating the operational mode of the flash cache size increase code. At + * present, only the following values are legal:

\c H5C_flash_incr__off: Flash cache size increase is + * disabled.

\c H5C_flash_incr__add_space: Flash cache size increase is enabled using the add space + * algorithm.

double flash_threshold OUT: The factor by which the current maximum cache size is multiplied to obtain the minimum size + * entry / entry size increase which may trigger a flash cache size + * increase.
double flash_multiple OUT: The factor by which the size of the triggering entry / entry size increase is multiplied to + * obtain the initial cache size increment. This increment may be reduced to reflect existing free space + * in the cache and the max_size field above.
+ * + * \par Decrement configuration section + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
Decrement configuration + * section:
enum H5C_cache_decr_mode decr_mode OUT: Enumerated value indicating the operational mode of the automatic cache size decrease code. At + * present, the following values are legal:

H5C_decr__off: Automatic cache size decrease is disabled, + * and the remaining decrement fields are ignored.

H5C_decr__threshold: Automatic cache size + * decrease is enabled using the hit rate threshold algorithm.

H5C_decr__age_out: Automatic cache + * size decrease is enabled using the ageout algorithm.

H5C_decr__age_out_with_threshold: + * Automatic cache size decrease is enabled using the ageout with hit rate threshold + * algorithm

double upper_hr_threshold OUT: Upper hit rate threshold. This value is only used if the decr_mode is either + * H5C_decr__threshold or H5C_decr__age_out_with_threshold.
double decrement OUT: Factor by which the current max cache size is multiplied to obtain an initial value for the + * new cache size when cache size reduction is triggered in the hit rate threshold cache size reduction + * algorithm.
hbool_t apply_max_decrement OUT: Boolean flag indicating whether an upper limit should be applied to the size of cache size + * decreases.
size_t max_decrement OUT: The maximum number of bytes by which cache size can be decreased if any single step, if + * applicable.
int epochs_before_eviction OUT: The minimum number of epochs that an entry must reside unaccessed in cache before being + * evicted under either of the ageout cache size reduction algorithms.
hbool_t apply_empty_reserve OUT: Boolean flag indicating whether an empty reserve should be maintained under either of the + * ageout cache size reduction algorithms.
double empty_reserve OUT: Empty reserve for use with the ageout cache size reduction algorithms, if applicable.
+ * + * \par Parallel configuration section + * + * + * + * + *
int dirty_bytes_threshold OUT: Threshold number of bytes of dirty metadata generation for triggering synchronizations of the + * metadata caches serving the target file in the parallel case.

Synchronization occurs whenever the + * number of bytes of dirty metadata created since the last synchronization exceeds this limit.

+ * + * \since 1.8.0 + * + * \todo Fix the reference! + * */ H5_DLL herr_t H5Fget_mdc_config(hid_t file_id, H5AC_cache_config_t *config_ptr); /** * \ingroup MDC * - * \todo Finish this! + * \brief Attempts to configure metadata cache of target file + * + * \file_id + * \param[in,out] config_ptr Pointer to the H5AC_cache_config_t instance + * containing the desired configuration. + * The fields of this structure are discussed + * \ref H5AC-cache-config-t "here". + * \return \herr_t + * + * \details H5Fset_mdc_config() attempts to configure the file's metadata cache + * according configuration supplied in \p config_ptr. + * + * \par General configuration fields + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
int versionIN: Integer field indicating the the version of the H5AC_cache_config_t in use. This + * field should be set to #H5AC__CURR_CACHE_CONFIG_VERSION (defined + * in H5ACpublic.h).
hbool_t rpt_fcn_enabledIN: Boolean flag indicating whether the adaptive cache resize report function is enabled. This + * field should almost always be set to disabled (0). Since resize algorithm activity is reported + * via stdout, it MUST be set to disabled (0) on Windows machines.

The report function is not + * supported code, and can be expected to change between versions of the library. Use it at your own + * risk.

hbool_t open_trace_FileIN: Boolean field indicating whether the trace_file_name field should be used to open + * a trace file for the cache.

The trace file is a debuging feature that allows the capture of top level + * metadata cache requests for purposes of debugging and/or optimization. This field should normally be set + * to 0, as trace file collection imposes considerable overhead.

This field should only be + * set to 1 when the trace_file_name contains the full path of the desired trace + * file, and either there is no open trace file on the cache, or the close_trace_file field is + * also 1.

The trace file feature is unsupported unless used at the direction of The HDF + * Group. It is intended to allow The HDF Group to collect a trace of cache activity in cases of occult + * failures and/or poor performance seen in the field, so as to aid in reproduction in the lab. If you use it + * absent the direction of The HDF Group, you are on your + * own.

hbool_t close_trace_fileIN: Boolean field indicating whether the current trace file (if any) should be closed.

See the + * above comments on the open_trace_file field. This field should be set to 0 unless + * there is an open trace file on the cache that you wish to close.

The trace file feature is + * unsupported unless used at the direction of The HDF Group. It is intended to allow The HDF Group to collect + * a trace of cache activity in cases of occult failures and/or poor performance seen in the field, so as to + * aid in reproduction in the lab. If you use it absent the direction of The HDF Group, you are on your + * own.

char trace_file_name[]IN: Full path of the trace file to be opened if the open_trace_file field is set + * to 1.

In the parallel case, an ascii representation of the mpi rank of the process will be + * appended to the file name to yield a unique trace file name for each process.

The length of the path + * must not exceed #H5AC__MAX_TRACE_FILE_NAME_LEN characters.

The trace file feature is + * unsupported unless used at the direction of The HDF Group. It is intended to allow The HDF Group to collect + * a trace of cache activity in cases of occult failures and/or poor performance seen in the field, so as to + * aid in reproduction in the lab. If you use it absent the direction of The HDF Group, you are on your + * own.

hbool_t evictions_enabledIN: A boolean flag indicating whether evictions from the metadata cache are enabled. This flag is + * initially set to enabled (1).

In rare circumstances, the raw data throughput requirements + * may be so high that the user wishes to postpone metadata writes so as to reserve I/O throughput for raw + * data. The evictions_enabled field exists to allow this. However, this is an extreme step, and + * you have no business doing it unless you have read the User Guide section on metadata caching, and have + * considered all other options carefully.

The evictions_enabled field may not be set to + * disabled (0) unless all adaptive cache resizing code is disabled via + * the incr_mode, flash_incr_mode, and decr_mode fields.

When + * this flag is set to disabled (0), the metadata cache will not attempt to evict entries to make + * space for new entries, and thus will grow without bound.

Evictions will be re-enabled when this field + * is set back to 1. This should be done as soon as + * possible.

hbool_t set_initial_sizeIN: Boolean flag indicating whether the cache should be forced to the user specified initial + * size.
size_t initial_sizeIN: If set_initial_size is set to 1, then initial_size must + * contain the desired initial size in bytes. This value must lie in the closed interval [min_size, + * max_size]. (see below)
double min_clean_fractionIN: This field specifies the minimum fraction of the cache that must be kept either clean or + * empty.

The value must lie in the interval [0.0, 1.0]. 0.01 is a good place to start in the serial case. + * In the parallel case, a larger value is needed -- + * see Metadata Caching in HDF5 in the collection + * "Advanced Topics in HDF5."

size_t max_sizeIN: Upper bound (in bytes) on the range of values that the adaptive cache resize code can select as + * the maximum cache size.
size_t min_sizeIN: Lower bound (in bytes) on the range of values that the adaptive cache resize code can select as + * the maximum cache size.
long int epoch_lengthIN: Number of cache accesses between runs of the adaptive cache resize code. 50,000 is a good + * starting number.
+ * + * \par Increment configuration fields + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
enum H5C_cache_incr_mode incr_modeIN: Enumerated value indicating the operational mode of the automatic cache size increase code. At + * present, only two values are legal:

\c H5C_incr__off: Automatic cache size increase is disabled, and the + * remaining increment fields are ignored.

\c H5C_incr__threshold: Automatic cache size increase is enabled + * using the hit rate threshold + * algorithm.

double lower_hr_thresholdIN: Hit rate threshold used by the hit rate threshold cache size increment algorithm.

When the + * hit rate over an epoch is below this threshold and the cache is full, the maximum size of the cache is + * multiplied by increment (below), and then clipped as necessary to stay within max_size, and possibly + * max_increment.

This field must lie in the interval [0.0, 1.0]. 0.8 or 0.9 is a good starting + * point.

double incrementIN: Factor by which the hit rate threshold cache size increment algorithm multiplies the current + * maximum cache size to obtain a tentative new cache size.

The actual cache size increase will be clipped + * to satisfy the max_size specified in the general configuration, and possibly max_increment below.

The + * parameter must be greater than or equal to 1.0 -- 2.0 is a reasonable value.

If you set it to 1.0, + * you will effectively disable cache size + * increases.

hbool_t apply_max_incrementIN: Boolean flag indicating whether an upper limit should be applied to the size of cache size + * increases.
size_t max_incrementIN: Maximum number of bytes by which cache size can be increased in a single step -- if + * applicable.
enum H5C_cache_flash_incr_mode flash_incr_modeIN: Enumerated value indicating the operational mode of the flash cache size increase code. At + * present, only the following values are legal:

\c H5C_flash_incr__off: Flash cache size increase is + * disabled.

\c H5C_flash_incr__add_space: Flash cache size increase is enabled using the add space + * algorithm.

double flash_thresholdIN: The factor by which the current maximum cache size is multiplied to obtain the minimum size + * entry / entry size increase which may trigger a flash cache size increase.

At present, this value must + * lie in the range [0.1, 1.0].

double flash_multipleIN: The factor by which the size of the triggering entry / entry size increase is multiplied to + * obtain the initial cache size increment. This increment may be reduced to reflect existing free space in + * the cache and the max_size field above.

At present, this field must lie in the range [0.1, + * 10.0].

+ * + * \par Decrement configuration fields + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
enum H5C_cache_decr_mode decr_modeIN: Enumerated value indicating the operational mode of the automatic cache size decrease code. At + * present, the following values are legal:

\c H5C_decr__off: Automatic cache size decrease is + * disabled.

\c H5C_decr__threshold: Automatic cache size decrease is enabled using the hit rate threshold + * algorithm.

\c H5C_decr__age_out: Automatic cache size decrease is enabled using the ageout + * algorithm.

\c H5C_decr__age_out_with_threshold: Automatic cache size decrease is enabled using the + * ageout with hit rate threshold + * algorithm

double upper_hr_thresholdIN: Hit rate threshold for the hit rate threshold and ageout with hit rate threshold cache size + * decrement algorithms.

When \c decr_mode is \c H5C_decr__threshold, and the hit rate over a given epoch exceeds + * the supplied threshold, the current maximum cache size is multiplied by decrement to obtain a tentative new + * (and smaller) maximum cache size.

When \c decr_mode is \c H5C_decr__age_out_with_threshold, there is no + * attempt to find and evict aged out entries unless the hit rate in the previous epoch exceeded the supplied + * threshold.

This field must lie in the interval [0.0, 1.0].

For \c H5C_incr__threshold, .9995 or + * .99995 is a good place to start.

For \c H5C_decr__age_out_with_threshold, .999 might be more + * useful.

double decrementIN: In the hit rate threshold cache size decrease algorithm, this parameter contains the factor by + * which the current max cache size is multiplied to produce a tentative new cache size.

The actual cache + * size decrease will be clipped to satisfy the min_size specified in the general configuration, and possibly + * max_decrement below.

The parameter must be be in the interval [0.0, 1.0].

If you set it to 1.0, + * you will effectively disable cache size decreases. 0.9 is a reasonable starting + * point.

hbool_t apply_max_decrementIN: Boolean flag indicating whether an upper limit should be applied to the size of cache size + * decreases.
size_t max_decrementIN: Maximum number of bytes by which the maximum cache size can be decreased in any single step -- + * if applicable.
int epochs_before_evictionIN: In the ageout based cache size reduction algorithms, this field contains the minimum number of + * epochs an entry must remain unaccessed in cache before the cache size reduction algorithm tries to evict + * it. 3 is a reasonable value.
hbool_t apply_empty_reserveIN: Boolean flag indicating whether the ageout based decrement algorithms will maintain a empty + * reserve when decreasing cache size.
double empty_reserveIN: Empty reserve as a fraction of maximum cache size if applicable.

When so directed, the ageout + * based algorithms will not decrease the maximum cache size unless the empty reserve can be met.

The + * parameter must lie in the interval [0.0, 1.0]. 0.1 or 0.05 is a good place to + * start.

+ * + * \par Parallel configuration fields + * + * + * + * + * + *
int dirty_bytes_thresholdIN: Threshold number of bytes of dirty metadata generation for triggering synchronizations of the + * metadata caches serving the target file in the parallel case.

Synchronization occurs whenever the number + * of bytes of dirty metadata created since the last synchronization exceeds this limit.

This field only + * applies to the parallel case. While it is ignored elsewhere, it can still draw a value out of bounds + * error.

It must be consistant across all caches on any given file.

By default, this field is set + * to 256 KB. It shouldn't be more than half the current maximum cache size times the minimum clean + * fraction.

+ * + * \since 1.8.0 + * + * \todo Fix the MDC document reference! */ H5_DLL herr_t H5Fset_mdc_config(hid_t file_id, H5AC_cache_config_t *config_ptr); /** * \ingroup MDC * - * \todo Finish this! + * \brief Obtains target file's metadata cache hit rate + * + * \file_id + * \param[out] hit_rate_ptr Pointer to the double in which the hit rate is returned. Note that + * \p hit_rate_ptr is undefined if the API call fails + * \return \herr_t + * + * \details H5Fget_mdc_hit_rate() queries the metadata cache of the target file to obtain its hit rate + * \Code{(cache hits / (cache hits + cache misses))} since the last time hit rate statistics + * were reset. If the cache has not been accessed since the last time the hit rate stats were + * reset, the hit rate is defined to be 0.0. + * + * The hit rate stats can be reset either manually (via H5Freset_mdc_hit_rate_stats()), or + * automatically. If the cache's adaptive resize code is enabled, the hit rate stats will be + * reset once per epoch. If they are reset manually as well, the cache may behave oddly. + * + * See the overview of the metadata cache in the special topics section of the user manual for + * details on the metadata cache and its adaptive resize algorithms. + * */ H5_DLL herr_t H5Fget_mdc_hit_rate(hid_t file_id, double *hit_rate_ptr); /** * \ingroup MDC * - * \todo Finish this! + * \brief Obtains current metadata cache size data for specified file + * + * \file_id + * \param[out] max_size_ptr Pointer to the location in which the current cache maximum size is to be + * returned, or NULL if this datum is not desired + * \param[out] min_clean_size_ptr Pointer to the location in which the current cache minimum clean + * size is to be returned, or NULL if that datum is not desired + * \param[out] cur_size_ptr Pointer to the location in which the current cache size is to be returned, + * or NULL if that datum is not desired + * \param[out] cur_num_entries_ptr Pointer to the location in which the current number of entries in + * the cache is to be returned, or NULL if that datum is not desired + * \returns \herr_t + * + * \details H5Fget_mdc_size() queries the metadata cache of the target file for the desired size + * information, and returns this information in the locations indicated by the pointer + * parameters. If any pointer parameter is NULL, the associated data is not returned. + * + * If the API call fails, the values returned via the pointer parameters are undefined. + * + * If adaptive cache resizing is enabled, the cache maximum size and minimum clean size + * may change at the end of each epoch. Current size and current number of entries can + * change on each cache access. + * + * Current size can exceed maximum size under certain conditions. See the overview of the + * metadata cache in the special topics section of the user manual for a discussion of this. + * */ H5_DLL herr_t H5Fget_mdc_size(hid_t file_id, size_t *max_size_ptr, size_t *min_clean_size_ptr, size_t *cur_size_ptr, int *cur_num_entries_ptr); /** * \ingroup MDC * - * \todo Finish this! + * \brief Resets hit rate statistics counters for the target file + * + * \file_id + * \returns \herr_t + * + * \details + * \parblock + * H5Freset_mdc_hit_rate_stats() resets the hit rate statistics counters in the metadata cache + * associated with the specified file. + * + * If the adaptive cache resizing code is enabled, the hit rate statistics are reset at the beginning + * of each epoch. This API call allows you to do the same thing from your program. + * + * The adaptive cache resizing code may behave oddly if you use this call when adaptive cache resizing + * is enabled. However, the call should be useful if you choose to control metadata cache size from your + * program. + * + * See "Metadata Caching in HDF5" for details about the metadata cache and the adaptive cache resizing + * algorithms. If you have not read, understood, and thought about the material covered in that + * documentation, + * you should not be using this API call. + * \endparblock + * + * \todo Fix the MDC document reference! */ H5_DLL herr_t H5Freset_mdc_hit_rate_stats(hid_t file_id); /** @@ -1068,13 +1580,118 @@ H5_DLL herr_t H5Fget_info2(hid_t obj_id, H5F_info2_t *file_info); /** * \ingroup SWMR * - * \todo Finish this! + * \brief Retrieves the collection of read retries for metadata entries with checksum + * + * \file_id + * \param[out] info Struct containing the collection of read retries for metadata + * entries with checksum + * \return \herr_t\n + * + * \details \Bold{Failure Modes:} + * \li When the input identifier is not a file identifier. + * \li When the pointer to the output structure is NULL. + * \li When the memory allocation for \p retries failed. + * + * \details H5Fget_metadata_read_retry_info() retrieves information regarding the number + * of read retries for metadata entries with checksum for the file \p file_id. + * This information is reported in the H5F_retry_info_t struct defined in + * H5Fpublic.h as follows: + * \snippet this H5F_retry_info_t_snip + * \c nbins is the number of bins for each \c retries[i] of metadata entry \c i. + * It is calculated based on the current number of read attempts used in the + * library and logarithmic base 10. + * + * If read retries are incurred for a metadata entry \c i, the library will + * allocate memory for \Code{retries[i] (nbins * sizeof(uint32_t)} and store + * the collection of retries there. If there are no retries for a metadata entry + * \c i, \Code{retries[i]} will be NULL. After a call to this routine, users should + * free each \Code{retries[i]} that is non-NULL, otherwise resource leak will occur. + * + * For the library default read attempts of 100 for SWMR access, nbins will be 2 + * as depicted below: + * \li \Code{retries[i][0]} is the number of 1 to 9 read retries. + * \li \Code{retries[i][1]} is the number of 10 to 99 read retries. + * For the library default read attempts of 1 for non-SWMR access, \c nbins will + * be 0 and each \Code{retries[i]} will be NULL. + * + * The following table lists the 21 metadata entries of \Code{retries[]}: + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
Index for \Code{retries[]}Metadata entries*
0Object header (version 2)
1Object header chunk (version 2)
2B-tree header (version 2)
3B-tree internal node (version 2)
4B-tree leaf node (version 2)
5Fractal heap header
6Fractal heap direct block (optional checksum)
7Fractal heap indirect block
8Free-space header
9Free-space sections
10Shared object header message table
11Shared message record list
12Extensive array header
13Extensive array index block
14Extensive array super block
15Extensive array data block
16Extensive array data block page
17Fixed array super block
18Fixed array data block
19Fixed array data block page
20File's superblock (version 2)
* All entries are of version 0 (zero) unless indicated otherwise.
+ * + * \note On a system that is not atomic, the library might possibly read inconsistent + * metadata with checksum when performing single-writer/multiple-reader (SWMR) + * operations for an HDF5 file. Upon encountering such situations, the library + * will try reading the metadata again for a set number of times to attempt to + * obtain consistent data. The maximum number of read attempts used by the library + * will be either the value set via H5Pset_metadata_read_attempts() or the library + * default value when a value is not set.\n + * When the current number of metadata read attempts used in the library is unable + * to remedy the reading of inconsistent metadata on a system, the user can assess + * the information obtained via this routine to derive a different maximum value. + * The information can also be helpful for debugging purposes to identify potential + * issues with metadata flush dependencies and SWMR implementation in general. + * + * \since 1.10.0 + * */ H5_DLL herr_t H5Fget_metadata_read_retry_info(hid_t file_id, H5F_retry_info_t *info); /** * \ingroup SWMR * - * \todo Finish this! + * \brief Retrieves free-space section information for a file + * + * \file_id + * + * \return \herr_t + * + * \details H5Fstart_swmr_write() will activate SWMR writing mode for a file + * associated with \p file_id. This routine will prepare and ensure + * the file is safe for SWMR writing as follows: + * \li Check that the file is opened with write access (#H5F_ACC_RDWR). + * \li Check that the file is opened with the latest library format to + * ensure data structures with check-summed metadata are used. + * \li Check that the file is not already marked in SWMR writing mode. + * \li Enable reading retries for check-summed metadata to remedy + * possible checksum failures from reading inconsistent metadata + * on a system that is not atomic. + * \li Turn off usage of the library's accumulator to avoid possible + * ordering problem on a system that is not atomic. + * \li Perform a flush of the file’s data buffers and metadata to set + * a consistent state for starting SWMR write operations. + * + * Library objects are groups, datasets, and committed datatypes. For + * the current implementation, groups and datasets can remain open when + * activating SWMR writing mode, but not committed datatypes. Attributes + * attached to objects cannot remain open either. + * + * \since 1.10.0 + * */ H5_DLL herr_t H5Fstart_swmr_write(hid_t file_id); /** @@ -1175,19 +1792,141 @@ H5_DLL herr_t H5Fset_libver_bounds(hid_t file_id, H5F_libver_t low, H5F_libver /** * \ingroup MDC * - * \todo Finish this! + * \brief Starts logging metadata cache events if logging was previously enabled + * + * \file_id + * + * \return \herr_t + * + * \details The metadata cache is a central part of the HDF5 library through + * which all \Emph{file metadata} reads and writes take place. File + * metadata is normally invisible to the user and is used by the + * library for purposes such as locating and indexing data. File + * metadata should not be confused with user metadata, which consists + * of attributes created by users and attached to HDF5 objects such + * as datasets via H5A API calls. + * + * Due to the complexity of the cache, a trace/logging feature has been + * created that can be used by HDF5 developers for debugging and performance + * analysis. The functions that control this functionality will normally be + * of use to a very limited number of developers outside of The HDF Group. + * The functions have been documented to help users create logs that can + * be sent with bug reports. + * + * Control of the log functionality is straightforward. Logging is enabled + * via the H5Pset_mdc_log_options() function, which will modify the file + * access property list used to open or create a file. This function has + * a flag that determines whether logging begins at file open or starts + * in a paused state. Log messages can then be controlled via the + * H5Fstart_mdc_logging() and H5Fstop_mdc_logging() functions. + * H5Pget_mdc_log_options() can be used to examine a file access property + * list, and H5Fget_mdc_logging_status() will return the current state of + * the logging flags. + * + * The log format is described in the \Emph{Metadata Cache Logging} document. + * + * \note Logging can only be started or stopped if metadata cache logging was enabled + * via H5Pset_mdc_log_options().\n + * When enabled and currently logging, the overhead of the logging feature will + * almost certainly be significant.\n + * The log file is opened when the HDF5 file is opened or created and not when + * this function is called for the first time.\n + * This function opens the log file and starts logging metadata cache operations + * for a particular file. Calling this function when logging has already been + * enabled will be considered an error. + * + * \since 1.10.0 + * + * \todo Fix the document reference! + * */ H5_DLL herr_t H5Fstart_mdc_logging(hid_t file_id); /** * \ingroup MDC * - * \todo Finish this! + * \brief Stops logging metadata cache events if logging was previously enabled and is currently ongoing + * + * \file_id + * + * \return \herr_t + * + * \details The metadata cache is a central part of the HDF5 library through + * which all \Emph{file metadata} reads and writes take place. File + * metadata is normally invisible to the user and is used by the + * library for purposes such as locating and indexing data. File + * metadata should not be confused with user metadata, which consists + * of attributes created by users and attached to HDF5 objects such + * as datasets via H5A API calls. + * + * Due to the complexity of the cache, a trace/logging feature has been + * created that can be used by HDF5 developers for debugging and performance + * analysis. The functions that control this functionality will normally be + * of use to a very limited number of developers outside of The HDF Group. + * The functions have been documented to help users create logs that can + * be sent with bug reports. + * + * Control of the log functionality is straightforward. Logging is enabled + * via the H5Pset_mdc_log_options() function, which will modify the file + * access property list used to open or create a file. This function has + * a flag that determines whether logging begins at file open or starts + * in a paused state. Log messages can then be controlled via the + * H5Fstart_mdc_logging() and H5Fstop_mdc_logging() functions. + * H5Pget_mdc_log_options() can be used to examine a file access property + * list, and H5Fget_mdc_logging_status() will return the current state of + * the logging flags. + * + * The log format is described in the \Emph{Metadata Cache Logging} document. + * + * \note Logging can only be started or stopped if metadata cache logging was enabled + * via H5Pset_mdc_log_options().\n + * This function only suspends the logging operations. The log file will remain + * open and will not be closed until the HDF5 file is closed. + * + * \since 1.10.0 + * */ H5_DLL herr_t H5Fstop_mdc_logging(hid_t file_id); /** * \ingroup MDC * - * \todo Finish this! + * \brief Gets the current metadata cache logging status + * + * \file_id + * \param[out] is_enabled Whether logging is enabled + * \param[out] is_currently_logging Whether events are currently being logged + * \return \herr_t + * + * \details The metadata cache is a central part of the HDF5 library through + * which all \Emph{file metadata} reads and writes take place. File + * metadata is normally invisible to the user and is used by the + * library for purposes such as locating and indexing data. File + * metadata should not be confused with user metadata, which consists + * of attributes created by users and attached to HDF5 objects such + * as datasets via H5A API calls. + * + * Due to the complexity of the cache, a trace/logging feature has been + * created that can be used by HDF5 developers for debugging and performance + * analysis. The functions that control this functionality will normally be + * of use to a very limited number of developers outside of The HDF Group. + * The functions have been documented to help users create logs that can + * be sent with bug reports. + * + * Control of the log functionality is straightforward. Logging is enabled + * via the H5Pset_mdc_log_options() function, which will modify the file + * access property list used to open or create a file. This function has + * a flag that determines whether logging begins at file open or starts + * in a paused state. Log messages can then be controlled via the + * H5Fstart_mdc_logging() and H5Fstop_mdc_logging() functions. + * H5Pget_mdc_log_options() can be used to examine a file access property + * list, and H5Fget_mdc_logging_status() will return the current state of + * the logging flags. + * + * The log format is described in the \Emph{Metadata Cache Logging} document. + * + * \note Unlike H5Fstart_mdc_logging() and H5Fstop_mdc_logging(), this function can + * be called on any open file identifier. + * + * \since 1.10.0 */ H5_DLL herr_t H5Fget_mdc_logging_status(hid_t file_id, /*OUT*/ hbool_t *is_enabled, @@ -1246,7 +1985,32 @@ H5_DLL herr_t H5Fget_page_buffering_stats(hid_t file_id, unsigned accesses[2], /** * \ingroup MDC * - * \todo Finish this! + * \brief Obtains information about a cache image if it exists + * + * \file_id + * \param[out] image_addr Offset of the cache image if it exists, or #HADDR_UNDEF if it does not + * \param[out] image_size Length of the cache image if it exists, or 0 if it does not + * \returns \herr_t + * + * \details + * \parblock + * H5Fget_mdc_image_info() returns information about a cache image if it exists. + * + * When an HDF5 file is opened in Read/Write mode, any metadata cache image will + * be read and deleted from the file on the first metadata cache access (or, if + * persistent free space managers are enabled, on the first file space + * allocation / deallocation, or read of free space manager status, whichever + * comes first). + * + * Thus, if the file is opened Read/Write, H5Fget_mdc_image_info() should be called + * immediately after file open and before any other operation. If H5Fget_mdc_image_info() + * is called after the cache image is loaded, it will correctly report that no cache image + * exists, as the image will have already been read and deleted from the file. In the Read Only + * case, the function may be called at any time, as any cache image will not be deleted + * from the file. + * \endparblock + * + * \since 1.10.1 */ H5_DLL herr_t H5Fget_mdc_image_info(hid_t file_id, haddr_t *image_addr, hsize_t *image_size); /** @@ -1308,13 +2072,91 @@ H5_DLL herr_t H5Fset_dset_no_attrs_hint(hid_t file_id, hbool_t minimize); /** * \ingroup PH5F * - * \todo Finish this! + * \brief Sets the MPI atomicity mode + * + * \file_id + * \param[in] flag Logical flag for atomicity setting. Valid values are: + * \li \c 1 -- Sets MPI file access to atomic mode. + * \li \c 0 -- Sets MPI file access to nonatomic mode. + * \returns \herr_t + * + * \par Motivation + * H5Fset_mpi_atomicity() is applicable only in parallel environments using MPI I/O. + * The function is one of the tools used to ensure sequential consistency. This means + * that a set of operations will behave as though they were performed in a serial + * order consistent with the program order. + * + * \details + * \parblock + * H5Fset_mpi_atomicity() sets MPI consistency semantics for data access to the file, + * \p file_id. + * + * If \p flag is set to \c 1, all file access operations will appear atomic, guaranteeing + * sequential consistency. If \p flag is set to \c 0, enforcement of atomic file access + * will be turned off. + * + * H5Fset_mpi_atomicity() is a collective function and all participating processes must + * pass the same values for \p file_id and \p flag. + * + * This function is available only when the HDF5 library is configured with parallel support + * (\Code{--enable-parallel}). It is useful only when used with the #H5FD_MPIO driver + * (see H5Pset_fapl_mpio()). + * \endparblock + * + * \attention + * \parblock + * H5Fset_mpi_atomicity() calls \Code{MPI_File_set_atomicity} underneath and is not supported + * if the execution platform does not support \Code{MPI_File_set_atomicity}. When it is + * supported and used, the performance of data access operations may drop significantly. + * + * In certain scenarios, even when \Code{MPI_File_set_atomicity} is supported, setting + * atomicity with H5Fset_mpi_atomicity() and \p flag set to 1 does not always yield + * strictly atomic updates. For example, some H5Dwrite() calls translate to multiple + * \Code{MPI_File_write_at} calls. This happens in all cases where the high-level file + * access routine translates to multiple lower level file access routines. + * The following scenarios will raise this issue: + * \li Non-contiguous file access using independent I/O + * \li Partial collective I/O using chunked access + * \li Collective I/O using filters or when data conversion is required + * + * This issue arises because MPI atomicity is a matter of MPI file access operations rather + * than HDF5 access operations. But the user is normally seeking atomicity at the HDF5 level. + * To accomplish this, the application must set a barrier after a write, H5Dwrite(), but before + * the next read, H5Dread(), in addition to calling H5Fset_mpi_atomicity().The barrier will + * guarantee that all underlying write operations execute atomically before the read + * operations starts. This ensures additional ordering semantics and will normally produce + * the desired behavior. + * \endparblock + * + * \see Enabling a Strict Consistency Semantics Model in Parallel HDF5 + * + * \since 1.8.9 + * + * \todo Fix the reference! */ H5_DLL herr_t H5Fset_mpi_atomicity(hid_t file_id, hbool_t flag); /** * \ingroup PH5F * - * \todo Finish this! + * \brief Retrieves the atomicity mode in use + * + * \file_id + * \param[out] flag Logical flag for atomicity setting. Valid values are: + * \li 1 -- MPI file access is set to atomic mode. + * \li 0 -- MPI file access is set to nonatomic mode. + * \returns \herr_t + * + * \details H5Fget_mpi_atomicity() retrieves the current consistency semantics mode for + * data access for the file \p file_id. + * + * Upon successful return, \p flag will be set to \c 1 if file access is set + * to atomic mode and \c 0 if file access is set to nonatomic mode. + * + * \see Enabling a Strict Consistency Semantics Model in Parallel HDF5 + * + * \since 1.8.9 + * + * \todo Fix the reference! */ H5_DLL herr_t H5Fget_mpi_atomicity(hid_t file_id, hbool_t *flag); #endif /* H5_HAVE_PARALLEL */ -- cgit v0.12