VFD ctl feature (#981)

* Added "ctl" callback to the VFD interface, and the associated
H5FDctl() and H5FD_ctl() calls.

Modified the MPIO VFD accordingly -- specifically:

Added ctl() call with op-code support to expose rank, size,
and communicator.

Modified H5FD_mpi_get_rank(), H5FD_mpi_get_size(),
and H5FD_mpi_get_comm() to use the new ctl() callback.  In passing
removed the const qualifier from the file parameter of these
functions, as the file parameter of the ctl callback is not
const.

Deleted the old  H5FD__mpio_mpi_rank(), H5FD__mpio_mpi_size(), and
H5FD__mpio_communicator() calls from the MPIO VFD.

Deleted H5FD_class_mpi_t from H5FDprivate.h, and modified the
MPIO VFD accordingly.  Note that all VFDs now use H5FD_class_t,
with no special class for VFDs that that support MPI.

Some minor touch ups to the Neil's selection I/O mods in passing.

Tested serial and parallel, debug and production on charis and
jelly.

* Reserve a range of VFD "ctl" opcodes for library and experimental usage

* Add "ctl" callbacks to passthrough VFDs

* Add RELEASE.txt entry for "ctl" callback

* Use H5FDopen with H5F_ACC_RDWR flag instead of H5F_ACC_TRUNC in vfd test

* Remove handling of passthrough "ctl" flag from multi VFD

* Move logic for testing H5FD_CTL__TEST_OPCODE into a testing VFD

Revise description of "ctl" callback in RELEASE.txt

Remove unused H5FD_CTL__NUM_OPCODES definition

Fix some warnings in multi VFD

Co-authored-by: mainzer <mainzer#hdfgroup.org>

1 files changed, 71 insertions, 0 deletions

diff --git a/src/H5FDpublic.h b/src/H5FDpublic.h
index 0cfb072..b8f4f12 100644
--- a/src/H5FDpublic.h
+++ b/src/H5FDpublic.h
@@ -138,6 +138,77 @@
  */
 #define H5FD_FEAT_DEFAULT_VFD_COMPATIBLE 0x00008000
 
+/* ctl function definitions: */
+#define H5FD_CTL_OPC_RESERVED 512 /* Opcodes below this value are reserved for library use */
+#define H5FD_CTL_OPC_EXPER_MIN                                                                               \
+    H5FD_CTL_OPC_RESERVED /* Minimum opcode value available for experimental use                             \
+                           */
+#define H5FD_CTL_OPC_EXPER_MAX                                                                               \
+    (H5FD_CTL_OPC_RESERVED + 511) /* Maximum opcode value available for experimental use */
+
+/* ctl function op codes: */
+#define H5FD_CTL__INVALID_OPCODE              0
+#define H5FD_CTL__TEST_OPCODE                 1
+#define H5FD_CTL__GET_MPI_COMMUNICATOR_OPCODE 2
+#define H5FD_CTL__GET_MPI_RANK_OPCODE         3
+#define H5FD_CTL__GET_MPI_SIZE_OPCODE         4
+
+/* ctl function flags: */
+
+/* Definitions:
+ *
+ * WARNING: While the following definitions of Terminal
+ * and Passthrough VFDs should be workable for now, they
+ * have to be adjusted as our use cases for VFDs expand.
+ *
+ *                                   JRM -- 8/4/21
+ *
+ *
+ * Terminal VFD: Lowest VFD in the VFD stack through
+ * which all VFD calls pass.  Note that this definition
+ * is situational.  For example, the sec2 VFD is typically
+ * terminal.  However, in the context of the family file
+ * driver, it is not -- the family file driver is the
+ * bottom VFD through which all VFD calls pass, and thus
+ * it is terminal.
+ *
+ * Similarly, on the splitter VFD, a sec2 VFD on the
+ * R/W channel is terminal, but a sec2 VFD on the W/O
+ * channel is not.
+ *
+ *
+ * Pass through VFD:  Any VFD that relays all VFD calls
+ * (with the possible exception of some non-I/O related
+ * calls) to underlying VFD(s).
+ */
+
+/* Unknown op codes should be ignored silently unless the
+ * H5FD_CTL__FAIL_IF_UNKNOWN_FLAG is set.
+ *
+ * On terminal VFDs, unknown op codes should generate an
+ * error unconditionally if this flag is set.
+ *
+ * On pass through VFDs, unknown op codes should be routed
+ * to the underlying VFD(s) as indicated by any routing
+ * flags.  In the absence of such flags, the VFD should
+ * generate an error.
+ */
+#define H5FD_CTL__FAIL_IF_UNKNOWN_FLAG 0x0001
+
+/* The H5FD_CTL__ROUTE_TO_TERMINAL_VFD_FLAG is used only
+ * by non-ternminal VFDs, and only applies to unknown
+ * opcodes. (known op codes should be handled as
+ * appropriate.)
+ *
+ * If this flag is set for an uknown op code, that
+ * op code should be passed to the next VFD down
+ * the VFD stack en-route to the terminal VFD.
+ * If that VFD does not support the ctl call, the
+ * pass through VFD should fail or succeed as directed
+ * by the  H5FD_CTL__FAIL_IF_UNKNOWN_FLAG.
+ */
+#define H5FD_CTL__ROUTE_TO_TERMINAL_VFD_FLAG 0x0002
+
 /*******************/
 /* Public Typedefs */
 /*******************/