Prevent buffer overrun in H5S_select_deserialize (#2931)

* Prevent buffer overrun in H5S_select_deserialize

The call to H5S_select_deserialize from H5S_decode doesn't have
the buffer size available to it, so to allow decoding there
I set it to assume a max size buffer for now.

Making the buffer size known in H5S_decode could be done by
modifying the external API's H5Sdecode, or splitting H5Sdecode
into two functions using a macro (similar to H5Sencode), with the
macro taking one argument and assuming a max buffer size.

* Conditional buffer check in H5S_select_deserialize

Moved and renamed a macro for only checking buffer overflow when
buffer size is known from H5Odtype.c to H5private.h,
so it can be used throughout the library.

Also silenced some build warnings about types.

1 files changed, 9 insertions, 0 deletions

diff --git a/src/H5private.h b/src/H5private.h
index eac2cba..91a4780 100644
--- a/src/H5private.h
+++ b/src/H5private.h
@@ -327,6 +327,15 @@
  */
 #define H5_IS_BUFFER_OVERFLOW(ptr, size, buffer_end) (((ptr) + (size)-1) > (buffer_end))
 
+/* Variant of H5_IS_BUFFER_OVERFLOW, used with functions such as H5Tdecode()
+ * that don't take a size parameter, where we need to skip the bounds checks.
+ *
+ * This is a separate macro since we don't want to inflict that behavior on
+ * the entire library.
+ */
+#define H5_IS_KNOWN_BUFFER_OVERFLOW(skip, ptr, size, buffer_end)                                             \
+    (skip ? FALSE : ((ptr) + (size)-1) > (buffer_end))
+
 /*
  * HDF Boolean type.
  */