Prevent buffer overrun in H5S_select_deserialize (#2931)

* Prevent buffer overrun in H5S_select_deserialize

The call to H5S_select_deserialize from H5S_decode doesn't have
the buffer size available to it, so to allow decoding there
I set it to assume a max size buffer for now.

Making the buffer size known in H5S_decode could be done by
modifying the external API's H5Sdecode, or splitting H5Sdecode
into two functions using a macro (similar to H5Sencode), with the
macro taking one argument and assuming a max buffer size.

* Conditional buffer check in H5S_select_deserialize

Moved and renamed a macro for only checking buffer overflow when
buffer size is known from H5Odtype.c to H5private.h,
so it can be used throughout the library.

Also silenced some build warnings about types.

1 files changed, 3 insertions, 2 deletions

diff --git a/src/H5S.c b/src/H5S.c
index 22de41f..c938a71 100644
--- a/src/H5S.c
+++ b/src/H5S.c
@@ -1655,9 +1655,10 @@ H5S_decode(const unsigned char **p)
     if (H5S_select_all(ds, FALSE) < 0)
         HGOTO_ERROR(H5E_DATASPACE, H5E_CANTSET, NULL, "unable to set all selection")
 
-    /* Decode the select part of dataspace.  I believe this part always exists. */
+    /* Decode the select part of dataspace.
+     *  Because size of buffer is unknown, assume arbitrarily large buffer to allow decoding. */
     *p = pp;
-    if (H5S_SELECT_DESERIALIZE(&ds, p) < 0)
+    if (H5S_SELECT_DESERIALIZE(&ds, p, SIZE_MAX) < 0)
         HGOTO_ERROR(H5E_DATASPACE, H5E_CANTDECODE, NULL, "can't decode space selection")
 
     /* Set return value */