Fix for CVE-2016-4332 (#3406)

This CVE issue was previously listed as fixed (via HDFFV-9950) back in
2016, but with no confirmation test. Now that test files exist for
the 2016 Talos CVE issues, we found that CVE-2016-4332 can raise an
assert in debug builds.

This fix replaces the assert with pointer checks that don't raise
errors or asserts. Since the function is in cleanup code, we do our
best to close and free things, even when presented with partially-
initialized structs.

Fixes CVE-2016-4332 and HDFFV-9950 (confirmed via the cve_hdf5 repo)

1 files changed, 7 insertions, 9 deletions

diff --git a/src/H5Omessage.c b/src/H5Omessage.c
index d76409d..e492ceb 100644
--- a/src/H5Omessage.c
+++ b/src/H5Omessage.c
@@ -619,13 +619,12 @@ H5O__msg_free_mesg(H5O_mesg_t *mesg)
 } /* end H5O__msg_free_mesg() */
 
 /*-------------------------------------------------------------------------
- * Function:	H5O_msg_free_real
+ * Function:    H5O_msg_free_real
  *
- * Purpose:	Similar to H5O_msg_reset() except it also frees the message
- *		pointer.
+ * Purpose:     Similar to H5O_msg_reset() except it also frees the message
+ *              pointer
  *
- * Return:	Success:	NULL
- *		Failure:	NULL
+ * Return:      NULL (always)
  *
  *-------------------------------------------------------------------------
  */
@@ -634,16 +633,15 @@ H5O_msg_free_real(const H5O_msg_class_t *type, void *msg_native)
 {
     FUNC_ENTER_NOAPI_NOINIT_NOERR
 
-    /* check args */
-    assert(type);
+    /* Don't assert on args since this could be called in cleanup code */
 
     if (msg_native) {
         H5O__msg_reset_real(type, msg_native);
-        if (NULL != (type->free))
+        if (type && type->free)
             (type->free)(msg_native);
         else
             H5MM_xfree(msg_native);
-    } /* end if */
+    }
 
     FUNC_LEAVE_NOAPI(NULL)
 } /* end H5O_msg_free_real() */