Fixed HDFFV-10586, HDFFV-10588, and HDFFV-10684

Description:
    HDFFV-10586 CVE-2018-17434 Divide by zero in h5repack_filters
        Added a check for zero value
    HDFFV-10588 CVE-2018-17437 Memory leak in H5O_dtype_decode_helper
        This is actually an Invalid read issue.  It was found that the
        attribute name length in an attribute message was corrupted,
        which caused the buffer pointer to be advanced too far and later
        caused an invalid read.
        Added a check to detect attribute name and its length mismatch.  The
        fix does not cover all cases, but it'll reduce the chance of this issue
        when a name length is corrupted or the attribute name is corrupted.
    HDFFV-10684 H5Ewalk does not stop until all errors in the stack are visited
        The test for HDFFV-10588 has revealed a bug in H5Ewalk.
        H5Ewalk did not stop midway even when the call back function returns
        H5_ITER_STOP. This is because a condition is missing from the for
        loops in H5E__walk causing the callback functions unable to stop until
        all the errors in the stack are iterated. Quincey advised on the final
        fix.  In this fix, "status" is switched to "ret_value" and HGOTO_ERROR
        to HERROR, and the for loops won't continue when "ret_value" is not 0.
Platforms tested:
    Linux/64 (jelly)
    Linux/64 (platypus)
    Darwin (osx1011test)

1 files changed, 4 insertions, 2 deletions

diff --git a/tools/src/h5repack/h5repack_filters.c b/tools/src/h5repack/h5repack_filters.c
index 0092abc..3d9472a 100644
--- a/tools/src/h5repack/h5repack_filters.c
+++ b/tools/src/h5repack/h5repack_filters.c
@@ -338,12 +338,14 @@ int apply_filters(const char* name, /* object name from traverse list */
 
             sm_nbytes = msize;
             for (i = rank; i > 0; --i) {
-                hsize_t size = H5TOOLS_BUFSIZE / sm_nbytes;
+                hsize_t size = 0;
+                if(sm_nbytes == 0)
+                    HGOTO_ERROR(FAIL, H5E_tools_min_id_g, "number of bytes per stripmine must be > 0");
+                size = H5TOOLS_BUFSIZE / sm_nbytes;
                 if (size == 0) /* datum size > H5TOOLS_BUFSIZE */
                     size = 1;
                 sm_size[i - 1] = MIN(dims[i - 1], size);
                 sm_nbytes *= sm_size[i - 1];
-                HDassert(sm_nbytes > 0);
             }
 
             for (i = 0; i < rank; i++) {