Hdffv 11052 (#2303)

* Fix for HDFFV-11052: h5debug fails on a corrupted file (h5_nrefs_POC) producing a core dump.
When h5debug closes the corrupted file, the library calls H5F__dest() which performs all the
closing operations for the file "f" (H5F_t *) but just keeping note of errors in "ret_value"
all the way till the end of the routine.  The user-provided corrupted file has an illegal
file size causing failure when reading the image during the closing process.
At the end of this routine it sets f->shared to NULL and then frees "f".
This is done whether there is error or not in "ret_value".
Due to the failure in reading the file earlier, the routine then returns error.
The error return from H5F__dest() causes the file object "f" not being removed from the
ID node table.  When the library finally exits, it will try to close the
file objects in the table.  This causes assert failure when H5F_ID_EXISTS(f) or H5F_NREFS(f).
Fix:
a) H5F_dest(): free the f only when there is no error in "ret_value" at the end of the routine.
b) H5VL__native_file_close(): if f->shared is NULL, free "f"; otherwise, perform closing on "f" as before.
c) h5debug.c main(): track error return from H5Fclose().

* Committing clang-format changes

* Add test and release note info for fix to HDFFV-11052 which is merged via PR#2291.

* Committing clang-format changes

* Add the test file to Cmake.

Co-authored-by: vchoi <vchoi@jelly.ad.hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

3 files changed, 40 insertions, 0 deletions

diff --git a/test/CMakeTests.cmake b/test/CMakeTests.cmake
index 74f63f4..5086194 100644
--- a/test/CMakeTests.cmake
+++ b/test/CMakeTests.cmake
@@ -126,6 +126,7 @@ set (HDF5_REFERENCE_TEST_FILES
     btree_idx_1_6.h5
     btree_idx_1_8.h5
     corrupt_stab_msg.h5
+    cve_2020_10812.h5
     deflate.h5
     family_v16-000000.h5
     family_v16-000001.h5
diff --git a/test/cve_2020_10812.h5 b/test/cve_2020_10812.h5
new file mode 100755
index 0000000..a20369d
--- /dev/null
+++ b/test/cve_2020_10812.h5
diff --git a/test/tmisc.c b/test/tmisc.c
index 5cad368..8a70a29 100644
--- a/test/tmisc.c
+++ b/test/tmisc.c
@@ -330,6 +330,11 @@ typedef struct {
 #define MISC35_SPACE_DIM3 13
 #define MISC35_NPOINTS    10
 
+/* Definitions for misc. test #37 */
+/* The test file is formerly named h5_nrefs_POC.
+   See https://nvd.nist.gov/vuln/detail/CVE-2020-10812 */
+#define CVE_2020_10812_FILENAME "cve_2020_10812.h5"
+
 /****************************************************************
 **
 **  test_misc1(): test unlinking a dataset from a group and immediately
@@ -6046,6 +6051,39 @@ test_misc36(void)
 
 /****************************************************************
 **
+**  test_misc37():
+**      Test for seg fault issue when closing the provided test file
+**      which has an illegal file size in its cache image.
+**      See HDFFV-11052/CVE-2020-10812 for details.
+**
+****************************************************************/
+static void
+test_misc37(void)
+{
+    const char *fname;
+    hid_t       fid;
+    herr_t      ret;
+
+    /* Output message about test being performed */
+    MESSAGE(5, ("Fix for HDFFV-11052/CVE-2020-10812"));
+
+    fname = H5_get_srcdir_filename(CVE_2020_10812_FILENAME);
+    fid   = H5Fopen(fname, H5F_ACC_RDONLY, H5P_DEFAULT);
+    CHECK(fid, FAIL, "H5Fopen");
+
+    /* This should fail due to the illegal file size.
+       It should fail gracefully and not seg fault */
+    H5E_BEGIN_TRY
+    {
+        ret = H5Fclose(fid);
+    }
+    H5E_END_TRY;
+    VERIFY(ret, FAIL, "H5Fclose");
+
+} /* end test_misc37() */
+
+/****************************************************************
+**
 **  test_misc(): Main misc. test routine.
 **
 ****************************************************************/
@@ -6111,6 +6149,7 @@ test_misc(void)
     test_misc34(); /* Test behavior of 0 and NULL in H5MM API calls */
     test_misc35(); /* Test behavior of free-list & allocation statistics API calls */
     test_misc36(); /* Exercise H5atclose and H5is_library_terminating */
+    test_misc37(); /* Test for seg fault failure at file close */
 
 } /* test_misc() */