Fix for HDFFV-11052: h5debug fails on a corrupted file (h5_nrefs_POC)… (#2291) (#2496)

* Fix for HDFFV-11052: h5debug fails on a corrupted file (h5_nrefs_POC) producing a core dump.
When h5debug closes the corrupted file, the library calls H5F__dest() which performs all the
closing operations for the file "f" (H5F_t *) but just keeping note of errors in "ret_value"
all the way till the end of the routine.  The user-provided corrupted file has an illegal
file size causing failure when reading the image during the closing process.
At the end of this routine it sets f->shared to NULL and then frees "f".
This is done whether there is error or not in "ret_value".
Due to the failure in reading the file earlier, the routine then returns error.
The error return from H5F__dest() causes the file object "f" not being removed from the
ID node table.  When the library finally exits, it will try to close the
file objects in the table.  This causes assert failure when H5F_ID_EXISTS(f) or H5F_NREFS(f).
Fix:
a) H5F_dest(): free the f only when there is no error in "ret_value" at the end of the routine.
b) H5VL__native_file_close(): if f->shared is NULL, free "f"; otherwise, perform closing on "f" as before.
c) h5debug.c main(): track error return from H5Fclose().

* Committing clang-format changes

Co-authored-by: vchoi <vchoi@jelly.ad.hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

1 files changed, 19 insertions, 0 deletions

diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt
index 3226165..acbde91 100644
--- a/release_docs/RELEASE.txt
+++ b/release_docs/RELEASE.txt
@@ -180,6 +180,25 @@ Bug Fixes since HDF5-1.12.1 release
 ===================================
     Library
     -------
+    - Seg fault on file close
+
+      h5debug fails at file close with core dump on a file that has an
+      illegal file size in its cache image.  In H5F_dest(), the library
+      performs all the closing operations for the file and keeps track of
+      the error encountered when reading the file cache image.  
+      At the end of the routine, it frees the file's file structure and 
+      returns error.  Due to the error return, the file object is not removed 
+      from the ID node table.  This eventually causes assertion failure in 
+      H5VL__native_file_close() when the library finally exits and tries to 
+      access that file object in the table for closing.
+
+      The closing routine, H5F_dest(), will not free the file structure if
+      there is error, keeping a valid file structure in the ID node table.
+      It will be freed later in H5VL__native_file_close() when the
+      library exits and terminates the file package.
+
+      (VC - 2022/12/14, HDFFV-11052, CVE-2020-10812)
+
     - Fixed an issue with variable length attributes
 
       Previously, if a variable length attribute was held open while its file