Fix for github issue #2599: (#2665)

* Fix for github issue #2599:
As indicated in the description, memory leak is detected when running "./h5dump pov".

The problem is: when calling H5O__add_cont_msg() from H5O__chunk_deserialize(),
memory is allocated for cont_msg_info->msgs.  Eventually, when the library tries to load
the continuation message via H5AC_protect() in H5O_protect(), error is
encountered due to illegal info in the continuation message.
Due to the error, H5O_protect() exits but the memory allocated for cont_msg_info->msgs is not freed.

When we figure out how to handle fuzzed files that we didn't generate,
a test needs to be added to run h5dump with the provided "pov" file.

* Add message to release notes for the fix to github issue #2599.

2 files changed, 21 insertions, 1 deletions

diff --git a/release_docs/RELEASE.txt b/release_docs/RELEASE.txt
index c04ead5..0fde1ab 100644
--- a/release_docs/RELEASE.txt
+++ b/release_docs/RELEASE.txt
@@ -151,6 +151,21 @@ Bug Fixes since HDF5-1.13.3 release
 ===================================
     Library
     -------
+
+    - Memory leak
+
+      Memory leak was detected when running h5dump with "pov".  The memory was allocated
+      via H5FL__malloc() in hdf5/src/H5FL.c
+
+      The fuzzed file "pov" was an HDF5 file containing an illegal continuation message.
+      When deserializing the object header chunks for the file, memory is allocated for the
+      array of continuation messages (cont_msg_info->msgs) in continuation message info struct.
+      As error is encountered in loading the illegal message, the memory allocated for
+      cont_msg_info->msgs needs to be freed.
+
+      (VC - 2023/04/11 GH-2599)
+
+
     - Fixed memory leaks that could occur when reading a dataset from a
       malformed file
 
diff --git a/src/H5Oint.c b/src/H5Oint.c
index cdcf6c8..cad27e3 100644
--- a/src/H5Oint.c
+++ b/src/H5Oint.c
@@ -1166,9 +1166,14 @@ H5O_protect(const H5O_loc_t *loc, unsigned prot_flags, hbool_t pin_all_chunks)
     ret_value = oh;
 
 done:
-    if (ret_value == NULL && oh)
+    if (ret_value == NULL && oh) {
+        /* Release any continuation messages built up */
+        if (cont_msg_info.msgs)
+            cont_msg_info.msgs = (H5O_cont_t *)H5FL_SEQ_FREE(H5O_cont_t, cont_msg_info.msgs);
+
         if (H5O_unprotect(loc, oh, H5AC__NO_FLAGS_SET) < 0)
             HDONE_ERROR(H5E_OHDR, H5E_CANTUNPROTECT, NULL, "unable to release object header")
+    }
 
     FUNC_LEAVE_NOAPI_TAG(ret_value)
 } /* end H5O_protect() */