diff options
| author | James Laird <jlaird@hdfgroup.org> | 2004-07-14 19:32:51 (GMT) |
|---|---|---|
| committer | James Laird <jlaird@hdfgroup.org> | 2004-07-14 19:32:51 (GMT) |
| commit | d89d73048a57c8dd1bffb3596c633c77d45d5ea6 (patch) | |
| tree | 25733c713399bb23e8e54ac8f86170dd0a814796 | |
| parent | f0fe9b0114591c13051ccc92a2fca81b1ec3ca95 (diff) | |
| download | hdf5-d89d73048a57c8dd1bffb3596c633c77d45d5ea6.zip hdf5-d89d73048a57c8dd1bffb3596c633c77d45d5ea6.tar.gz hdf5-d89d73048a57c8dd1bffb3596c633c77d45d5ea6.tar.bz2 | |
[svn-r8876]
Purpose:
Bug Fix
Description:
If an HDF5 file grows larger than its address space, it dies and is unable to
write any data. This is more likely to happen since users are able to change
the number of bytes used to store addresses in the file.
Solution:
HDF5 now throws an error instead of dying. In addition, it "reserves" address
space for the local heap and for object headers (which do not allocate space
immediately). This ensures that after the error occurs, there is enough address
space left to flush the entire file to disk, so no data is lost.
A more complete explanation is at /doc/html/TechNotes/ReservedFileSpace.html
Platforms tested:
sleipnir, copper (parallel), verbena, arabica, Windows (Visual Studio 7)
Misc. update:
| -rw-r--r-- | MANIFEST | 2 | ||||
| -rw-r--r-- | doc/html/TechNotes.html | 5 | ||||
| -rw-r--r-- | doc/html/TechNotes/ReservedFileSpace.html | 29 | ||||
| -rw-r--r-- | src/H5B.c | 5 | ||||
| -rw-r--r-- | src/H5FD.c | 1 | ||||
| -rw-r--r-- | src/H5FDpublic.h | 1 | ||||
| -rw-r--r-- | src/H5HG.c | 6 | ||||
| -rw-r--r-- | src/H5HL.c | 22 | ||||
| -rw-r--r-- | src/H5MF.c | 137 | ||||
| -rw-r--r-- | src/H5MFprivate.h | 3 | ||||
| -rw-r--r-- | src/H5O.c | 32 | ||||
| -rw-r--r-- | src/H5Tconv.c | 8 | ||||
| -rw-r--r-- | test/Makefile.in | 8 | ||||
| -rw-r--r-- | test/lheap.c | 3 | ||||
| -rw-r--r-- | test/ohdr.c | 3 | ||||
| -rwxr-xr-x | test/reserved.c | 399 |
16 files changed, 647 insertions, 17 deletions
@@ -221,6 +221,7 @@ ./doc/html/TechNotes/NamingScheme.html ./doc/html/TechNotes/ObjectHeader.html ./doc/html/TechNotes/RawDStorage.html +./doc/html/TechNotes/ReservedFileSpace.html ./doc/html/TechNotes/SWControls.html ./doc/html/TechNotes/SymbolTables.html ./doc/html/TechNotes/TestReview.html @@ -1044,6 +1045,7 @@ ./test/ntypes.c ./test/noencoder.h5 ./test/ohdr.c +./test/reserved.c ./test/space_overflow.c _DO_NOT_DISTRIBUTE_ ./test/gen_deflate.c _DO_NOT_DISTRIBUTE_ ./test/gen_old_array.c _DO_NOT_DISTRIBUTE_ diff --git a/doc/html/TechNotes.html b/doc/html/TechNotes.html index 83ec113..dd15b02 100644 --- a/doc/html/TechNotes.html +++ b/doc/html/TechNotes.html @@ -214,6 +214,11 @@ HDF5 Technical Notes A report on the implementation of a thread safe HDF5 library. </td></tr> +<tr><td valign=top><a href="TechNotes/ReservedFileSpace.html">Reserved File Address Space</a> + </td><td> </td><td valign=top> + Description of HDF5's internal system for ensuring that files stay within their address space. +</td></tr> + <tr><td valign=top> <a href="TechNotes/openmp-hdf5.html">Using HDF5 with OpenMP</a> </td><td> </td><td valign=top> diff --git a/doc/html/TechNotes/ReservedFileSpace.html b/doc/html/TechNotes/ReservedFileSpace.html new file mode 100644 index 0000000..819e050 --- /dev/null +++ b/doc/html/TechNotes/ReservedFileSpace.html @@ -0,0 +1,29 @@ +<html> + <head> + <title>Reserved File Address Space</title> + </head> + + <body> + <hl>Reserved File Address Space</hl> + + <p>HDF5 files use 8-byte addresses by default, but users can change this to 2, 4, or even 16 bytes. This means that it is possible to have files that only address 64 KB of space, and thus that HDF must handle the case of files that have enough space on disk but not enough internal address space to be written.</p> + + <p>Thus, every time space is allocated in a file, HDF needs to check that this allocation is within the file’s address space. If not, HDF should output an error and ensure that all the data currently in the file (everything that is still addressable) is successfully written to disk.</p> + + <p>Unfortunately, some structures are stored in memory and do not allocate space for themselves until the file is actually flushed to disk (object headers and the local heap). This is good for efficiency, since these structures can grow without creating the fragmentation that would result from frequent allocation and deallocation, but means that if the library runs out of addressable space while allocating memory, these structures will not be present in the file. Without them, HDF5 does not know how to parse the data in the file, rendering it unreadable.</p> + + <p>Thus, HDF keeps track of the space “reserved for allocation” in the file (H5FD_t struct). When a function tries to allocate space in the file, it first checks that the allocation would not overflow the address space, taking the reserved space into account. When object headers or the heap finally allocate the space they have reserved, they free the reserved space before allocating file space.</p> + + <p>A given object header is only flushed to disk once, but the heap can be flushed to disk multiple times over the life of the file and will require contiguous space every time. To handle this, the heap keeps track of how much space it has reserved. This allows it to reserve space only when it grows (when it is dirty and needs to be re-written to disk).</p> + + <p>For instance, if the heap is flushed to disk, it frees its reserved space. If new data is inserted into the heap in memory, the heap may need to flush to disk again in a new, larger section of memory. Thus, not only does it reserve space in the file for this new data, but also for all of the previously-existing data in the heap to be re-written. The next insert, however, will only need to reserve space for its new data, since the rest of the heap already has space reserved for it.</p> + + <p>Potential issues: + <ol> + <li>This system does not take into accout deleted space. Deleted space can still be allocated as usual, but "reserved" space is always taken off the end of a file. This means that a file that was filled to capacity but then had a significant number of objects deleted will still throw errors if more data is written. This occurs because the file's free space is in the middle of the file, not at the end. A more complete space-reservation system would test if the reserved data can fit into the file's free list, but this would be significantly more complicated to implement.</li> + + <li>HDF5 currently claims to support 16-byte addresses, but since the size of a long long is only 8 bytes, addresses of this size cannot be represented in memory. This solution does not attempt to address this issue.</li> + </ol> + </p> + </body> +</html> @@ -231,7 +231,7 @@ H5B_create(H5F_t *f, hid_t dxpl_id, const H5B_class_t *type, void *udata, bt->right = HADDR_UNDEF; bt->nchildren = 0; if((bt->rc_shared=(type->get_shared)(f, udata))==NULL) - HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, NULL, "can't retrieve B-tree node buffer") + HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, FAIL, "can't retrieve B-tree node buffer") shared=H5RC_GET_OBJ(bt->rc_shared); HDassert(shared); if (NULL==(bt->native=H5FL_BLK_MALLOC(native_block,shared->sizeof_keys)) || @@ -538,9 +538,6 @@ H5B_dest(H5F_t UNUSED *f, H5B_t *bt) */ assert(bt); - /* Verify that node is clean */ - assert(bt->cache_info.dirty==0); - H5FL_SEQ_FREE(haddr_t,bt->child); H5FL_BLK_FREE(native_block,bt->native); H5RC_DEC(bt->rc_shared); @@ -968,6 +968,7 @@ H5FD_open(const char *name, unsigned flags, hid_t fapl_id, haddr_t maxaddr) HGOTO_ERROR(H5E_VFL, H5E_CANTINC, NULL, "unable to increment ref count on VFL driver") file->cls = driver; file->maxaddr = maxaddr; + file->reserved_alloc = 0; HDmemset(file->fl, 0, sizeof(file->fl)); if(H5P_get(plist, H5F_ACS_META_BLOCK_SIZE_NAME, &(meta_block_size)) < 0) HGOTO_ERROR(H5E_PLIST, H5E_CANTGET, NULL, "can't get meta data block size") diff --git a/src/H5FDpublic.h b/src/H5FDpublic.h index 0f541f9..ba762db 100644 --- a/src/H5FDpublic.h +++ b/src/H5FDpublic.h @@ -187,6 +187,7 @@ struct H5FD_t { unsigned long feature_flags; /* VFL Driver feature Flags */ hsize_t threshold; /* Threshold for alignment */ hsize_t alignment; /* Allocation alignment */ + hsize_t reserved_alloc; /* Space reserved for later alloc calls */ /* Metadata aggregation fields */ hsize_t def_meta_block_size; /* Metadata allocation @@ -889,7 +889,13 @@ H5HG_insert (H5F_t *f, hid_t dxpl_id, size_t size, void *obj, H5HG_t *hobj/*out* * we can extend any of the collections to make enough room. */ if (!found) { + size_t new_need; + for (cwfsno=0; cwfsno<f->shared->ncwfs; cwfsno++) { + new_need = need; + new_need -= f->shared->cwfs[cwfsno]->obj[0].size; + new_need = MAX(f->shared->cwfs[cwfsno]->size, new_need); + if((f->shared->cwfs[cwfsno]->size+need)<=H5HG_MAXSIZE && H5MF_can_extend(f,H5FD_MEM_GHEAP,f->shared->cwfs[cwfsno]->addr,(hsize_t)f->shared->cwfs[cwfsno]->size,(hsize_t)need)) { if(H5HG_extend(f,f->shared->cwfs[cwfsno],size)<0) HGOTO_ERROR (H5E_HEAP, H5E_CANTINIT, FAIL, "unable to extend global heap collection"); @@ -71,6 +71,7 @@ typedef struct H5HL_t { haddr_t addr; /*address of data */ size_t disk_alloc; /*data bytes allocated on disk */ size_t mem_alloc; /*data bytes allocated in mem */ + size_t disk_resrv; /*data bytes "reserved" on disk */ uint8_t *chunk; /*the chunk, including header */ H5HL_free_t *freelist; /*the free list */ } H5HL_t; @@ -175,6 +176,7 @@ H5HL_create(H5F_t *f, hid_t dxpl_id, size_t size_hint, haddr_t *addr_p/*out*/) heap->addr = *addr_p + (hsize_t)sizeof_hdr; heap->disk_alloc = size_hint; heap->mem_alloc = size_hint; + heap->disk_resrv = 0; if (NULL==(heap->chunk = H5FL_BLK_CALLOC(heap_chunk,(sizeof_hdr + size_hint)))) HGOTO_ERROR (H5E_RESOURCE, H5E_NOSPACE, FAIL, "memory allocation failed"); @@ -370,6 +372,13 @@ H5HL_flush(H5F_t *f, hid_t dxpl_id, hbool_t destroy, haddr_t addr, H5HL_t *heap) sizeof_hdr= H5HL_SIZEOF_HDR(f); /* + * Since the file is being flushed to disk, release the file space reserved + * for it. + */ + H5MF_free_reserved(f, heap->disk_resrv); + heap->disk_resrv = 0; + + /* * Check to see if we can reduce the size of the heap in memory by * eliminating free blocks at the tail of the buffer before flushing the * buffer out. @@ -772,6 +781,7 @@ H5HL_insert(H5F_t *f, hid_t dxpl_id, haddr_t addr, size_t buf_size, const void * size_t offset = 0; size_t need_size, old_size, need_more; hbool_t found; + size_t disk_resrv; /* Amount of additional space to reserve in file */ size_t sizeof_hdr; /* Cache H5HL header size for file */ size_t ret_value; /* Return value */ @@ -880,6 +890,18 @@ H5HL_insert(H5F_t *f, hid_t dxpl_id, haddr_t addr, size_t buf_size, const void * } } + /* Reserve space in file to hold the increased heap size */ + if( heap->disk_resrv == heap->mem_alloc) + disk_resrv = need_more; + else + disk_resrv = heap->mem_alloc + need_more - heap->disk_resrv; + + if( H5MF_reserve(f, disk_resrv) < 0) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, (size_t) (-1), "unable to reserve space for heap"); + + /* Update heap's record of how much space it has reserved */ + heap->disk_resrv += disk_resrv; + #ifdef H5HL_DEBUG if (H5DEBUG(HL)) { fprintf(H5DEBUG(HL), @@ -80,6 +80,9 @@ H5MF_alloc(H5F_t *f, H5FD_mem_t type, hid_t dxpl_id, hsize_t size) /* Fail if we don't have write access */ if (0==(f->intent & H5F_ACC_RDWR)) HGOTO_ERROR(H5E_RESOURCE, H5E_CANTINIT, HADDR_UNDEF, "file is read-only"); + /* Check that the file can address the new space */ + if( H5MF_alloc_overflow(f, size) != 0 ) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "not enough address space in file"); /* Allocate space from the virtual file layer */ if (HADDR_UNDEF==(ret_value=H5FD_alloc(f->shared->lf, type, dxpl_id, size))) @@ -210,6 +213,132 @@ done: FUNC_LEAVE_NOAPI(ret_value); } +/*------------------------------------------------------------------------- + * Function: H5MF_reserve + * + * Purpose: Sets aside file space that has not yet been allocated, but will + * be (or might be in the worst case). This number is used to + * ensure that there is room in the file when it is flushed to disk. + * + * Nothing changes (and no error is generated) if the file is opened + * as read-only. + * + * Return: Success: 0 + * + * Failure: negative + * + * Programmer: James Laird + * Nat Furrer + * Thursday, May 27, 2004 + * + * Modifications: + *------------------------------------------------------------------------- + */ +herr_t H5MF_reserve(H5F_t *f, hsize_t size) +{ + herr_t ret_value = SUCCEED; + FUNC_ENTER_NOAPI(H5MF_reserve, FAIL); + + /* Check arguments */ + assert(f); + + /* Check that there is room in the file to reserve this space */ + if( H5MF_alloc_overflow( f, size ) != 0) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "not enough address space in file"); + + f->shared->lf->reserved_alloc += size; + +done: + FUNC_LEAVE_NOAPI(ret_value); +} + +/*------------------------------------------------------------------------- + * Function: H5MF_free_reserved + * + * Purpose: Releases the file space set aside by H5MF_reserve. This should + * be called immediately before allocating the file space for which + * the space was reserved. + * + * Return: None + * + * Programmer: James Laird + * Nat Furrer + * Thursday, May 27, 2004 + * + * Modifications: + *------------------------------------------------------------------------- + */ +void H5MF_free_reserved(H5F_t *f, hsize_t size) +{ + /* Check arguments */ + assert(f); + + /* If this assert breaks, it means that HDF5 is trying to free file space + * that was never reserved. + */ + assert(size <= f->shared->lf->reserved_alloc); + + f->shared->lf->reserved_alloc -= size; +} + +/*------------------------------------------------------------------------- + * Function: H5MF_alloc_overflow + * + * Purpose: Checks if an allocation of file space would cause an overflow. + * F is the file whose space is being allocated, SIZE is the amount + * of space needed. + * + * Return: 0 if no overflow would result + * 1 if overflow would result (the allocation should not be allowed) + * + * Programmer: James Laird + * Nat Furrer + * Tuesday, June 1, 2004 + * + * Modifications: + *------------------------------------------------------------------------- + */ +int H5MF_alloc_overflow(H5F_t *f, hsize_t size) +{ + unsigned long long space_needed; /* Accumulator variable */ + hsize_t c; + + /* Start with the current end of the file's address. */ + space_needed = f->shared->lf->cls->get_eoa(f->shared->lf); + + /* Subtract the file's base address to get the actual amount of + * space being used: + * (end of allocated space - beginning of allocated space) + */ + assert(f->shared->base_addr < space_needed); + space_needed -= f->shared->base_addr; + + /* Add the amount of space requested for this allocation */ + space_needed += size; + + /* Also add space that is "reserved" for data to be flushed + * to disk (e.g., for object headers and the heap). + * This is the total amount of file space that will be + * allocated. + */ + space_needed += f->shared->lf->reserved_alloc; + + /* Ensure that this final number is less than the file's + * address space. We do this by shifting in multiples + * of 16 bits because some systems will do nothing if + * we shift by the size of a long long (64 bits) all at + * once (<cough> Linux <cough>). Thus, we break one shift + * into several smaller shifts. + */ + for(c=0; c < H5F_SIZEOF_ADDR(f); c += 2) + space_needed = space_needed >> 16; + + if(space_needed != 0) + return 1; + else + return 0; +} + /*------------------------------------------------------------------------- * Function: H5MF_can_extend @@ -246,6 +375,10 @@ H5MF_can_extend(H5F_t *f, H5FD_mem_t type, haddr_t addr, hsize_t size, hsize_t e if((ret_value=H5FD_can_extend(f->shared->lf, type, addr, size, extra_requested))<0) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "unable to allocate new file memory"); + /* Make sure there is enough addressable space to satisfy the request */ + if (ret_value == TRUE) + ret_value = !H5MF_alloc_overflow(f, extra_requested); + done: FUNC_LEAVE_NOAPI(ret_value); } /* end H5MF_can_extend() */ @@ -277,6 +410,10 @@ H5MF_extend(H5F_t *f, H5FD_mem_t type, haddr_t addr, hsize_t size, hsize_t extra /* Convert old relative address to absolute address */ addr += f->shared->base_addr; + /* Make sure there is enough addressable space to satisfy the request */ + if ( H5MF_alloc_overflow(f, extra_requested) ) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "unable to allocate new file memory: out of address space"); + /* Pass the request down to the virtual file layer */ if((ret_value=H5FD_extend(f->shared->lf, type, addr, size, extra_requested))<0) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "unable to allocate new file memory"); diff --git a/src/H5MFprivate.h b/src/H5MFprivate.h index 14b2761..2a129ea 100644 --- a/src/H5MFprivate.h +++ b/src/H5MFprivate.h @@ -48,6 +48,9 @@ H5_DLL herr_t H5MF_xfree(H5F_t *f, H5FD_mem_t type, hid_t dxpl_id, haddr_t addr, hsize_t size); H5_DLL haddr_t H5MF_realloc(H5F_t *f, H5FD_mem_t type, hid_t dxpl_id, haddr_t old_addr, hsize_t old_size, hsize_t new_size); +H5_DLL herr_t H5MF_reserve(H5F_t *f, hsize_t size); +H5_DLL void H5MF_free_reserved(H5F_t *f, hsize_t size); +H5_DLL int H5MF_alloc_overflow(H5F_t *f, hsize_t size); H5_DLL htri_t H5MF_can_extend(H5F_t *f, H5FD_mem_t type, haddr_t addr, hsize_t size, hsize_t extra_requested); H5_DLL htri_t H5MF_extend(H5F_t *f, H5FD_mem_t type, haddr_t addr, hsize_t size, @@ -72,7 +72,7 @@ static herr_t H5O_remove_real(H5G_entry_t *ent, const H5O_class_t *type, int sequence, hid_t dxpl_id); static unsigned H5O_alloc(H5F_t *f, H5O_t *oh, const H5O_class_t *type, size_t size); -static unsigned H5O_alloc_extend_chunk(H5O_t *oh, unsigned chunkno, size_t size); +static unsigned H5O_alloc_extend_chunk(H5F_t *f, H5O_t *oh, unsigned chunkno, size_t size); static unsigned H5O_alloc_new_chunk(H5F_t *f, H5O_t *oh, size_t size); static herr_t H5O_delete_oh(H5F_t *f, hid_t dxpl_id, H5O_t *oh); static herr_t H5O_delete_mesg(H5F_t *f, hid_t dxpl_id, H5O_mesg_t *mesg); @@ -715,6 +715,10 @@ H5O_flush(H5F_t *f, hid_t dxpl_id, hbool_t destroy, haddr_t addr, H5O_t *oh) assert(cont->chunkno < oh->nchunks); assert(!H5F_addr_defined(oh->chunk[cont->chunkno].addr)); cont->size = oh->chunk[cont->chunkno].size; + + /* Free the space we'd previously reserved to hold this chunk */ + H5MF_free_reserved(f, cont->size); + if (HADDR_UNDEF==(cont->addr=H5MF_alloc(f, H5FD_MEM_OHDR, dxpl_id, (hsize_t)cont->size))) HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, FAIL, "unable to allocate space for object header data"); @@ -2536,6 +2540,10 @@ done: * that message will be extended with the chunk. Otherwise a * new null message is created. * + * F is the file into which the new chunk will be written. It is + * included to ensure that there is enough space to extend this + * chunk. + * * Return: Success: Message index for null message which * is large enough to hold SIZE bytes. * @@ -2554,7 +2562,7 @@ done: *------------------------------------------------------------------------- */ static unsigned -H5O_alloc_extend_chunk(H5O_t *oh, unsigned chunkno, size_t size) +H5O_alloc_extend_chunk(H5F_t *f, H5O_t *oh, unsigned chunkno, size_t size) { unsigned u; unsigned idx; @@ -2582,6 +2590,11 @@ H5O_alloc_extend_chunk(H5O_t *oh, unsigned chunkno, size_t size) delta = MAX (H5O_MIN_SIZE, aligned_size - oh->mesg[idx].raw_size); assert (delta=H5O_ALIGN (delta)); + + /* Reserve space in the file to hold the increased chunk size */ + if( H5MF_reserve(f, delta) < 0 ) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, UFAIL, "unable to reserve space for chunk"); + oh->mesg[idx].dirty = TRUE; oh->mesg[idx].raw_size += delta; @@ -2609,6 +2622,13 @@ H5O_alloc_extend_chunk(H5O_t *oh, unsigned chunkno, size_t size) } /* end if */ } + /* Reserve space in the file */ + delta = MAX(H5O_MIN_SIZE, aligned_size+H5O_SIZEOF_MSGHDR(f)); + delta = H5O_ALIGN(delta); + + if( H5MF_reserve(f, delta) < 0 ) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, UFAIL, "unable to reserve space in file"); + /* create a new null message */ if (oh->nmesgs >= oh->alloc_nmesgs) { unsigned na = oh->alloc_nmesgs + H5O_NMESGS; @@ -2619,8 +2639,6 @@ H5O_alloc_extend_chunk(H5O_t *oh, unsigned chunkno, size_t size) oh->alloc_nmesgs = na; oh->mesg = x; } - delta = MAX(H5O_MIN_SIZE, aligned_size+H5O_SIZEOF_MSGHDR(f)); - delta = H5O_ALIGN(delta); idx = oh->nmesgs++; oh->mesg[idx].type = H5O_NULL; oh->mesg[idx].dirty = TRUE; @@ -2749,6 +2767,10 @@ H5O_alloc_new_chunk(H5F_t *f, H5O_t *oh, size_t size) size = MAX(H5O_MIN_SIZE, size + H5O_SIZEOF_MSGHDR(f)); assert (size == H5O_ALIGN (size)); + /* Reserve space in the file to hold the new chunk */ + if( H5MF_reserve(f, size) < 0 ) + HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, UFAIL, "unable to reserve space in file for new chunk"); + /* * Create the new chunk without giving it a file address. */ @@ -2913,7 +2935,7 @@ H5O_alloc(H5F_t *f, H5O_t *oh, const H5O_class_t *type, size_t size) * since we can just increase the size of that chunk. */ for (chunkno = 0; chunkno < oh->nchunks; chunkno++) { - if ((idx = H5O_alloc_extend_chunk(oh, chunkno, size)) != UFAIL) { + if ((idx = H5O_alloc_extend_chunk(f, oh, chunkno, size)) != UFAIL) { break; } H5E_clear(); diff --git a/src/H5Tconv.c b/src/H5Tconv.c index ece4341..8f3e77c 100644 --- a/src/H5Tconv.c +++ b/src/H5Tconv.c @@ -2487,10 +2487,6 @@ H5T_conv_vlen(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, nelmts-=safe; } /* end while */ - /* Reset the conversion buffer pointer, so it doesn't get freed */ - if(write_to_file && noop_conv) - conv_buf=NULL; - /* Release the temporary datatype IDs used */ if (tsrc_id >= 0) H5I_dec_ref(tsrc_id); @@ -2503,6 +2499,10 @@ H5T_conv_vlen(hid_t src_id, hid_t dst_id, H5T_cdata_t *cdata, size_t nelmts, } /* end switch */ done: + /* If the conversion buffer doesn't need to be freed, reset its pointer */ + if(write_to_file && noop_conv) + conv_buf = NULL; + /* Release the conversion buffer (always allocated, except on errors) */ if(conv_buf!=NULL) H5FL_BLK_FREE(vlen_seq,conv_buf); diff --git a/test/Makefile.in b/test/Makefile.in index 358dd25..6b7ff37 100644 --- a/test/Makefile.in +++ b/test/Makefile.in @@ -28,7 +28,7 @@ TEST_PROGS=testhdf5 lheap ohdr stab gheap hyperslab istore bittests dtypes dsets cmpd_dset extend external links unlink big mtime fillval mount \ flush1 flush2 enum gass_write gass_read gass_append set_extent \ srb_write srb_append srb_read ttsafe stream_test getname file_handle \ - ntypes dangle filename + ntypes dangle filename reserved TIMINGS=testmeta @@ -79,7 +79,8 @@ TEST_SRC=big.c bittests.c cmpd_dset.c dsets.c dtypes.c extend.c \ unlink.c enum.c ttsafe.c ttsafe_dcreate.c ttsafe_error.c \ ttsafe_cancel.c ttsafe_acreate.c gass_write.c gass_read.c \ gass_append.c srb_read.c srb_write.c srb_append.c stream_test.c \ - set_extent.c getname.c file_handle.c ntypes.c dangle.c filename.c + set_extent.c getname.c file_handle.c ntypes.c dangle.c filename.c \ + reserved.c TEST_OBJ=$(TEST_SRC:.c=.lo) @@ -223,4 +224,7 @@ dangle: dangle.lo filename: filename.lo @$(LT_LINK_EXE) $(CFLAGS) -o $@ filename.lo $(LIB) $(LIBHDF5) $(LDFLAGS) $(LIBS) +reserved: reserved.lo + @$(LT_LINK_EXE) $(CFLAGS) -o $@ reserved.lo $(LIB) $(LIBHDF5) $(LDFLAGS) $(LIBS) + @CONCLUDE@ diff --git a/test/lheap.c b/test/lheap.c index 9ed2b60..04ca275 100644 --- a/test/lheap.c +++ b/test/lheap.c @@ -129,8 +129,9 @@ main(void) if (H5Fclose(file)<0) goto error; PASSED(); - puts("All local heap tests passed."); h5_cleanup(FILENAME, fapl); + + puts("All local heap tests passed."); return 0; error: diff --git a/test/ohdr.c b/test/ohdr.c index 44d07d8..877ae19 100644 --- a/test/ohdr.c +++ b/test/ohdr.c @@ -309,8 +309,9 @@ main(void) } PASSED(); - puts("All object header tests passed."); h5_cleanup(FILENAME, fapl); + + puts("All object header tests passed."); return 0; error: diff --git a/test/reserved.c b/test/reserved.c new file mode 100755 index 0000000..99cc53d --- /dev/null +++ b/test/reserved.c @@ -0,0 +1,399 @@ +/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * + * Copyright by the Board of Trustees of the University of Illinois. * + * All rights reserved. * + * * + * This file is part of HDF5. The full HDF5 copyright notice, including * + * terms governing use, modification, and redistribution, is contained in * + * the files COPYING and Copyright.html. COPYING can be found at the root * + * of the source code distribution tree; Copyright.html can be found at the * + * root level of an installed copy of the electronic HDF5 document set and * + * is linked from the top-level documents page. It can also be found at * + * http://hdf.ncsa.uiuc.edu/HDF5/doc/Copyright.html. If you do not have * + * access to either file, you may request a copy from hdfhelp@ncsa.uiuc.edu. * + * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */ + +#include "h5test.h" + +/*------------------------------------------------------------------------- + * Function: rsrv_heap + * + * Purpose: Ensure that heaps reserve file address space. + * This function does this by creating datasets up to and past + * the limit of the file, then ensuring that an error (not an + * assert) was generated and that the file is readable. + * + * Return: Success: 0 + * + * Failure: 1 + * + * Programmer: James Laird + * Nat Furrer + * Friday, May 28, 2004 + * + * Modifications: + * + *------------------------------------------------------------------------- + */ +herr_t rsrv_heap() +{ + hid_t file_id, dataset_id, dataspace_id; + hid_t pfc_id; + hsize_t dims[1] = {1}; + herr_t ret_value = 0; + char dset_name[10]; + int i; + + TESTING("Reserving file space for heap"); + + /* Create a new file. */ + pfc_id = H5Pcreate(H5P_FILE_CREATE); + H5Pset_userblock(pfc_id, (int)0); + H5Pset_sym_k(pfc_id, (int)1, (int)1); + H5Pset_istore_k(pfc_id, (int)1); + /* Set file address sizes to be very small. */ + H5Pset_sizes(pfc_id, (size_t)2,(size_t)2); + + file_id = H5Fcreate("rsrv_heap", H5F_ACC_TRUNC, pfc_id, H5P_DEFAULT); + + /* Write datasets until the file is full, at which point HDF5 + * should throw an error. + */ + for (i = 0; i < 200; i++) + { + H5E_BEGIN_TRY { + dataspace_id = H5Screate_simple(1, dims, dims); + } H5E_END_TRY + + sprintf(dset_name, "Dset %d", i); + + H5E_BEGIN_TRY { + dataset_id = H5Dcreate(file_id, dset_name, H5T_NATIVE_INT, dataspace_id, H5P_DEFAULT); + } H5E_END_TRY + + if(dataset_id < 0) + break; + + H5E_BEGIN_TRY { + H5Dwrite(dataset_id, H5T_NATIVE_INT, dataspace_id, dataspace_id, H5P_DEFAULT, &i); + } H5E_END_TRY + + H5Dclose(dataset_id); + H5Sclose(dataspace_id); + } + + /* Close the file, property lists, and library */ + H5Fclose(file_id); + H5Pclose (pfc_id); + H5close(); + + /* The loop should have broken before completing--the file should not have had + * enough address space to hold 200 datasets (or this test needs to be updated!). + */ + if(i == 200) + { + ret_value = -1; + H5_FAILED(); + } + + /* Re-open the library and try to read a dataset from the file we created */ + H5open(); + + sprintf(dset_name, "Dset %d", i - 2); + + file_id = H5Fopen("rsrv_heap", H5F_ACC_RDONLY, H5P_DEFAULT); + + dataset_id = H5Dopen(file_id, dset_name); + + /* If we can read a dataset from the file, the file has been flushed to disk + * (if the heap or object headers weren't flushed, the file would be empty). + */ + if(dataset_id == H5I_BADID) + { + ret_value = -1; + H5_FAILED(); + } + + H5Dclose(dataset_id); + H5Fclose(file_id); + remove("rsrv_heap"); + + if(ret_value == 0) + { + PASSED(); + } + + return ret_value; +} + +/*------------------------------------------------------------------------- + * Function: rsrv_ohdr + * + * Purpose: Ensure that object headers reserve file address space. + * This function does this by creating attributes of a dataset + * past the limit of the file, then ensuring that an error (not + * an assert) was generated and that the file is readable. + * + * Return: Success: 0 + * + * Failure: 1 + * + * Programmer: James Laird + * Nat Furrer + * Friday, May 28, 2004 + * + * Modifications: + * + *------------------------------------------------------------------------- + */ +herr_t rsrv_ohdr() +{ + hid_t file_id, dataset_id, dataspace_id; + hid_t pfc_id, aid, attr_id; + hsize_t dims[2]; + herr_t status, ret_value = 0; + int attrval[4][6]; + char *basenm = "attr"; + char attrname[20], nm_suf[10]; + int i; + + TESTING("Reserving file space for object headers"); + + /* Create a new file using default properties. */ + pfc_id = H5Pcreate(H5P_FILE_CREATE); + H5Pset_userblock(pfc_id, (int)0); + H5Pset_sym_k(pfc_id, (int)1, (int)1); + H5Pset_istore_k(pfc_id, (int)1); + H5Pset_sizes(pfc_id, (size_t)2,(size_t)2); + + file_id = H5Fcreate("rsrv_ohdr", H5F_ACC_TRUNC, pfc_id, H5P_DEFAULT); + + /* Create the data space for the dataset. */ + dims[0] = 4; + dims[1] = 6; + dataspace_id = H5Screate_simple(2, dims, NULL); + + /* Create the dataset. */ + dataset_id = H5Dcreate(file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); + + for(i=0; i<6; i++) + { + attrval[0][i] = 0; + attrval[1][i] = 1; + attrval[2][i] = 2; + attrval[3][i] = 3; + } + + for (i = 0; i< 2000; i++) + { + strcpy(attrname, basenm); + sprintf(nm_suf, "%d", i); + strcat(attrname, nm_suf); + H5E_BEGIN_TRY{ + aid = H5Screate_simple(2, dims, NULL); + attr_id = H5Acreate (dataset_id, attrname, H5T_STD_I32BE, aid, H5P_DEFAULT); + status = H5Awrite(attr_id, H5T_NATIVE_INT, attrval); + status = H5Aclose(attr_id); + }H5E_END_TRY + + if(status < 0) + break; + } + + /* End access to the dataset and dataspace and release resources. */ + H5Dclose(dataset_id); + H5Pclose (pfc_id); + H5Sclose(dataspace_id); + + /* Close the file and the library. */ + H5Fclose(file_id); + H5close(); + + /* The loop should have broken before completing--the file should not have had + * enough address space to hold 2000 attributes (or this test needs to be updated!). + */ + if(i == 2000) + { + ret_value = -1; + H5_FAILED(); + } + + /* Re-open the library and try to read a dataset from the file we created */ + H5open(); + + file_id = H5Fopen("rsrv_ohdr", H5F_ACC_RDONLY, H5P_DEFAULT); + + dataset_id = H5Dopen(file_id, "/dset"); + + /* If we can read the dataset from the file, the file has been flushed to disk + * (if the heap or object headers weren't flushed, the file would be empty). + */ + if(dataset_id == H5I_BADID) + { + H5_FAILED(); + ret_value = -1; + } + + if(ret_value == 0) + PASSED(); + + H5Dclose(dataset_id); + H5Fclose(file_id); + + remove("rsrv_ohdr"); + + return ret_value; +} + +/*------------------------------------------------------------------------- + * Function: rsrv_vlen + * + * Purpose: Ensure that variable length datatypes properly ensure that + * enough file address space exists before writing. + * This function does this by creating a dataset containing + * variable length data past the limit of the file, then + * ensuring that an error (not an assert) was generated and + * that the file is readable. + * + * Return: Success: 0 + * + * Failure: 1 + * + * Programmer: James Laird + * Nat Furrer + * Thursday, July 1, 2004 + * + * Modifications: + * + *------------------------------------------------------------------------- + */ +herr_t rsrv_vlen() +{ + hid_t file_id, dataset_id, dataspace_id, type_id; + hid_t pfc_id, mem_space_id; + hsize_t dims[1], start[1], count[1]; + herr_t status, ret_value = 0; + int i; + int write_buf[20]; + hvl_t vlen_data; + + TESTING("Reserved space with variable length data"); + + /* Create a new file using default properties. */ + pfc_id = H5Pcreate(H5P_FILE_CREATE); + H5Pset_userblock(pfc_id, (int)0); + H5Pset_sym_k(pfc_id, (int)1, (int)1); + H5Pset_istore_k(pfc_id, (int)1); + H5Pset_sizes(pfc_id, (size_t)2,(size_t)2); + + file_id = H5Fcreate("rsrv_vlen", H5F_ACC_TRUNC, pfc_id, H5P_DEFAULT); + + /* Create the data space for the dataset. */ + dims[0] = 2000; + dataspace_id = H5Screate_simple(1, dims, NULL); + + /* Create a variable length type */ + type_id = H5Tvlen_create(H5T_NATIVE_INT); + + /* Create the dataset. */ + dataset_id = H5Dcreate(file_id, "/dset", type_id, dataspace_id, H5P_DEFAULT); + + /* Create some data to write */ + for (i = 0; i < 20; i++) + write_buf[i] = i+1; + vlen_data.p = write_buf; + + /* Create a memory dataspace for writing */ + dims[0] = 1; + mem_space_id = H5Screate_simple(1, dims, NULL); + + /* Create a selection to write to */ + start[0] = 0; + count[0] = 1; + H5Sselect_hyperslab(dataspace_id, H5S_SELECT_SET, start, NULL, count, NULL); + + for (i = 0; i< 2000; i++) + { + vlen_data.len = (i%20) + 1; + + dims[0] = i; + H5Soffset_simple(dataspace_id, dims); + + H5E_BEGIN_TRY + status = H5Dwrite(dataset_id, type_id, mem_space_id, dataspace_id, H5P_DEFAULT, &vlen_data); + H5E_END_TRY + + if(status < 0) + break; + } + + /* End access to the dataset and dataspace and release resources. */ + H5Dclose(dataset_id); + H5Pclose (pfc_id); + H5Sclose(dataspace_id); + H5Tclose(type_id); + H5Sclose(mem_space_id); + + /* Close the file and the library. */ + H5Fclose(file_id); + H5close(); + + /* The loop should have broken before completing--the file should not have had + * enough address space to hold 2000 attributes (or this test needs to be updated!). + */ + if(i == 2000) + { + ret_value = -1; + H5_FAILED(); + } + + /* Re-open the library and try to read a dataset from the file we created */ + H5open(); + + file_id = H5Fopen("rsrv_vlen", H5F_ACC_RDONLY, H5P_DEFAULT); + + dataset_id = H5Dopen(file_id, "/dset"); + + /* If we can read the dataset from the file, the file has been flushed to disk + * (if the heap or object headers weren't flushed, the file would be empty). + */ + if(dataset_id == H5I_BADID) + { + H5_FAILED(); + ret_value = -1; + } + + if(ret_value == 0) + PASSED(); + + H5Dclose(dataset_id); + H5Fclose(file_id); + + remove("rsrv_vlen"); + + return ret_value; +} + +/*------------------------------------------------------------------------- + * Function: main + * + * Purpose: + * + * Return: Success: + * + * Failure: + * + * Programmer: Nat Furrer and James Laird + * Thursday, July 1, 2004 + * + * Modifications: + * + *------------------------------------------------------------------------- + */ +int +main(void) +{ + rsrv_ohdr(); + rsrv_heap(); + rsrv_vlen(); +} + |