1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
|
<HTML><HEAD>
<TITLE>HDF5 Tutorial - Chunking and Extendible Datasets
</TITLE>
</HEAD>
<body bgcolor="#ffffff">
<!-- BEGIN MAIN BODY -->
<A HREF="http://www.ncsa.uiuc.edu/"><img border=0
src="http://www.ncsa.uiuc.edu/Images/NCSAhome/footerlogo.gif"
width=78 height=27 alt="NCSA"><P></A>
[ <A HREF="title.html"><I>HDF5 Tutorial Top</I></A> ]
<H1>
<BIG><BIG><BIG><FONT COLOR="#c101cd">Chunking and Extendible Datasets</FONT>
</BIG></BIG></BIG></H1>
<hr noshade size=1>
<BODY>
<H2>Contents:</H2>
<UL>
<LI><A HREF="#def">Creating an Extendible Dataset</A>
<LI>Programming Example
<UL>
<LI> <A HREF="#desc">Description</A>
<LI> <A HREF="#rem">Remarks</A>
<!--
<LI> <A HREF="#fc">File Contents</A>
<LI> <A HREF="#ddl">Dataset Definition in DDL</A>
-->
</UL>
</UL>
<HR>
<A NAME="def">
<H2>Creating an Extendible Dataset</H2>
An extendible dataset is one whose dimensions can grow. In HDF5, it is possible to define a dataset to have certain initial dimensions, then later
to increase the size of any of the initial dimensions.
<P>
HDF5 requires you to use chunking in order to define extendible datasets. Chunking makes it possible to extend datasets efficiently, without
having to reorganize storage excessively.
<P>
The following operations are required in order to write an extendible dataset:
<OL>
<LI>Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended.
<LI>Set dataset creation properties to enable chunking and create a dataset.
<LI>Extend the size of the dataset.
</OL>
<H2> Programming Example</H2>
<A NAME="desc">
<H3><U>Description</U></H3>
This example shows how to create a 3 x 3 extendible dataset, write to that
dataset, extend the dataset to 10x3, and write to the dataset again.
[<A HREF="examples/h5_extend.c">Download h5_extend.c</A>]
<PRE>
/**************************************************************
*
* This example shows how to work with extendible datasets.
* In the current version of the library a dataset MUST be
* chunked in order to be extendible.
*
* This example is derived from the h5_extend_write.c and
* h5_read_chunk.c examples that are in the "Introduction
* to HDF5".
*
*************************************************************/
#include "hdf5.h"
#define FILE "ext.h5"
#define DATASETNAME "ExtendibleArray"
#define RANK 2
int
main (void)
{
hid_t file; /* handles */
hid_t dataspace, dataset;
hid_t filespace;
hid_t cparms;
hid_t memspace;
hsize_t dims[2] = { 3, 3}; /* dataset dimensions
at creation time */
hsize_t dims1[2] = { 3, 3}; /* data1 dimensions */
hsize_t dims2[2] = { 7, 1}; /* data2 dimensions */
hsize_t maxdims[2] = {H5S_UNLIMITED, H5S_UNLIMITED};
hsize_t size[2];
hssize_t offset[2];
hsize_t i,j;
herr_t status, status_n;
int data1[3][3] = { {1, 1, 1}, /* data to write */
{1, 1, 1},
{1, 1, 1} };
int data2[7] = { 2, 2, 2, 2, 2, 2, 2};
/* Variables used in reading data back */
hsize_t chunk_dims[2] ={2, 5};
hsize_t chunk_dimsr[2];
hsize_t dimsr[2];
int data_out[10][3];
int rank, rank_chunk;
/* Create the data space with unlimited dimensions. */
dataspace = H5Screate_simple (RANK, dims, maxdims);
/* Create a new file. If file exists its contents will be overwritten. */
file = H5Fcreate (FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/* Modify dataset creation properties, i.e. enable chunking */
cparms = H5Pcreate (H5P_DATASET_CREATE);
status = H5Pset_chunk ( cparms, RANK, chunk_dims);
/* Create a new dataset within the file using cparms
creation properties. */
dataset = H5Dcreate (file, DATASETNAME, H5T_NATIVE_INT, dataspace,
cparms);
/* Extend the dataset. This call assures that dataset is 3 x 3.*/
size[0] = 3;
size[1] = 3;
status = H5Dextend (dataset, size);
/* Select a hyperslab */
filespace = H5Dget_space (dataset);
offset[0] = 0;
offset[1] = 0;
status = H5Sselect_hyperslab (filespace, H5S_SELECT_SET, offset, NULL,
dims1, NULL);
/* Write the data to the hyperslab */
status = H5Dwrite (dataset, H5T_NATIVE_INT, dataspace, filespace,
H5P_DEFAULT, data1);
/* Extend the dataset. Dataset becomes 10 x 3 */
dims[0] = dims1[0] + dims2[0];
size[0] = dims[0];
size[1] = dims[1];
status = H5Dextend (dataset, size);
/* Select a hyperslab */
filespace = H5Dget_space (dataset);
offset[0] = 3;
offset[1] = 0;
status = H5Sselect_hyperslab (filespace, H5S_SELECT_SET, offset, NULL,
dims2, NULL);
/* Define memory space */
dataspace = H5Screate_simple (RANK, dims2, NULL);
/* Write the data to the hyperslab */
status = H5Dwrite (dataset, H5T_NATIVE_INT, dataspace, filespace,
H5P_DEFAULT, data2);
/* Close resources */
status = H5Dclose (dataset);
status = H5Sclose (dataspace);
status = H5Sclose (filespace);
status = H5Fclose (file);
/****************************************************************
Read the data back
***************************************************************/
file = H5Fopen (FILE, H5F_ACC_RDONLY, H5P_DEFAULT);
dataset = H5Dopen (file, DATASETNAME);
filespace = H5Dget_space (dataset);
rank = H5Sget_simple_extent_ndims (filespace);
status_n = H5Sget_simple_extent_dims (filespace, dimsr, NULL);
cparms = H5Dget_create_plist (dataset);
if (H5D_CHUNKED == H5Pget_layout (cparms))
{
rank_chunk = H5Pget_chunk (cparms, 2, chunk_dimsr);
}
memspace = H5Screate_simple (rank,dimsr,NULL);
status = H5Dread (dataset, H5T_NATIVE_INT, memspace, filespace,
H5P_DEFAULT, data_out);
printf("\n");
printf("Dataset: \n");
for (j = 0; j < dimsr[0]; j++)
{
for (i = 0; i < dimsr[1]; i++)
printf("%d ", data_out[j][i]);
printf("\n");
}
status = H5Pclose (cparms);
status = H5Dclose (dataset);
status = H5Sclose (filespace);
status = H5Sclose (memspace);
status = H5Fclose (file);
}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
</PRE>
<A NAME="rem">
<H3><U>Remarks</U></H3>
<P>
<UL>
<LI>The function H5Pcreate creates a new property as an instance of
a property list. The signature of this function is as follows:
<PRE>
hid_t H5Pcreate ( H5P_class_t type )
</PRE>
<UL>
<LI>The parameter <I>type</I> is the type of property list to create.<BR>
The class types are: H5P_FILE_CREATE, H5P_FILE_ACCESS, H5P_DATASET_CREATE,
H5P_DATASET_XFER, and H5P_MOUNT
</UL>
<P>
<LI>The function H5Pset_chunk sets the size of the chunks used
to store a chunked layout dataset.
The signature of this function is as follows:
<PRE>
herr_t H5Pset_chunk ( hid_t plist, int ndims, const hsize_t * dim )
</PRE>
<UL>
<LI>The first parameter, <I>plist</I>, is the identifier for the property
list to query.
<LI>The second parameter, <I>ndims</I>, is the number of dimensions of
each chunk.
<LI>The third parameter, <I>dim</I>, is an array containing the size of
each chunk.
</UL>
<P>
A non-negative value is returned if successful; otherwise a negative
value is returned.
<P>
<LI>The function H5Dextend extends a dataset that has an unlimited
dimension. The signature is as follows:
<PRE>
herr_t H5Dextend ( hid_t dataset_id, const hsize_t * size )
</PRE>
<UL>
<LI>The first parameter, <I>dataset_id</I>, is the identifier of
the dataset.
<LI>The second parater, <I>size</I>, is an array containing the
new magnitude of each dimension.
</UL>
<P>
This function returns a non-negative value if successful; otherwise
it returns a negative value.
<P>
<LI>The H5Dget_create_plist function returns an identifier for a
copy of the dataset creation property list for a dataset.
<P>
<LI>The H5Pget_layout function returns the layout of the raw data for a
dataset. Valid types are H5D_COMPACT, H5D_CONTIGUOUS, and H5D_CHUNKED.
<P>
<LI>The H5Pget_chunk function retrieves the size of chunks for the
raw data of a chunked layout dataset.
The signature of this function is:
<PRE>
int H5Pget_chunk ( hid_t plist, int max_ndims, hsize_t * dims )
</PRE>
<UL>
<LI>The first parameter, <I>plist</I>, is the identifier of the
property list to query.
<LI>The second parameter, <I>max_ndims</I>, is the size of the <I>dims</I>
array.
<LI>The third parameter, <I>dims</I>, is the array to store the chunk
dimensions
</UL>
<P>
<LI>The H5Pclose function terminates access to a property list.
The signature of this function is:
<PRE>
herr_t H5Pclose ( hid_t plist )
</PRE>
where <I>plist</I> is the identifier of the property list to terminate
access to.
</UL>
<!--
<A NAME="fc">
<H3><U>File Contents</U></H3>
-->
<!-- BEGIN FOOTER INFO -->
<P><hr noshade size=1>
<font face="arial,helvetica" size="-1">
<a href="http://www.ncsa.uiuc.edu/"><img border=0
src="http://www.ncsa.uiuc.edu/Images/NCSAhome/footerlogo.gif"
width=78 height=27 alt="NCSA"><br>
The National Center for Supercomputing Applications</A><br>
<a href="http://www.uiuc.edu/">University of Illinois
at Urbana-Champaign</a><br>
<br>
<!-- <A HREF="helpdesk.mail.html"> -->
<A HREF="mailto:hdfhelp@ncsa.uiuc.edu">
hdfhelp@ncsa.uiuc.edu</A>
<BR> <H6>Last Modified: August 27, 1999</H6><BR>
<!-- modified by Barbara Jones - bljones@ncsa.uiuc.edu -->
</FONT>
<BR>
<!-- <A HREF="mailto:hdfhelp@ncsa.uiuc.edu"> -->
</BODY>
</HTML>
|
