1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
|
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<title>References</title>
</head>
<body bgcolor="#FFFFFF">
<hr>
<center>
<table border=0 width=98%>
<tr><td valign=top align=left>
<a href="H5.intro.html">Introduction to HDF5</a> <br>
<a href="RM_H5Front.html">HDF5 Reference Manual</a> <br>
<a href="index.html">Other HDF5 documents and links</a> <br>
<!--
<a href="Glossary.html">Glossary</a><br>
-->
</td>
<td valign=top align=right>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:
<a href="Files.html">Files</a>
<br>
<a href="Datasets.html">Datasets</a>
<a href="Datatypes.html">Data Types</a>
<a href="Dataspaces.html">Dataspaces</a>
<a href="Groups.html">Groups</a>
References
<br>
<a href="Attributes.html">Attributes</a>
<a href="Properties.html">Property Lists</a>
<a href="Errors.html">Error Handling</a>
<a href="Filters.html">Filters</a>
<a href="Caching.html">Caching</a>
<br>
<a href="Chunking.html">Chunking</a>
<a href="Debugging.html">Debugging</a>
<a href="Environment.html">Environment</a>
<a href="ddl.html">DDL</a>
<a href="Ragged.html">Ragged Arrays</a>
<!--
<hr>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:
<a href="Attributes.html">H5A</a>
<a href="Datasets.html">H5D</a>
<a href="Errors.html">H5E</a>
<a href="Files.html">H5F</a>
<a href="Groups.html">H5G</a>
<a href="Properties.html">H5P</a>
<a href="References.html">H5R & H5I</a>
<a href="Ragged.html">H5RA</a>
<a href="Dataspaces.html">H5S</a>
<a href="Datatypes.html">H5T</a>
<a href="Filters.html">H5Z</a>
<a href="Caching.html">Caching</a>
<a href="Chunking.html">Chunking</a>
<a href="Debugging.html">Debugging</a>
<a href="Environment.html">Environment</a>
<a href="ddl.html">DDL</a>
-->
</td></tr>
</table>
</center>
<hr>
<h1>The Reference Interface (H5R) and <br> the Indentifier Interface (H5I)</h1>
<h2>1. Introduction</h2>
This document discusses the kinds of references implemented
(and planned) in HDF5 and the functions implemented (and planned)
to support them.
<h2>2. References</h2>
This section contains an overview of the kinds of references
implemented, or planned for implementation, in HDF5.
<p>
<dl>
<dt>Object reference
<dd>Reference to an entire object in the current HDF5 file.
<br>
<em>The only kind of reference currently implemented.</em>
<p>
An object reference points to an entire object in the
current HDF5 file by storing the relative file address
(OID) of the object header for the object pointed to.
The relative file address of an object header is
constant for the life of the object.
An object reference is of a fixed size in the file.
<p>
<dt>Dataset region reference
<dd>Reference to a specific dataset region.
<br>
<em>Not yet implemented.</em>
<p>
A dataset region reference points to a region of a
dataset in the current HDF5 file by storing the OID
of the dataset and the global heap offset of the
region referenced. The region referenced is located
by retrieving the coordinates of the areas in the
region from the global heap. A dataset region
reference is of a variable size in the file.
<p>
<dt>Internal dataset region reference
<dd>Reference to a region within the current dataset.
<br>
<em>Not yet implemented.</em>
<p>
An internal dataset region reference points to a
region of the current dataset by storing the
coordinates of the region. An internal dataset
region reference is of a fixed size in the file.
</dl>
<b>Note:</b> All references are treated as soft links for the
purposes of reference counting. The library does not keep track of
reference links and they may dangle if the object they refer to
is deleted, moved, or not yet available.
<h2>3. Reference Types</h2>
This section lists valid HDF5 reference types for use in the
H5R functions.
<center>
<table>
<tr><th align=left>Reference Type</th><th align=left>Value </th><th align=left>Description</th></tr>
<tr><td><code>H5R_BADTYPE</code></td>
<td align=right><code>-1 </code></td>
<td>Invalid reference type</td></tr>
<tr><td><code>H5R_OBJECT</code></td>
<td align=right><code>0 </code></td>
<td>Object reference</td></tr>
<tr><td><code>H5R_DATASET_REGION</code></td>
<td align=right><code>1 </code></td>
<td>Dataset region reference</td></tr>
<tr><td><code>H5R_INTERNAL</code></td>
<td align=right><code>2 </code></td>
<td>Internal reference</td></tr>
</table>
</center>
<h2>4. Functions</h2>
Four functions, three in the H5R interface and one in the
H5I interface, have been implemented to support references.
The H5I function is also useful outside the context of references.
<p>
<dl>
<dt><em>herr_t</em> <code>H5Rcreate(</code><em>void *</em><code>reference,</code>
<em>hid_t</em> <code>loc_id,</code>
<em>const char *</em><code>name,</code>
<em>H5R_type_t</em> <code>type,</code>
<em>hid_t</em> <code>space_id)</code>
<dd><code>H5Rcreate</code> creates an object which is a
particular type of reference (specified with the
<code>type</code> parameter) to some file object and/or
location specified with the <code>space_id</code> parameter.
For dataset region references, the selection specified
in the dataspace is the portion of the dataset which
will be referred to.
<p>
<dt><em>hid_t</em> <code>H5Rdereference(</code><em>hid_t</em> <code>dset,</code>
<em>H5R_type_t</em> <code>rtype,</code>
<em>void *</em><code>reference)</code>
<dd><code>H5Rdereference</code> opens the object referenced
and returns an identifier for that object.
The parameter <code>reference</code> specifies a reference of
type <code>rtype</code> that is stored in the dataset
<code>dset</code>.
<p>
<dt><em>H5S_t</em> <code>H5Rget_region(</code><em>H5D_t</em> <code>dataset,</code>
<em>H5R_type_t</em> <code>type,</code>
<em>void *</em><code>reference)</code>
<dd><code>H5Rget_region</code> creates a copy of dataspace of
the dataset that is pointed to and defines a selection in
the copy which is the location (or region) pointed to.
The parameter <code>ref</code> specifies a reference of
type <code>rtype</code> that is stored in the dataset
<code>dset</code>.
<p>
<dt><em>H5I_type_t</em> <code>H5Iget_type(</code><em>hid_t</em> <code>id)</code>
<dd>Returns the type of object referred to by the
identifier <code>id</code>. Valid return values appear
in the following list:
<center>
<table>
<tr><td><code>H5I_BADID</code></td>
<td>Invalid ID</td></tr>
<tr><td><code>H5I_FILE</code></td>
<td>File objects</td></tr>
<tr><td><code>H5I_GROUP</code></td>
<td>Group objects</td></tr>
<tr><td><code>H5I_DATATYPE</code></td>
<td>Data type objects</td></tr>
<tr><td><code>H5I_DATASPACE</code></td>
<td>Dataspace objects</td></tr>
<tr><td><code>H5I_DATASET</code></td>
<td>Dataset objects</td></tr>
<tr><td><code>H5I_ATTR</code></td>
<td>Attribute objects</td></tr>
</table>
</center>
<p>
This function was inspired by the need of users to figure
out which type of object closing function
(<code>H5Dclose</code>, <code>H5Gclose</code>, etc.)
to call after a call to <code>H5Ddereference</code>,
but it is also of general use.
<p>
</dl>
<h2>5. Examples</h2>
<b>Object Reference Writing Example</b>
<br>
Create a dataset which has links to other datasets as part
of its raw data and write the dataset to the file.
<p>
<pre>
{
hid_t file1;
hid_t dataset1;
hid_t datatype, dataspace;
char buf[128];
hobj_ref_t link;
hobj_ref_t data[10][10];
int rank;
size_t dimsf[2];
int i, j;
/* Open the file */
file1=H5Fopen("example.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Describe the size of the array and create the data space */
rank=2;
dimsf[0] = 10;
dimsf[1] = 10;
dataspace = H5Screate_simple(rank, dimsf, NULL);
/* Define datatype */
datatype = H5Tcopy(H5T_STD_REF_OBJ);
/* Create a dataset */
dataset1=H5Dcreate(file1,"Dataset One",datatype,dataspace,H5P_DEFAULT);
/* Construct array of OIDs for other datasets in the file */
/* somewhat hokey and artificial, but demonstrates the point */
for(i=0; i<10; i++)
for(j=0; j<10; i++)
{
sprintf(buf,"/Group/Linked Set %d-%d",i,j);
if(H5Rcreate(&link,file1,buf,H5R_OBJECT,-1)>0)
data[i][j]=link;
} /* end for */
/* Write the data to the dataset using default transfer properties. */
H5Dwrite(dataset, H5T_STD_REF_OBJ, H5S_ALL, H5S_ALL, H5P_DEFAULT, data);
/* Close everything */
H5Sclose(dataspace);
H5Tclose(datatype);
H5Dclose(dataset1);
H5Fclose(file1);
}
</pre>
<b>Object Reference Reading Example</b>
<br>
Open a dataset which has links to other datasets as part of
its raw data and read in those links.
<p>
<pre>
{
hid_t file1;
hid_t dataset1, tmp_dset;
href_t data[10][10];
int i, j;
/* Open the file */
file1=H5Fopen("example.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Open the dataset */
dataset1=H5Dopen(file1,"Dataset One",H5P_DEFAULT);
/*
* Read the data to the dataset using default transfer properties.
* (we are assuming the dataset is the same and not querying the
* dimensions, etc.)
*/
H5Dread(dataset, H5T_STD_REF_OBJ, H5S_ALL, H5S_ALL, H5P_DEFAULT, data);
/* Analyze array of OIDs of linked datasets in the file */
/* somewhat hokey and artificial, but demonstrates the point */
for(i=0; i<10; i++)
for(j=0; j<10; i++)
{
if((tmp_dset=H5Rdereference(dataset, H5T_STD_REF_OBJ, data[i][j]))>0)
{
<perform operations on linked dataset>
} /* end if */
H5Dclose(tmp_dset);
} /* end for */
/* Close everything */
H5Dclose(dataset1);
H5Fclose(file1);
}
</pre>
<b>Dataset Region Reference Writing Example</b>
<br>
Create a dataset which has links to other dataset regions
(single elements in this case) as part of its raw data and
write the dataset to the file.
<p>
<pre>
{
hid_t file1;
hid_t dataset1, dataset2;
hid_t datatype, dataspace1, dataspace2;
char buf[128];
href_t link;
href_t data[10][10]; /* HDF5 reference type */
int rank;
size_t dimsf[2];
hssize_t start[3],count[3];
int i, j;
/* Open the file */
file1=H5Fopen("example.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Describe the size of the array and create the data space */
rank=2;
dimsf[0] = 10;
dimsf[1] = 10;
dataspace1 = H5Screate_simple(rank, dimsf, NULL);
/* Define Dataset Region Reference datatype */
datatype = H5Tcopy(H5T_STD_REF_DATAREG);
/* Create a dataset */
dataset1=H5Dcreate(file1,"Dataset One",datatype,dataspace1,H5P_DEFAULT);
/* Construct array of OIDs for other datasets in the file */
/* (somewhat artificial, but demonstrates the point) */
for(i=0; i<10; i++)
for(j=0; j<10; i++)
{
sprintf(buf,"/Group/Linked Set %d-%d",i,j);
/* Get the dataspace for the object to point to */
dataset2=H5Dopen(file1,buf,H5P_DEFAULT);
dataspace2=H5Dget_space(dataspace2);
/* Select the region to point to */
/* (could be different region for each pointer) */
start[0]=5; start[1]=4; start[2]=3;
count[0]=2; count[1]=4; count[2]=1;
H5Sselect_hyperslab(dataspace2,H5S_SELECT_SET,start,NULL,count,NULL);
if(H5Rcreate(&link,file1,buf,H5R_REF_DATAREG,dataspace2)>0)
/* Store the reference */
data[i][j]=link;
H5Sclose(dataspace2);
H5Dclose(dataspace2);
} /* end for */
/* Write the data to the dataset using default transfer properties. */
H5Dwrite(dataset, H5T_STD_REF_DATAREG, H5S_ALL, H5S_ALL, H5P_DEFAULT, data);
/* Close everything */
H5Sclose(dataspace);
H5Tclose(datatype);
H5Dclose(dataset1);
H5Fclose(file1);
}
</pre>
<b>Dataset Region Reference Reading Example</b>
<br>
Open a dataset which has links to other datasets regions
(single elements in this case) as part of its raw data and
read in those links.
<p>
<pre>
{
hid_t file1;
hid_t dataset1, tmp_dset;
hid_t dataspace;
href_t data[10][10]; /* HDF5 reference type */
int i, j;
/* Open the file */
file1=H5Fopen("example.h5", H5F_ACC_RDWR, H5P_DEFAULT);
/* Open the dataset */
dataset1=H5Dopen(file1,"Dataset One",H5P_DEFAULT);
/*
* Read the data to the dataset using default transfer properties.
* (we are assuming the dataset is the same and not querying the
* dimensions, etc.)
*/
H5Dread(dataset, H5T_STD_REF_DATAREG, H5S_ALL, H5S_ALL, H5P_DEFAULT, data);
/* Analyze array of OIDs of linked datasets in the file */
/* (somewhat artificial, but demonstrates the point) */
for(i=0; i<10; i++)
for(j=0; j<10; i++)
{
if((tmp_dset=H5Rdereference(dataset, H5D_STD_REF_DATAREG,data[i][j]))>0)
{
/* Get the dataspace with the pointed to region selected */
dataspace=H5Rget_space(data[i][j]);
<perform operation on linked dataset region>
H5Sclose(dataspace);
} /* end if */
H5Dclose(tmp_dset);
} /* end for */
/* Close everything */
H5Dclose(dataset1);
H5Fclose(file1);
}
</pre>
<!--
<h1>Material to Be Omitted!!!</h1>
Additional material above will also need to be deleted or
commented out.
<p>
<h2>"Kinds of Reference" Information</h2>
<dl>
<dt>Dataset Offset Reference
<dd>Reference to a specific byte sequence in a dataset (3)
<dt>Disk Offset Reference
<dd>Reference to a specific byte sequence in a file (3)
<dt>External Object Reference
<dd>Reference to an entire object in another HDF5 file (3)
<dt>External Dataset Region Reference
<dd>Reference an a specific dataset region in another HDF5 file (3)
<dt>External Dataset Offset Reference
<dd>Reference to a specific byte sequence in a dataset in
another HDF5 file (3)
<dt>External Disk Offset Reference
<dd>Reference to a specific byte sequence in another HDF5 file (3)
<dt>Generic Reference
<dd>A reference which may be any of the types defined above. (3)
</dl>
Notes:
<ul>
<li>1 - Planned for in final release.
<li>2 - Most useful reference types to tackle next
<li>3 - Optional reference types.
</ul>
<h2>Comments</h2>
<pre>
Reference types are atomic types and may be included as fields in compound
data types.
There are (at least) three levels of reference strength:
Weak - We allow the user to store any type of reference in an array
of references. (u.e., the array of references in the example above
could be a mix of Object, Dataset Region and Internal references)
Medium - We force the user to stick with a particular type of
reference within a dataset, but the datasets pointed to (with
Object and Dataset Region references) may be of any data type
or dataspace.
Strong - We force the user to stick with a particular type of
reference and Object and Dataset Region references must point to
datasets with the same data type.
Extra Strong - We force the user to stick with a particular type of
reference and Object and Dataset Region references must point to
datasets with the same data type _and_ dataspace.
The library is currently implemented with "medium" strength references.
</pre>
<h2>Reference Type</h2>
<center>
<table>
<tr><td><code>H5R_MAXTYPE</code></td>
<td align=right><code>3 </code></td>
<td>Highest type in group (invalid as true type)</td></tr>
</table>
</center>
<h2>Information Regarding Specific Kinds of References</h2>
<dl>
<dt>Dataset Offset Reference
<dd>Points to a sequence of bytes within a dataset in
the current HDF5 file by storing the OID of the dataset and the byte
length and offset of the sequence within the dataset. The offset is
the logical byte offset within the dataset, meaning that the data is
de-compressed before returning the sequence of bytes requested. No
interpretation of the data at that location is provided. However,
if the dataset is extendible and the size of the dimensions are changed,
the element(s) that the sequence is located within may vary. Fixed size
in file.
<dt>Disk Offset Reference
<dd>Points to a sequence of bytes in the current
HDF5 file by storing the byte length and offset of the sequence within
the file, relative to the boot-block (as are all the other high-level
addresses used in the file). The offset is the absolute byte offset
within the file, no interpretation of the data at that location is
provided. Fixed size in file.
<dt>External Object Reference
<dd>Points to an entire object in another HDF5 file by
storing a global heap offset which points to the URL of the external
file and the OID of the object pointed to. Variable size in file.
<dt>External Dataset Region Reference
<dd>Points to a region of a dataset in
another HDF5 file by storing a global heap offset which points to the
URL of the external file, OID of the dataset and the coordinates of the
region. Variable size in file.
<dt>External Dataset Offset Reference
<dd>Points to a sequence of bytes within a
dataset in another HDF5 file by storing a global heap offset which
points to the URL of the external file, the OID of the dataset and the
byte length and offset of the sequence within the dataset. The offset
is the logical byte offset within the dataset, meaning that the data is
de-compressed before returning the sequence of bytes requested.
However, if the dataset is not stored contiguously and the size of the
dimensions are changed, the element(s) that the sequence is located
within may vary. Variable size in file.
<dt>External Disk Offset Reference
<dd>Points to a sequence of bytes in another
HDF5 file by storing a global heap reference which points to the URL
of the external file and the byte length and offset of the sequence
within the file. The offset is the absolute byte offset within the
file, no interpretation of the data at that location is provided.
Variable size in file.
<dt>Generic Reference
<dd>A reference which may contain any of the other
references defined above. (Mostly useful for implementing "weak"
strength pointers within the medium strength model we are using)
Variable size in file.
</dl>
<h2>Implementation Details</h2>
<dl>
<dt>File Storage
<dd>In order to efficiently index an array, each element must be the
same size when stored in the dataset on disk. Fixed-sized references
will be stored directly in the dataset array on disk; variable-sized
references will have a fixed-size head offset stored in the array on
disk with a file heap used to store the actual variable-sized
information stored in the heap.
<dt>Memory Storage
<dd>Each <code>href_t</code> object in memory is a struct containing
a pointer type and union of information required for each
pointer type.
Information in this structure is not designed for users to view.
Non-C APIs may have to mangle this structure in some way, in order
to provide users with access to references in a language-appropriate
way.
</dl>
-->
<hr>
<center>
<table border=0 width=98%>
<tr><td valign=top align=left>
<a href="H5.intro.html">Introduction to HDF5</a> <br>
<a href="RM_H5Front.html">HDF5 Reference Manual</a> <br>
<a href="index.html">Other HDF5 documents and links</a> <br>
<!--
<a href="Glossary.html">Glossary</a><br>
-->
</td>
<td valign=top align=right>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:
<a href="Files.html">Files</a>
<br>
<a href="Datasets.html">Datasets</a>
<a href="Datatypes.html">Data Types</a>
<a href="Dataspaces.html">Dataspaces</a>
<a href="Groups.html">Groups</a>
References
<br>
<a href="Attributes.html">Attributes</a>
<a href="Properties.html">Property Lists</a>
<a href="Errors.html">Error Handling</a>
<a href="Filters.html">Filters</a>
<a href="Caching.html">Caching</a>
<br>
<a href="Chunking.html">Chunking</a>
<a href="Debugging.html">Debugging</a>
<a href="Environment.html">Environment</a>
<a href="ddl.html">DDL</a>
<a href="Ragged.html">Ragged Arrays</a>
<!--
<hr>
And in this document, the
<a href="H5.user.html">HDF5 User's Guide</a>:
<a href="Attributes.html">H5A</a>
<a href="Datasets.html">H5D</a>
<a href="Errors.html">H5E</a>
<a href="Files.html">H5F</a>
<a href="Groups.html">H5G</a>
<a href="Properties.html">H5P</a>
<a href="References.html">H5R & H5I</a>
<a href="Ragged.html">H5RA</a>
<a href="Dataspaces.html">H5S</a>
<a href="Datatypes.html">H5T</a>
<a href="Filters.html">H5Z</a>
<a href="Caching.html">Caching</a>
<a href="Chunking.html">Chunking</a>
<a href="Debugging.html">Debugging</a>
<a href="Environment.html">Environment</a>
<a href="ddl.html">DDL</a>
-->
</td></tr>
</table>
</center>
<hr>
<address>
<a href="mailto:hdfhelp@ncsa.uiuc.edu">HDF Help Desk</a>
</address>
Last modified: 30 October 1998
</body>
</html>
|
