1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
|
<HTML>
<HEAD>
<!-- This HTML file has been created by texi2html 1.51
from VFL.texi on 18 November 1999 -->
<TITLE>HDF5 Virtual File Layer</TITLE>
</HEAD>
<BODY>
<H1>HDF5</H1>
<H2>Virtual File Layer</H2>
<H2>Proposal 1999-08-11</H2>
<ADDRESS>Robb Matzke</ADDRESS>
<P>
<P><HR><P>
<H1>Table of Contents</H1>
<UL>
<LI><A NAME="TOC1" HREF="VFL.html#SEC1">Introduction</A>
<LI><A NAME="TOC2" HREF="VFL.html#SEC2">Using a File Driver</A>
<UL>
<LI><A NAME="TOC3" HREF="VFL.html#SEC3">Driver Header Files</A>
<LI><A NAME="TOC4" HREF="VFL.html#SEC4">Creating and Opening Files</A>
<LI><A NAME="TOC5" HREF="VFL.html#SEC5">Performing I/O</A>
<LI><A NAME="TOC6" HREF="VFL.html#SEC6">File Driver Interchangeability</A>
</UL>
<LI><A NAME="TOC7" HREF="VFL.html#SEC7">Implementation of a Driver</A>
<UL>
<LI><A NAME="TOC8" HREF="VFL.html#SEC8">Mode Functions</A>
<LI><A NAME="TOC9" HREF="VFL.html#SEC9">File Functions</A>
<UL>
<LI><A NAME="TOC10" HREF="VFL.html#SEC10">Opening Files</A>
<LI><A NAME="TOC11" HREF="VFL.html#SEC11">Closing Files</A>
<LI><A NAME="TOC12" HREF="VFL.html#SEC12">File Keys</A>
<LI><A NAME="TOC13" HREF="VFL.html#SEC13">Saving Modes Across Opens</A>
</UL>
<LI><A NAME="TOC14" HREF="VFL.html#SEC14">Address Space Functions</A>
<UL>
<LI><A NAME="TOC15" HREF="VFL.html#SEC15">Userblock and Superblock</A>
<LI><A NAME="TOC16" HREF="VFL.html#SEC16">Allocation of Format Regions</A>
<LI><A NAME="TOC17" HREF="VFL.html#SEC17">Freeing Format Regions</A>
<LI><A NAME="TOC18" HREF="VFL.html#SEC18">Querying Address Range</A>
</UL>
<LI><A NAME="TOC19" HREF="VFL.html#SEC19">Data Functions</A>
<UL>
<LI><A NAME="TOC20" HREF="VFL.html#SEC20">Contiguous I/O Functions</A>
<LI><A NAME="TOC21" HREF="VFL.html#SEC21">Flushing Cached Data</A>
</UL>
<LI><A NAME="TOC22" HREF="VFL.html#SEC22">Optimization Functions</A>
<LI><A NAME="TOC23" HREF="VFL.html#SEC23">Registration of a Driver</A>
<LI><A NAME="TOC24" HREF="VFL.html#SEC24">Querying Driver Information</A>
</UL>
<LI><A NAME="TOC25" HREF="VFL.html#SEC25">Miscellaneous</A>
</UL>
<P><HR><P>
<H1><A NAME="SEC1" HREF="VFL.html#TOC1">Introduction</A></H1>
<P>
The HDF5 file format describes how HDF5 data structures and dataset raw
data are mapped to a linear <STRONG>format address space</STRONG> and the HDF5
library implements that bidirectional mapping in terms of an
API. However, the HDF5 format specifications do <EM>not</EM> indicate how
the format address space is mapped onto storage and HDF (version 5 and
earlier) simply mapped the format address space directly onto a single
file by convention.
</P>
<P>
Since early versions of HDF5 it became apparent that users want the ability to
map the format address space onto different types of storage (a single file,
multiple files, local memory, global memory, network distributed global
memory, a network protocol, <I>etc</I>.) with various types of maps. For
instance, some users want to be able to handle very large format address
spaces on operating systems that support only 2GB files by partitioning the
format address space into equal-sized parts each served by a separate
file. Other users want the same multi-file storage capability but want to
partition the address space according to purpose (raw data in one file, object
headers in another, global heap in a third, <I>etc.</I>) in order to improve I/O
speeds.
</P>
<P>
In fact, the number of storage variations is probably larger than the
number of methods that the HDF5 team is capable of implementing and
supporting. Therefore, a <STRONG>Virtual File Layer</STRONG> API is being
implemented which will allow application teams or departments to design
and implement their own mapping between the HDF5 format address space
and storage, with each mapping being a separate <STRONG>file driver</STRONG>
(possibly written in terms of other file drivers). The HDF5 team will
provide a small set of useful file drivers which will also serve as
examples for those who which to write their own:
</P>
<DL COMPACT>
<DT><CODE>H5FD_SEC2</CODE>
<DD>
This is the default driver which uses Posix file-system functions like
<CODE>read</CODE> and <CODE>write</CODE> to perform I/O to a single file. All I/O
requests are unbuffered although the driver does optimize file seeking
operations to some extent.
<DT><CODE>H5FD_STDIO</CODE>
<DD>
This driver uses functions from <TT>`stdio.h'</TT> to perform buffered I/O
to a single file.
<DT><CODE>H5FD_CORE</CODE>
<DD>
This driver performs I/O directly to memory and can be used to create small
temporary files that never exist on permanent storage. This type of storage is
generally very fast since the I/O consists only of memory-to-memory copy
operations.
<DT><CODE>H5FD_MPIIO</CODE>
<DD>
This is the driver of choice for accessing files in parallel using MPI and
MPI-IO. It is only predefined if the library is compiled with parallel I/O
support.
<DT><CODE>H5FD_FAMILY</CODE>
<DD>
Large format address spaces are partitioned into more manageable pieces and
sent to separate storage locations using an underlying driver of the user's
choice. The <CODE>h5repart</CODE> tool can be used to change the sizes of the
family members when stored as files or to convert a family of files to a
single file or vice versa.
<DT><CODE>H5FD_SPLIT</CODE>
<DD>
The format address space is split into meta data and raw data and each is
mapped onto separate storage using underlying drivers of the user's
choice. The meta data storage can be read by itself (for limited
functionality) or both files can be accessed together.
</DL>
<H1><A NAME="SEC2" HREF="VFL.html#TOC2">Using a File Driver</A></H1>
<P>
Most application writers will use a driver defined by the HDF5 library or
contributed by another programming team. This chapter describes how existing
drivers are used.
</P>
<H2><A NAME="SEC3" HREF="VFL.html#TOC3">Driver Header Files</A></H2>
<P>
Each file driver is defined in its own public header file which should
be included by any application which plans to use that driver. The
predefined drivers are in header files whose names begin with
<SAMP>`H5FD'</SAMP> followed by the driver name and <SAMP>`.h'</SAMP>. The <TT>`hdf5.h'</TT>
header file includes all the predefined driver header files.
</P>
<P>
Once the appropriate header file is included a symbol of the form
<SAMP>`H5FD_'</SAMP> followed by the upper-case driver name will be the driver
identification number.<A NAME="DOCF1" HREF="VFL.html#FOOT1">(1)</A> However, the
value may change if the library is closed (<I>e.g.</I>, by calling
<CODE>H5close</CODE>) and the symbol is referenced again.
</P>
<H2><A NAME="SEC4" HREF="VFL.html#TOC4">Creating and Opening Files</A></H2>
<P>
In order to create or open a file one must define the method by which the
storage is accessed<A NAME="DOCF2" HREF="VFL.html#FOOT2">(2)</A> and does so by creating a file access property list<A NAME="DOCF3" HREF="VFL.html#FOOT3">(3)</A> which is passed to the <CODE>H5Fcreate</CODE> or
<CODE>H5Fopen</CODE> function. A default file access property list is created by
calling <CODE>H5Pcreate</CODE> and then the file driver information is inserted by
calling a driver initialization function such as <CODE>H5Pset_fapl_family</CODE>:
</P>
<PRE>
hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
size_t member_size = 100*1024*1024; /*100MB*/
H5Pset_fapl_family(fapl, member_size, H5P_DEFAULT);
hid_t file = H5Fcreate("foo%05d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
H5Pclose(fapl);
</PRE>
<P>
Each file driver will have its own initialization function
whose name is <CODE>H5Pset_fapl_</CODE> followed by the driver name and which
takes a file access property list as the first argument followed by
additional driver-dependent arguments.
</P>
<P>
An alternative to using the driver initialization function is to set the
driver directly using the <CODE>H5Pset_driver</CODE> function.<A NAME="DOCF4" HREF="VFL.html#FOOT4">(4)</A> Its second argument is the file driver identifier, which may
have a different numeric value from run to run depending on the order in which
the file drivers are registered with the library. The third argument
encapsulates the additional arguments of the driver initialization
function. This method only works if the file driver writer has made the
driver-specific property list structure a public datatype, which is
often not the case.
</P>
<PRE>
hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
static H5FD_family_fapl_t fa = {100*1024*1024, H5P_DEFAULT};
H5Pset_driver(fapl, H5FD_FAMILY, &fa);
hid_t file = H5Fcreate("foo.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
H5Pclose(fapl);
</PRE>
<P>
It is also possible to query the file driver information from a file access
property list by calling <CODE>H5Pget_driver</CODE> to determine the driver and then
calling a driver-defined query function to obtain the driver information:
</P>
<PRE>
hid_t driver = H5Pget_driver(fapl);
if (H5FD_SEC2==driver) {
/*nothing further to get*/
} else if (H5FD_FAMILY==driver) {
hid_t member_fapl;
haddr_t member_size;
H5Pget_fapl_family(fapl, &member_size, &member_fapl);
} else if (....) {
....
}
</PRE>
<H2><A NAME="SEC5" HREF="VFL.html#TOC5">Performing I/O</A></H2>
<P>
The <CODE>H5Dread</CODE> and <CODE>H5Dwrite</CODE> functions transfer data between
application memory and the file. They both take an optional data transfer
property list which has some general driver-independent properties and
optional driver-defined properties. An application will typically perform I/O
in one of three styles via the <CODE>H5Dread</CODE> or <CODE>H5Dwrite</CODE> function:
</P>
<P>
Like file access properties in the previous section, data transfer properties
can be set using a driver initialization function or a general purpose
function. For example, to set the MPI-IO driver to use independent access for
I/O operations one would say:
</P>
<PRE>
hid_t dxpl = H5Pcreate(H5P_DATA_XFER);
H5Pset_dxpl_mpio(dxpl, H5FD_MPIO_INDEPENDENT);
H5Dread(dataset, type, mspace, fspace, buffer, dxpl);
H5Pclose(dxpl);
</PRE>
<P>
The alternative is to initialize a driver defined C <CODE>struct</CODE> and pass it
to the <CODE>H5Pset_driver</CODE> function:
</P>
<PRE>
hid_t dxpl = H5Pcreate(H5P_DATA_XFER);
static H5FD_mpio_dxpl_t dx = {H5FD_MPIO_INDEPENDENT};
H5Pset_driver(dxpl, H5FD_MPIO, &dx);
H5Dread(dataset, type, mspace, fspace, buffer, dxpl);
</PRE>
<P>
The transfer propery list can be queried in a manner similar to the file
access property list: the driver provides a function (or functions) to return
various information about the transfer property list:
</P>
<PRE>
hid_t driver = H5Pget_driver(dxpl);
if (H5FD_MPIO==driver) {
H5FD_mpio_xfer_t xfer_mode;
H5Pget_dxpl_mpio(dxpl, &xfer_mode);
} else {
....
}
</PRE>
<H2><A NAME="SEC6" HREF="VFL.html#TOC6">File Driver Interchangeability</A></H2>
<P>
The HDF5 specifications describe two things: the mapping of data onto a linear
<STRONG>format address space</STRONG> and the C API which performs the mapping.
However, the mapping of the format address space onto storage intentionally
falls outside the scope of the HDF5 specs. This is a direct result of the fact
that it is not generally possible to store information about how to access
storage inside the storage itself. For instance, given only the file name
<TT>`/arborea/1225/work/f%03d'</TT> the HDF5 library is unable to tell whether the
name refers to a file on the local file system, a family of files on the local
file system, a file on host <SAMP>`arborea'</SAMP> port 1225, a family of files on a
remote system, <I>etc</I>.
</P>
<P>
Two ways which library could figure out where the storage is located are:
storage access information can be provided by the user, or the library can try
all known file access methods. This implementation uses the former method.
</P>
<P>
In general, if a file was created with one driver then it isn't possible to
open it with another driver. There are of course exceptions: a file created
with MPIO could probably be opened with the sec2 driver, any file created
by the sec2 driver could be opened as a family of files with one member,
<I>etc</I>. In fact, sometimes a file must not only be opened with the same
driver but also with the same driver properties. The predefined drivers are
written in such a way that specifying the correct driver is sufficient for
opening a file.
</P>
<H1><A NAME="SEC7" HREF="VFL.html#TOC7">Implementation of a Driver</A></H1>
<P>
A driver is simply a collection of functions and data structures which are
registered with the HDF5 library at runtime. The functions fall into these
categories:
</P>
<UL>
<LI>Functions which operate on modes
<LI>Functions which operate on files
<LI>Functions which operate on the address space
<LI>Functions which operate on data
<LI>Functions for driver initialization
<LI>Optimization functions
</UL>
<H2><A NAME="SEC8" HREF="VFL.html#TOC8">Mode Functions</A></H2>
<P>
Some drivers need information about file access and data transfers which are
very specific to the driver. The information is usually implemented as a pair
of pointers to C structs which are allocated and initialized as part of an
HDF5 property list and passed down to various driver functions. There are two
classes of settings: file access modes that describe how to access the file
through the driver, and data transfer modes which are settings that control
I/O operations. Each file opened by a particular driver may have a different
access mode; each dataset I/O request for a particular file may have a
different data transfer mode.
</P>
<P>
Since each driver has its own particular requirements for various settings,
each driver is responsible for defining the mode structures that it
needs. Higher layers of the library treat the structures as opaque but must be
able to copy and free them. Thus, the driver provides either the size of the
structure or a pair of function pointers for each of the mode types.
</P>
<P>
<STRONG>Example:</STRONG> The family driver needs to know how the format address
space is partitioned and the file access property list to use for the
family members.
</P>
<PRE>
/* Driver-specific file access properties */
typedef struct H5FD_family_fapl_t {
hsize_t memb_size; /*size of each member */
hid_t memb_fapl_id; /*file access property list of each memb*/
} H5FD_family_fapl_t;
/* Driver specific data transfer properties */
typedef struct H5FD_family_dxpl_t {
hid_t memb_dxpl_id; /*data xfer property list of each memb */
} H5FD_family_dxpl_t;
</PRE>
<P>
In order to copy or free one of these structures the member file access
or data transfer properties must also be copied or freed. This is done
by providing a copy and close function for each structure:
</P>
<P>
<STRONG>Example:</STRONG> The file access property list copy and close functions
for the family driver:
</P>
<PRE>
static void *
H5FD_family_fapl_copy(const void *_old_fa)
{
const H5FD_family_fapl_t *old_fa = (const H5FD_family_fapl_t*)_old_fa;
H5FD_family_fapl_t *new_fa = malloc(sizeof(H5FD_family_fapl_t));
assert(new_fa);
memcpy(new_fa, old_fa, sizeof(H5FD_family_fapl_t));
new_fa->memb_fapl_id = H5Pcopy(old_fa->memb_fapl_id);
return new_fa;
}
static herr_t
H5FD_family_fapl_free(void *_fa)
{
H5FD_family_fapl_t *fa = (H5FD_family_fapl_t*)_fa;
H5Pclose(fa->memb_fapl_id);
free(fa);
return 0;
}
</PRE>
<P>
Generally when a file is created or opened the file access properties
for the driver are copied into the file pointer which is returned and
they may be modified from their original value (for instance, the file
family driver modifies the member size property when opening an existing
family). In order to support the <CODE>H5Fget_access_plist</CODE> function the
driver must provide a <CODE>fapl_get</CODE> callback which creates a copy of
the driver-specific properties based on a particular file.
</P>
<P>
<STRONG>Example:</STRONG> The file family driver copies the member size file
access property list into the return value:
</P>
<PRE>
static void *
H5FD_family_fapl_get(H5FD_t *_file)
{
H5FD_family_t *file = (H5FD_family_t*)_file;
H5FD_family_fapl_t *fa = calloc(1, sizeof(H5FD_family_fapl_t*));
fa->memb_size = file->memb_size;
fa->memb_fapl_id = H5Pcopy(file->memb_fapl_id);
return fa;
}
</PRE>
<H2><A NAME="SEC9" HREF="VFL.html#TOC9">File Functions</A></H2>
<P>
The higher layers of the library expect files to have a name and allow the
file to be accessed in various modes. The driver must be able to create a new
file, replace an existing file, or open an existing file. Opening or creating
a file should return a handle, a pointer to a specialization of the
<CODE>H5FD_t</CODE> struct, which allows read-only or read-write access and which
will be passed to the other driver functions as they are
called.<A NAME="DOCF5" HREF="VFL.html#FOOT5">(5)</A>
</P>
<PRE>
typedef struct {
/* Public fields */
H5FD_class_t *cls; /*class data defined below*/
/* Private fields -- driver-defined */
} H5FD_t;
</PRE>
<P>
<STRONG>Example:</STRONG> The family driver requires handles to the underlying
storage, the size of the members for this particular file (which might be
different than the member size specified in the file access property list if
an existing file family is being opened), the name used to open the file in
case additional members must be created, and the flags to use for creating
those additional members. The <CODE>eoa</CODE> member caches the size of the format
address space so the family members don't have to be queried in order to find
it.
</P>
<PRE>
/* The description of a file belonging to this driver. */
typedef struct H5FD_family_t {
H5FD_t pub; /*public stuff, must be first */
hid_t memb_fapl_id; /*file access property list for members */
hsize_t memb_size; /*maximum size of each member file */
int nmembs; /*number of family members */
int amembs; /*number of member slots allocated */
H5FD_t **memb; /*dynamic array of member pointers */
haddr_t eoa; /*end of allocated addresses */
char *name; /*name generator printf format */
unsigned flags; /*flags for opening additional members */
} H5FD_family_t;
</PRE>
<P>
<STRONG>Example:</STRONG> The sec2 driver needs to keep track of the underlying Unix
file descriptor and also the end of format address space and current Unix file
size. It also keeps track of the current file position and last operation
(read, write, or unknown) in order to optimize calls to <CODE>lseek</CODE>. The
<CODE>device</CODE> and <CODE>inode</CODE> fields are defined on Unix in order to uniquely
identify the file and will be discussed below.
</P>
<PRE>
typedef struct H5FD_sec2_t {
H5FD_t pub; /*public stuff, must be first */
int fd; /*the unix file */
haddr_t eoa; /*end of allocated region */
haddr_t eof; /*end of file; current file size*/
haddr_t pos; /*current file I/O position */
int op; /*last operation */
dev_t device; /*file device number */
ino_t inode; /*file i-node number */
} H5FD_sec2_t;
</PRE>
<H3><A NAME="SEC10" HREF="VFL.html#TOC10">Opening Files</A></H3>
<P>
All drivers must define a function for opening/creating a file. This
function should have a prototype which is:
</P>
<P>
<DL>
<DT><U>Function:</U> static H5FD_t * <B>open</B> <I>(const char *<VAR>name</VAR>, unsigned <VAR>flags</VAR>, hid_t <VAR>fapl</VAR>, haddr_t <VAR>maxaddr</VAR>)</I>
<DD><A NAME="IDX1"></A>
</P>
<P>
The file name <VAR>name</VAR> and file access property list <VAR>fapl</VAR> are
the same as were specified in the <CODE>H5Fcreate</CODE> or <CODE>H5Fopen</CODE>
call. The <VAR>flags</VAR> are the same as in those calls also except the
flag <CODE>H5F_ACC_CREATE</CODE> is also present if the call was to
<CODE>H5Fcreate</CODE> and they are documented in the <TT>`H5Fpublic.h'</TT>
file. The <VAR>maxaddr</VAR> argument is the maximum format address that the
driver should be prepared to handle (the minimum address is always
zero).
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver opens a Unix file with the requested name
and saves information which uniquely identifies the file (the Unix device
number and inode).
</P>
<PRE>
static H5FD_t *
H5FD_sec2_open(const char *name, unsigned flags, hid_t fapl_id/*unused*/,
haddr_t maxaddr)
{
unsigned o_flags;
int fd;
struct stat sb;
H5FD_sec2_t *file=NULL;
/* Check arguments */
if (!name || !*name) return NULL;
if (0==maxaddr || HADDR_UNDEF==maxaddr) return NULL;
if (ADDR_OVERFLOW(maxaddr)) return NULL;
/* Build the open flags */
o_flags = (H5F_ACC_RDWR & flags) ? O_RDWR : O_RDONLY;
if (H5F_ACC_TRUNC & flags) o_flags |= O_TRUNC;
if (H5F_ACC_CREAT & flags) o_flags |= O_CREAT;
if (H5F_ACC_EXCL & flags) o_flags |= O_EXCL;
/* Open the file */
if ((fd=open(name, o_flags, 0666))<0) return NULL;
if (fstat(fd, &sb)<0) {
close(fd);
return NULL;
}
/* Create the new file struct */
file = calloc(1, sizeof(H5FD_sec2_t));
file->fd = fd;
file->eof = sb.st_size;
file->pos = HADDR_UNDEF;
file->op = OP_UNKNOWN;
file->device = sb.st_dev;
file->inode = sb.st_ino;
return (H5FD_t*)file;
}
</PRE>
<H3><A NAME="SEC11" HREF="VFL.html#TOC11">Closing Files</A></H3>
<P>
Closing a file simply means that all cached data should be flushed to the next
lower layer, the file should be closed at the next lower layer, and all
file-related data structures should be freed. All information needed by the
close function is already present in the file handle.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>close</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX2"></A>
</P>
<P>
The <VAR>file</VAR> argument is the handle which was returned by the <CODE>open</CODE>
function, and the <CODE>close</CODE> should free only memory associated with the
driver-specific part of the handle (the public parts will have already been released by HDF5's virtual file layer).
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver just closes the underlying Unix file,
making sure that the actual file size is the same as that known to the
library by writing a zero to the last file position it hasn't been
written by some previous operation (which happens in the same code which
flushes the file contents and is shown below).
</P>
<PRE>
static herr_t
H5FD_sec2_close(H5FD_t *_file)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
if (H5FD_sec2_flush(_file)<0) return -1;
if (close(file->fd)<0) return -1;
free(file);
return 0;
}
</PRE>
<H3><A NAME="SEC12" HREF="VFL.html#TOC12">File Keys</A></H3>
<P>
Occasionally an application will attempt to open a single file more than one
time in order to obtain multiple handles to the file. HDF5 allows the files to
share information<A NAME="DOCF6" HREF="VFL.html#FOOT6">(6)</A> but in order to
accomplish this HDF5 must be able to tell when two names refer to the same
file. It does this by associating a driver-defined key with each file opened
by a driver and comparing the key for an open request with the keys for all
other files currently open by the same driver.
</P>
<P>
<DL>
<DT><U>Function:</U> const int <B>cmp</B> <I>(const H5FD_t *<VAR>f1</VAR>, const H5FD_t *<VAR>f2</VAR>)</I>
<DD><A NAME="IDX3"></A>
</P>
<P>
The driver may provide a function which compares two files <VAR>f1</VAR> and
<VAR>f2</VAR> belonging to the same driver and returns a negative, positive, or
zero value <I>a la</I> the <CODE>strcmp</CODE> function.<A NAME="DOCF7" HREF="VFL.html#FOOT7">(7)</A> If this
function is not provided then HDF5 assumes that all calls to the <CODE>open</CODE>
callback return unique files regardless of the arguments and it is up to the
application to avoid doing this if that assumption is incorrect.
</DL>
</P>
<P>
Each time a file is opened the library calls the <CODE>cmp</CODE> function to
compare that file with all other files currently open by the same driver and
if one of them matches (at most one can match) then the file which was just
opened is closed and the previously opened file is used instead.
</P>
<P>
Opening a file twice with incompatible flags will result in failure. For
instance, opening a file with the truncate flag is a two step process which
first opens the file without truncation so keys can be compared, and if no
matching file is found already open then the file is closed and immediately
reopened with the truncation flag set (if a matching file is already open then
the truncating open will fail).
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver uses the Unix device and i-node as the
key. They were initialized when the file was opened.
</P>
<PRE>
static int
H5FD_sec2_cmp(const H5FD_t *_f1, const H5FD_t *_f2)
{
const H5FD_sec2_t *f1 = (const H5FD_sec2_t*)_f1;
const H5FD_sec2_t *f2 = (const H5FD_sec2_t*)_f2;
if (f1->device < f2->device) return -1;
if (f1->device > f2->device) return 1;
if (f1->inode < f2->inode) return -1;
if (f1->inode > f2->inode) return 1;
return 0;
}
</PRE>
<H3><A NAME="SEC13" HREF="VFL.html#TOC13">Saving Modes Across Opens</A></H3>
<P>
Some drivers may also need to store certain information in the file superblock
in order to be able to reliably open the file at a later date. This is done by
three functions: one to determine how much space will be necessary to store
the information in the superblock, one to encode the information, and one to
decode the information. These functions are optional, but if any one is
defined then the other two must also be defined.
</P>
<P>
<DL>
<DT><U>Function:</U> static hsize_t <B>sb_size</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX4"></A>
<DT><U>Function:</U> static herr_t <B>sb_encode</B> <I>(H5FD_t *<VAR>file</VAR>, char *<VAR>name</VAR>, unsigned char *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX5"></A>
<DT><U>Function:</U> static herr_t <B>sb_decode</B> <I>(H5FD_t *<VAR>file</VAR>, const char *<VAR>name</VAR>, const unsigned char *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX6"></A>
</P>
<P>
The <CODE>sb_size</CODE> function returns the number of bytes necessary to encode
information needed later if the file is reopened. The <CODE>sb_encode</CODE>
function encodes information from the file into buffer <VAR>buf</VAR>
allocated by the caller. It also writes an 8-character (plus null
termination) into the <CODE>name</CODE> argument, which should be a unique
identification for the driver. The <CODE>sb_decode</CODE> function looks at
the <VAR>name</VAR>
</P>
<P>
decodes
data from the buffer <VAR>buf</VAR> and updates the <VAR>file</VAR> argument with the new information,
advancing <VAR>*p</VAR> in the process.
</DL>
</P>
<P>
The part of this which is somewhat tricky is that the file must be readable
before the superblock information is decoded. File access modes fall outside
the scope of the HDF5 file format, but they are placed inside the boot block
for convenience.<A NAME="DOCF8" HREF="VFL.html#FOOT8">(8)</A>
</P>
<P>
<STRONG>Example:</STRONG> <EM>To be written later.</EM>
</P>
<H2><A NAME="SEC14" HREF="VFL.html#TOC14">Address Space Functions</A></H2>
<P>
HDF5 does not assume that a file is a linear address space of bytes. Instead,
the library will call functions to allocate and free portions of the HDF5
format address space, which in turn map onto functions in the file driver to
allocate and free portions of file address space. The library tells the file
driver how much format address space it wants to allocate and the driver
decides what format address to use and how that format address is mapped onto
the file address space. Usually the format address is chosen so that the file
address can be calculated in constant time for data I/O operations (which are
always specified by format addresses).
</P>
<H3><A NAME="SEC15" HREF="VFL.html#TOC15">Userblock and Superblock</A></H3>
<P>
The HDF5 format allows an optional userblock to appear before the actual HDF5
data in such a way that if the userblock is <STRONG>sucked out</STRONG> of the file and
everything remaining is shifted downward in the file address space, then the
file is still a valid HDF5 file. The userblock size can be zero or any
multiple of two greater than or equal to 512 and the file superblock begins
immediately after the userblock.
</P>
<P>
HDF5 allocates space for the userblock and superblock by calling an
allocation function defined below, which must return a chunk of memory at
format address zero on the first call.
</P>
<H3><A NAME="SEC16" HREF="VFL.html#TOC16">Allocation of Format Regions</A></H3>
<P>
The library makes many types of allocation requests:
</P>
<DL COMPACT>
<DT><CODE>H5FD_MEM_SUPER</CODE>
<DD>
An allocation request for the userblock and/or superblock.
<DT><CODE>H5FD_MEM_BTREE</CODE>
<DD>
An allocation request for a node of a B-tree.
<DT><CODE>H5FD_MEM_DRAW</CODE>
<DD>
An allocation request for the raw data of a dataset.
<DT><CODE>H5FD_MEM_META</CODE>
<DD>
An allocation request for the raw data of a dataset which
the user has indicated will be relatively small.
<DT><CODE>H5FD_MEM_GROUP</CODE>
<DD>
An allocation request for a group leaf node (internal nodes of the group tree
are allocated as H5MF_BTREE).
<DT><CODE>H5FD_MEM_GHEAP</CODE>
<DD>
An allocation request for a global heap collection. Global heaps are used to
store certain types of references such as dataset region references. The set
of all global heap collections can become quite large.
<DT><CODE>H5FD_MEM_LHEAP</CODE>
<DD>
An allocation request for a local heap. Local heaps are used to store the
names which are members of a group. The combined size of all local heaps is a
function of the number of object names in the file.
<DT><CODE>H5FD_MEM_OHDR</CODE>
<DD>
An allocation request for (part of) an object header. Object headers are
relatively small and include meta information about objects (like the data
space and type of a dataset) and attributes.
</DL>
<P>
When a chunk of memory is freed the library adds it to a free list and
allocation requests are satisfied from the free list before requesting memory
from the file driver. Each type of allocation request enumerated above has its
own free list, but the file driver can specify that certain object types can
share a free list. It does so by providing an array which maps a request type
to a free list. If any value of the map is <CODE>H5MF_DEFAULT</CODE> (zero) then the
object's own free list is used. The special value <CODE>H5MF_NOLIST</CODE> indicates
that the library should not attempt to maintain a free list for that
particular object type, instead calling the file driver each time an object of
that type is freed.
</P>
<P>
Mappings predefined in the <TT>`H5FDpublic.h'</TT> file are:
<DL COMPACT>
<DT><CODE>H5FD_FLMAP_SINGLE</CODE>
<DD>
All memory usage types are mapped to a single free list.
<DT><CODE>H5FD_FLMAP_DICHOTOMY</CODE>
<DD>
Memory usage is segregated into meta data and raw data for the purposes of
memory management.
<DT><CODE>H5FD_FLMAP_DEFAULT</CODE>
<DD>
Each memory usage type has its own free list.
</DL>
<P>
<STRONG>Example:</STRONG> To make a map that manages object headers on one free list
and everything else on another free list one might initialize the map with the
following code: (the use of <CODE>H5FD_MEM_SUPER</CODE> is arbitrary)
</P>
<PRE>
H5FD_mem_t mt, map[H5FD_MEM_NTYPES];
for (mt=0; mt<H5FD_MEM_NTYPES; mt++) {
map[mt] = (H5FD_MEM_OHDR==mt) ? mt : H5FD_MEM_SUPER;
}
</PRE>
<P>
If an allocation request cannot be satisfied from the free list then one of
two things happen. If the driver defines an allocation callback then it is
used to allocate space; otherwise new memory is allocated from the end of the
format address space by incrementing the end-of-address marker.
</P>
<P>
<DL>
<DT><U>Function:</U> static haddr_t <B>alloc</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, hsize_t <VAR>size</VAR>)</I>
<DD><A NAME="IDX7"></A>
</P>
<P>
The <VAR>file</VAR> argument is the file from which space is to be allocated,
<VAR>type</VAR> is the type of memory being requested (from the list above) without
being mapped according to the freelist map and <VAR>size</VAR> is the number of
bytes being requested. The library is allowed to allocate large chunks of
storage and manage them in a layer above the file driver (although the current
library doesn't do that). The allocation function should return a format
address for the first byte allocated. The allocated region extends from that
address for <VAR>size</VAR> bytes. If the request cannot be honored then the
undefined address value is returned (<CODE>HADDR_UNDEF</CODE>). The first call to
this function for a file which has never had memory allocated <EM>must</EM>
return a format address of zero or <CODE>HADDR_UNDEF</CODE> since this is how the
library allocates space for the userblock and/or superblock.
</DL>
</P>
<P>
<STRONG>Example:</STRONG> <EM>To be written later.</EM>
</P>
<H3><A NAME="SEC17" HREF="VFL.html#TOC17">Freeing Format Regions</A></H3>
<P>
When the library is finished using a certain region of the format address
space it will return the space to the free list according to the type of
memory being freed and the free list map described above. If the free list has
been disabled for a particular memory usage type (according to the free list
map) and the driver defines a <CODE>free</CODE> callback then it will be
invoked. The <CODE>free</CODE> callback is also invoked for all entries on the free
list when the file is closed.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>free</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>)</I>
<DD><A NAME="IDX8"></A>
</P>
<P>
The <VAR>file</VAR> argument is the file for which space is being freed; <VAR>type</VAR>
is the type of object being freed (from the list above) without being mapped
according to the freelist map; <VAR>addr</VAR> is the first format address to free;
and <VAR>size</VAR> is the size in bytes of the region being freed. The region
being freed may refer to just part of the region originally allocated and/or
may cross allocation boundaries provided all regions being freed have the same
usage type. However, the library will never attempt to free regions which have
already been freed or which have never been allocated.
</DL>
</P>
<P>
A driver may choose to not define the <CODE>free</CODE> function, in which case
format addresses will be leaked. This isn't normally a huge problem since the
library contains a simple free list of its own and freeing parts of the format
address space is not a common occurrence.
</P>
<P>
<STRONG>Example:</STRONG> <EM>To be written later.</EM>
</P>
<H3><A NAME="SEC18" HREF="VFL.html#TOC18">Querying Address Range</A></H3>
<P>
Each file driver must have some mechanism for setting and querying the end of
address, or <STRONG>EOA</STRONG>, marker. The EOA marker is the first format address
after the last format address ever allocated. If the last part of the
allocated address range is freed then the driver may optionally decrease the
eoa marker.
</P>
<P>
<DL>
<DT><U>Function:</U> static haddr_t <B>get_eoa</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX9"></A>
</P>
<P>
This function returns the current value of the EOA marker for the specified
file.
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver just returns the current eoa marker value
which is cached in the file structure:
</P>
<PRE>
static haddr_t
H5FD_sec2_get_eoa(H5FD_t *_file)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
return file->eoa;
}
</PRE>
<P>
The eoa marker is initially zero when a file is opened and the library may set
it to some other value shortly after the file is opened (after the superblock
is read and the saved eoa marker is determined) or when allocating additional
memory in the absence of an <CODE>alloc</CODE> callback (described above).
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver simply caches the eoa marker in the file
structure and does not extend the underlying Unix file. When the file is
flushed or closed then the Unix file size is extended to match the eoa marker.
</P>
<PRE>
static herr_t
H5FD_sec2_set_eoa(H5FD_t *_file, haddr_t addr)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
file->eoa = addr;
return 0;
}
</PRE>
<H2><A NAME="SEC19" HREF="VFL.html#TOC19">Data Functions</A></H2>
<P>
These functions operate on data, transferring a region of the format address
space between memory and files.
</P>
<H3><A NAME="SEC20" HREF="VFL.html#TOC20">Contiguous I/O Functions</A></H3>
<P>
A driver must specify two functions to transfer data from the library to the
file and vice versa.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>read</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, void *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX10"></A>
<DT><U>Function:</U> static herr_t <B>write</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, const void *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX11"></A>
</P>
<P>
The <CODE>read</CODE> function reads data from file <VAR>file</VAR> beginning at address
<VAR>addr</VAR> and continuing for <VAR>size</VAR> bytes into the buffer <VAR>buf</VAR>
supplied by the caller. The <CODE>write</CODE> function transfers data in the
opposite direction. Both functions take a data transfer property list
<VAR>dxpl</VAR> which indicates the fine points of how the data is to be
transferred and which comes directly from the <CODE>H5Dread</CODE> or
<CODE>H5Dwrite</CODE> function. Both functions receive <VAR>type</VAR> of
data being written, which may allow a driver to tune it's behavior for
different kinds of data.
</DL>
</P>
<P>
Both functions should return a negative value if they fail to transfer the
requested data, or non-negative if they succeed. The library will never
attempt to read from unallocated regions of the format address space.
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver just makes system calls. It tries not to
call <CODE>lseek</CODE> if the current operation is the same as the previous
operation and the file position is correct. It also fills the output buffer
with zeros when reading between the current EOF and EOA markers and restarts
system calls which were interrupted.
</P>
<PRE>
static herr_t
H5FD_sec2_read(H5FD_t *_file, H5FD_mem_t type/*unused*/, hid_t dxpl_id/*unused*/,
haddr_t addr, hsize_t size, void *buf/*out*/)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
ssize_t nbytes;
assert(file && file->pub.cls);
assert(buf);
/* Check for overflow conditions */
if (REGION_OVERFLOW(addr, size)) return -1;
if (addr+size>file->eoa) return -1;
/* Seek to the correct location */
if ((addr!=file->pos || OP_READ!=file->op) &&
file_seek(file->fd, (file_offset_t)addr, SEEK_SET)<0) {
file->pos = HADDR_UNDEF;
file->op = OP_UNKNOWN;
return -1;
}
/*
* Read data, being careful of interrupted system calls, partial results,
* and the end of the file.
*/
while (size>0) {
do nbytes = read(file->fd, buf, size);
while (-1==nbytes && EINTR==errno);
if (-1==nbytes) {
/* error */
file->pos = HADDR_UNDEF;
file->op = OP_UNKNOWN;
return -1;
}
if (0==nbytes) {
/* end of file but not end of format address space */
memset(buf, 0, size);
size = 0;
}
assert(nbytes>=0);
assert((hsize_t)nbytes<=size);
size -= (hsize_t)nbytes;
addr += (haddr_t)nbytes;
buf = (char*)buf + nbytes;
}
/* Update current position */
file->pos = addr;
file->op = OP_READ;
return 0;
}
</PRE>
<P>
<STRONG>Example:</STRONG> The sec2 <CODE>write</CODE> callback is similar except it updates
the file EOF marker when extending the file.
</P>
<H3><A NAME="SEC21" HREF="VFL.html#TOC21">Flushing Cached Data</A></H3>
<P>
Some drivers may desire to cache data in memory in order to make larger I/O
requests to the underlying file and thus improving bandwidth. Such drivers
should register a cache flushing function so that the library can insure that
data has been flushed out of the drivers in response to the application
calling <CODE>H5Fflush</CODE>.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>flush</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX12"></A>
</P>
<P>
Flush all data for file <VAR>file</VAR> to storage.
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver doesn't cache any data but it also doesn't
extend the Unix file as agressively as it should. Therefore, when finalizing a
file it should write a zero to the last byte of the allocated region so that
when reopening the file later the EOF marker will be at least as large as the
EOA marker saved in the superblock (otherwise HDF5 will refuse to open the
file, claiming that the data appears to be truncated).
</P>
<PRE>
static herr_t
H5FD_sec2_flush(H5FD_t *_file)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
if (file->eoa>file->eof) {
if (-1==file_seek(file->fd, file->eoa-1, SEEK_SET)) return -1;
if (write(file->fd, "", 1)!=1) return -1;
file->eof = file->eoa;
file->pos = file->eoa;
file->op = OP_WRITE;
}
return 0;
}
</PRE>
<H2><A NAME="SEC22" HREF="VFL.html#TOC22">Optimization Functions</A></H2>
<P>
The library is capable of performing several generic optimizations on I/O, but
these types of optimizations may not be appropriate for a given VFL driver.
</P>
<P>
Each driver may provide a query function to allow the library to query whether
to enable these optimizations. If a driver lacks a query function, the library
will disable all types of optimizations which can be queried.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>query</B> <I>(const H5FD_t *<VAR>file</VAR>, unsigned long *<VAR>flags</VAR>)</I>
<DD><A NAME="IDX17"></A>
</P>
<P>
This function is called by the library to query which optimizations to enable
for I/O to this driver. These are the flags which are currently defined:
<UL>
<DL>
<DT>H5FD_FEAT_AGGREGATE_METADATA (0x00000001)
<DD>Defining the H5FD_FEAT_AGGREGATE_METADATA for a VFL driver means that
the library will attempt to allocate a larger block for metadata and
then sub-allocate each metadata request from that larger block.
<DT>H5FD_FEAT_ACCUMULATE_METADATA (0x00000002)
<DD>Defining the H5FD_FEAT_ACCUMULATE_METADATA for a VFL driver means that
the library will attempt to cache metadata as it is written to the file
and build up a larger block of metadata to eventually pass to the VFL
'write' routine.
<DT>H5FD_FEAT_DATA_SIEVE (0x00000004)
<DD>Defining the H5FD_FEAT_DATA_SIEVE for a VFL driver means that
the library will attempt to cache raw data as it is read from/written to
a file in a "data sieve" buffer. See Rajeev Thakur's papers:
<UL>
<DL>
<DT>http://www.mcs.anl.gov/~thakur/papers/romio-coll.ps.gz
<DT>http://www.mcs.anl.gov/~thakur/papers/mpio-high-perf.ps.gz
</DL>
</UL>
</DL>
</UL>
</P>
</DL>
</P>
<H2><A NAME="SEC23" HREF="VFL.html#TOC23">Registration of a Driver</A></H2>
<P>
Before a driver can be used the HDF5 library needs to be told of its
existence. This is done by registering the driver, which results in a driver
identification number. Instead of passing many arguments to the registration
function, the driver information is entered into a structure and the address
of the structure is passed to the registration function where it is
copied. This allows the HDF5 API to be extended while providing backward
compatibility at the source level.
</P>
<P>
<DL>
<DT><U>Function:</U> hid_t <B>H5FDregister</B> <I>(H5FD_class_t *<VAR>cls</VAR>)</I>
<DD><A NAME="IDX13"></A>
</P>
<P>
The driver described by struct <VAR>cls</VAR> is registered with the library and an
ID number for the driver is returned.
</DL>
</P>
<P>
The <CODE>H5FD_class_t</CODE> type is a struct with the following fields:
</P>
<DL COMPACT>
<DT><CODE>const char *name</CODE>
<DD>
A pointer to a constant, null-terminated driver name to be used for debugging
purposes.
<DT><CODE>size_t fapl_size</CODE>
<DD>
The size in bytes of the file access mode structure or zero if the driver
supplies a copy function or doesn't define the structure.
<DT><CODE>void *(*fapl_copy)(const void *fapl)</CODE>
<DD>
An optional function which copies a driver-defined file access mode structure.
This field takes precedence over <CODE>fm_size</CODE> when both are defined.
<DT><CODE>void (*fapl_free)(void *fapl)</CODE>
<DD>
An optional function to free the driver-defined file access mode structure. If
null, then the library calls the C <CODE>free</CODE> function to free the
structure.
<DT><CODE>size_t dxpl_size</CODE>
<DD>
The size in bytes of the data transfer mode structure or zero if the driver
supplies a copy function or doesn't define the structure.
<DT><CODE>void *(*dxpl_copy)(const void *dxpl)</CODE>
<DD>
An optional function which copies a driver-defined data transfer mode
structure. This field takes precedence over <CODE>xm_size</CODE> when both are
defined.
<DT><CODE>void (*dxpl_free)(void *dxpl)</CODE>
<DD>
An optional function to free the driver-defined data transfer mode
structure. If null, then the library calls the C <CODE>free</CODE> function to
free the structure.
<DT><CODE>H5FD_t *(*open)(const char *name, unsigned flags, hid_t fapl, haddr_t maxaddr)</CODE>
<DD>
The function which opens or creates a new file.
<DT><CODE>herr_t (*close)(H5FD_t *file)</CODE>
<DD>
The function which ends access to a file.
<DT><CODE>int (*cmp)(const H5FD_t *f1, const H5FD_t *f2)</CODE>
<DD>
An optional function to determine whether two open files have the same key. If
this function is not present then the library assumes that two files will
never be the same.
<DT><CODE>int (*query)(const H5FD_t *f, unsigned long *flags)</CODE>
<DD>
An optional function to determine which library optimizations a driver can
support.
<DT><CODE>haddr_t (*alloc)(H5FD_t *file, H5FD_mem_t type, hsize_t size)</CODE>
<DD>
An optional function to allocate space in the file.
<DT><CODE>herr_t (*free)(H5FD_t *file, H5FD_mem_t type, haddr_t addr, hsize_t size)</CODE>
<DD>
An optional function to free space in the file.
<DT><CODE>haddr_t (*get_eoa)(H5FD_t *file)</CODE>
<DD>
A function to query how much of the format address space has been allocated.
<DT><CODE>herr_t (*set_eoa)(H5FD_t *file, haddr_t)</CODE>
<DD>
A function to set the end of address space.
<DT><CODE>haddr_t (*get_eof)(H5FD_t *file)</CODE>
<DD>
A function to return the current end-of-file marker value.
<DT><CODE>herr_t (*read)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, void *buffer)</CODE>
<DD>
A function to read data from a file.
<DT><CODE>herr_t (*write)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, const void *buffer)</CODE>
<DD>
A function to write data to a file.
<DT><CODE>herr_t (*flush)(H5FD_t *file)</CODE>
<DD>
A function which flushes cached data to the file.
<DT><CODE>H5FD_mem_t fl_map[H5FD_MEM_NTYPES]</CODE>
<DD>
An array which maps a file allocation request type to a free list.
</DL>
<P>
<STRONG>Example:</STRONG> The sec2 driver would be registered as:
</P>
<PRE>
static const H5FD_class_t H5FD_sec2_g = {
"sec2", /*name */
MAXADDR, /*maxaddr */
NULL, /*sb_size */
NULL, /*sb_encode */
NULL, /*sb_decode */
0, /*fapl_size */
NULL, /*fapl_get */
NULL, /*fapl_copy */
NULL, /*fapl_free */
0, /*dxpl_size */
NULL, /*dxpl_copy */
NULL, /*dxpl_free */
H5FD_sec2_open, /*open */
H5FD_sec2_close, /*close */
H5FD_sec2_cmp, /*cmp */
H5FD_sec2_query, /*query */
NULL, /*alloc */
NULL, /*free */
H5FD_sec2_get_eoa, /*get_eoa */
H5FD_sec2_set_eoa, /*set_eoa */
H5FD_sec2_get_eof, /*get_eof */
H5FD_sec2_read, /*read */
H5FD_sec2_write, /*write */
H5FD_sec2_flush, /*flush */
H5FD_FLMAP_SINGLE, /*fl_map */
};
hid_t
H5FD_sec2_init(void)
{
if (!H5FD_SEC2_g) {
H5FD_SEC2_g = H5FDregister(&H5FD_sec2_g);
}
return H5FD_SEC2_g;
}
</PRE>
<P>
A driver can be removed from the library by unregistering it
</P>
<P>
<DL>
<DT><U>Function:</U> herr_t <B>H5Dunregister</B> <I>(hid_t <VAR>driver</VAR>)</I>
<DD><A NAME="IDX14"></A>
Where <VAR>driver</VAR> is the ID number returned when the driver was registered.
</DL>
</P>
<P>
Unregistering a driver makes it unusable for creating new file access or data
transfer property lists but doesn't affect any property lists or files that
already use that driver.
</P>
<H2><A NAME="SEC24" HREF="VFL.html#TOC24">Querying Driver Information</A></H2>
<P>
<DL>
<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fapl</VAR>)</I>
<DD><A NAME="IDX15"></A>
<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fxpl</VAR>)</I>
<DD><A NAME="IDX16"></A>
</P>
<P>
This function is intended to be used by driver functions, not applications.
It returns a pointer directly into the file access property list
<CODE><VAR>fapl</VAR></CODE> which is a copy of the driver's file access mode originally
provided to the <CODE>H5Pset_driver</CODE> function. If its argument is a data
transfer property list <CODE>fxpl</CODE> then it returns a pointer to the
driver-specific data transfer information instead.
</DL>
</P>
<H1><A NAME="SEC25" HREF="VFL.html#TOC25">Miscellaneous</A></H1>
<P>
The various private <CODE>H5F_low_*</CODE> functions will be replaced by public
<CODE>H5FD*</CODE> functions so they can be called from drivers.
</P>
<P>
All private functions <CODE>H5F_addr_*</CODE> which operate on addresses will be
renamed as public functions by removing the first underscore so they can be
called by drivers.
</P>
<P>
The <CODE>haddr_t</CODE> address data type will be passed by value throughout the
library. The original intent was that this type would eventually be a union of
file address types for the various drivers and may become quite large, but
that was back when drivers were part of HDF5. It will become an alias for an
unsigned integer type (32 or 64 bits depending on how the library was
configured).
</P>
<P>
The various <CODE>H5F*.c</CODE> driver files will be renamed <CODE>H5FD*.c</CODE> and each
will have a corresponding header file. All driver functions except the
initializer and API will be declared static.
</P>
<P>
This documentation didn't cover optimization functions which would be useful
to drivers like MPI-IO. Some drivers may be able to perform data pipeline
operations more efficiently than HDF5 and need to be given a chance to
override those parts of the pipeline. The pipeline would be designed to call
various H5FD optimization functions at various points which return one of
three values: the operation is not implemented by the driver, the operation is
implemented but failed in a non-recoverable manner, the operation is
implemented and succeeded.
</P>
<P>
Various parts of HDF5 check the only the top-level file driver and do
something special if it is the MPI-IO driver. However, we might want to be
able to put the MPI-IO driver under other drivers such as the raw part of a
split driver or under a debug driver whose sole purpose is to accumulate
statistics as it passes all requests through to the MPI-IO driver. Therefore
we will probably need a function which takes a format address and or object
type and returns the driver which would have been used at the lowest level to
process the request.
</P>
<P><HR><P>
<H1>Footnotes</H1>
<H3><A NAME="FOOT1" HREF="VFL.html#DOCF1">(1)</A></H3>
<P>The driver name is by convention and might
not apply to drivers which are not distributed with HDF5.
<H3><A NAME="FOOT2" HREF="VFL.html#DOCF2">(2)</A></H3>
<P>The access method also indicates how to translate
the storage name to a storage server such as a file, network protocol, or
memory.
<H3><A NAME="FOOT3" HREF="VFL.html#DOCF3">(3)</A></H3>
<P>The term
"<EM>file</EM> access property list" is a misnomer since storage isn't
required to be a file.
<H3><A NAME="FOOT4" HREF="VFL.html#DOCF4">(4)</A></H3>
<P>This
function is overloaded to operate on data transfer property lists also, as
described below.
<H3><A NAME="FOOT5" HREF="VFL.html#DOCF5">(5)</A></H3>
<P>Read-only access is only appropriate when opening an existing
file.
<H3><A NAME="FOOT6" HREF="VFL.html#DOCF6">(6)</A></H3>
<P>For instance, writing data to one handle will cause
the data to be immediately visible on the other handle.
<H3><A NAME="FOOT7" HREF="VFL.html#DOCF7">(7)</A></H3>
<P>The ordering is
arbitrary as long as it's consistent within a particular file driver.
<H3><A NAME="FOOT8" HREF="VFL.html#DOCF8">(8)</A></H3>
<P>File access modes do not describe data, but rather
describe how the HDF5 format address space is mapped to the underlying
file(s). Thus, in general the mapping must be known before the file superblock
can be read. However, the user usually knows enough about the mapping for the
superblock to be readable and once the superblock is read the library can fill
in the missing parts of the mapping.
<P><HR><P>
This document was generated on 18 November 1999 using the
<A HREF="http://wwwcn.cern.ch/dci/texi2html/">texi2html</A>
translator version 1.51.</P>
<P>
Updated on 10/24/00 by hand, Quincey Koziol
</P>
</BODY>
</HTML>
|
