1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
|
=pod
=head1 NAME
B<FunColumnSelect - select Funtools columns>
=head1 SYNOPSIS
#include <funtools.h>
int FunColumnSelect(Fun fun, int size, char *plist,
char *name1, char *type1, char *mode1, int offset1,
char *name2, char *type2, char *mode2, int offset2,
...,
NULL)
int FunColumnSelectArr(Fun fun, int size, char *plist,
char **names, char **types, char **modes,
int *offsets, int nargs);
=head1 DESCRIPTION
The B<FunColumnSelect()> routine is used to select the columns
from a Funtools binary table extension or raw event file for
processing. This routine allows you to specify how columns in a file
are to be read into a user record structure or written from a user
record structure to an output FITS file.
The first argument is the Fun handle associated with this set of
columns. The second argument specifies the size of the user record
structure into which columns will be read. Typically, the sizeof()
macro is used to specify the size of a record structure. The third
argument allows you to specify keyword directives for the selection
and is described in more detail below.
Following the first three required arguments is a variable length list of
column specifications. Each column specification will consist of four
arguments:
=over 4
=item *
B<name>: the name of the column
=item *
B<type>: the data type of the column as it will be stored in
the user record struct (not the data type of the input file). The
following basic data types are recognized:
=over 4
=item *
A: ASCII characters
=item *
B: unsigned 8-bit char
=item *
I: signed 16-bit int
=item *
U: unsigned 16-bit int (not standard FITS)
=item *
J: signed 32-bit int
=item *
V: unsigned 32-bit int (not standard FITS)
=item *
E: 32-bit float
=item *
D: 64-bit float
=back
The syntax used is similar to that which defines the TFORM parameter
in FITS binary tables. That is, a numeric repeat value can precede
the type character, so that "10I" means a vector of 10 short ints, "E"
means a single precision float, etc. Note that the column value from
the input file will be converted to the specified data type as the
data is read by
FunTableRowGet().
[ A short digression regarding bit-fields: Special attention is
required when reading or writing the FITS bit-field type
("X"). Bit-fields almost always have a numeric repeat character
preceding the 'X' specification. Usually this value is a multiple of 8
so that bit-fields fit into an integral number of bytes. For all
cases, the byte size of the bit-field B is (N+7)/8, where N is the
numeric repeat character.
A bit-field is most easily declared in the user struct as an array of
type char of size B as defined above. In this case, bytes are simply
moved from the file to the user space. If, instead, a short or int
scalar or array is used, then the algorithm for reading the bit-field
into the user space depends on the size of the data type used along
with the value of the repeat character. That is, if the user data
size is equal to the byte size of the bit-field, then the data is
simply moved (possibly with endian-based byte-swapping) from one to
the other. If, on the other hand, the data storage is larger than the
bit-field size, then a data type cast conversion is performed to move
parts of the bit-field into elements of the array. Examples will help
make this clear:
=over 4
=item *
If the file contains a 16X bit-field and user space specifies a 2B
char array[2], then the bit-field is moved directly into the char array.
=item *
If the file contains a 16X bit-field and user space specifies a 1I
scalar short int, then the bit-field is moved directly into the short int.
=item *
If the file contains a 16X bit-field and user space specifies a 1J
scalar int, then the bit-field is type-cast to unsigned int before
being moved (use of unsigned avoids possible sign extension).
=item *
If the file contains a 16X bit-field and user space specifies a 2J
int array[2], then the bit-field is handled as 2 chars, each of which
are type-cast to unsigned int before being moved (use of unsigned
avoids possible sign extension).
=item *
If the file contains a 16X bit-field and user space specifies a 1B
char, then the bit-field is treated as a char, i.e., truncation will
occur.
=item *
If the file contains a 16X bit-field and user space specifies a 4J
int array[4], then the results are undetermined.
=back
For all user data types larger than char, the bit-field is byte-swapped
as necessary to convert to native format, so that bits in the
resulting data in user space can be tested, masked, etc. in the same
way regardless of platform.]
In addition to setting data type and size, the B<type>
specification allows a few ancillary parameters to be set, using the
full syntax for B<type>:
[@][n]<type>[[['B']poff]][:[tlmin[:tlmax[:binsiz]]]]
The special character "@" can be prepended to this specification to
indicated that the data element is a pointer in the user record,
rather than an array stored within the record.
The [n] value is an integer that specifies the
number of elements that are in this column (default is 1). TLMIN,
TLMAX, and BINSIZ values also can be specified for this column after
the type, separated by colons. If only one such number is specified,
it is assumed to be TLMAX, and TLMIN and BINSIZ are set to 1.
The [poff] value can be used to specify the offset into an
array. By default, this offset value is set to zero and the data
specified starts at the beginning of the array. The offset usually
is specified in terms of the data type of the column. Thus an offset
specification of [5] means a 20-byte offset if the data type is a
32-bit integer, and a 40-byte offset for a double. If you want to
specify a byte offset instead of an offset tied to the column data type,
precede the offset value with 'B', e.g. [B6] means a 6-bye offset,
regardless of the column data type.
The [poff] is especially useful in conjunction with the pointer @
specification, since it allows the data element to anywhere stored
anywhere in the allocated array. For example, a specification such as
"@I[2]" specifies the third (i.e., starting from 0) element in the
array pointed to by the pointer value. A value of "@2I[4]" specifies
the fifth and sixth values in the array. For example, consider the
following specification:
typedef struct EvStruct{
short x[4], *atp;
} *Event, EventRec;
/* set up the (hardwired) columns */
FunColumnSelect( fun, sizeof(EventRec), NULL,
"2i", "2I ", "w", FUN_OFFSET(Event, x),
"2i2", "2I[2]", "w", FUN_OFFSET(Event, x),
"at2p", "@2I", "w", FUN_OFFSET(Event, atp),
"at2p4", "@2I[4]", "w", FUN_OFFSET(Event, atp),
"atp9", "@I[9]", "w", FUN_OFFSET(Event, atp),
"atb20", "@I[B20]", "w", FUN_OFFSET(Event, atb),
NULL);
Here we have specified the following columns:
=over 4
=item *
2i: two short ints in an array which is stored as part the
record
=item *
2i2: the 3rd and 4th elements of an array which is stored
as part of the record
=item *
an array of at least 10 elements, not stored in the record but
allocated elsewhere, and used by three different columns:
=over 4
=item *
at2p: 2 short ints which are the first 2 elements of the allocated array
=item *
at2p4: 2 short ints which are the 5th and 6th elements of
the allocated array
=item *
atp9: a short int which is the 10th element of the allocated array
=back
=item *
atb20: a short int which is at byte offset 20 of another allocated array
=back
In this way, several columns can be specified, all of which are in a
single array. B<NB>: it is the programmer's responsibility to
ensure that specification of a positive value for poff does not point
past the end of valid data.
=item *
B<read/write mode>: "r" means that the column is read from an
input file into user space by
FunTableRowGet(), "w" means that
the column is written to an output file. Both can specified at the same
time.
=item *
B<offset>: the offset into the user data to store
this column. Typically, the macro FUN_OFFSET(recname, colname) is used
to define the offset into a record structure.
=back
When all column arguments have been specified, a final NULL argument
must added to signal the column selection list.
As an alternative to the varargs
FunColumnSelect()
routine, a non-varargs routine called
FunColumnSelectArr()
also is available. The first three arguments (fun, size, plist) of this
routine are the same as in
FunColumnSelect().
Instead of a variable
argument list, however,
FunColumnSelectArr()
takes 5 additional arguments. The first 4 arrays arguments contain the
names, types, modes, and offsets, respectively, of the columns being
selected. The final argument is the number of columns that are
contained in these arrays. It is the user's responsibility to free
string space allocated in these arrays.
Consider the following example:
typedef struct evstruct{
int status;
float pi, pha, *phas;
double energy;
} *Ev, EvRec;
FunColumnSelect(fun, sizeof(EvRec), NULL,
"status", "J", "r", FUN_OFFSET(Ev, status),
"pi", "E", "r", FUN_OFFSET(Ev, pi),
"pha", "E", "r", FUN_OFFSET(Ev, pha),
"phas", "@9E", "r", FUN_OFFSET(Ev, phas),
NULL);
Each time a row is read into the Ev struct, the "status" column is
converted to an int data type (regardless of its data type in the
file) and stored in the status value of the struct. Similarly, "pi"
and "pha", and the phas vector are all stored as floats. Note that the
"@" sign indicates that the "phas" vector is a pointer to a 9 element
array, rather than an array allocated in the struct itself. The row
record can then be processed as required:
/* get rows -- let routine allocate the row array */
while( (ebuf = (Ev)FunTableRowGet(fun, NULL, MAXROW, NULL, &got)) ){
/* process all rows */
for(i=0; i<got; i++){
/* point to the i'th row */
ev = ebuf+i;
ev->pi = (ev->pi+.5);
ev->pha = (ev->pi-.5);
}
FunColumnSelect()
can also be called to define "writable" columns in order to generate a FITS
Binary Table, without reference to any input columns. For
example, the following will generate a 4-column FITS binary table when
FunTableRowPut() is used to
write Ev records:
typedef struct evstruct{
int status;
float pi, pha
double energy;
} *Ev, EvRec;
FunColumnSelect(fun, sizeof(EvRec), NULL,
"status", "J", "w", FUN_OFFSET(Ev, status),
"pi", "E", "w", FUN_OFFSET(Ev, pi),
"pha", "E", "w", FUN_OFFSET(Ev, pha),
"energy", "D", "w", FUN_OFFSET(Ev, energy),
NULL);
All columns are declared to be write-only, so presumably the column
data is being generated or read from some other source.
In addition,
FunColumnSelect()
can be called to define B<both> "readable" and "writable" columns.
In this case, the "read" columns
are associated with an input file, while the "write" columns are
associated with the output file. Of course, columns can be specified as both
"readable" and "writable", in which case they are read from input
and (possibly modified data values are) written to the output.
The
FunColumnSelect()
call itself is made by passing the input Funtools handle, and it is
assumed that the output file has been opened using this input handle
as its
Funtools reference handle.
Consider the following example:
typedef struct evstruct{
int status;
float pi, pha, *phas;
double energy;
} *Ev, EvRec;
FunColumnSelect(fun, sizeof(EvRec), NULL,
"status", "J", "r", FUN_OFFSET(Ev, status),
"pi", "E", "rw", FUN_OFFSET(Ev, pi),
"pha", "E", "rw", FUN_OFFSET(Ev, pha),
"phas", "@9E", "rw", FUN_OFFSET(Ev, phas),
"energy", "D", "w", FUN_OFFSET(Ev, energy),
NULL);
As in the "read" example above, each time an row is read into the Ev
struct, the "status" column is converted to an int data type
(regardless of its data type in the file) and stored in the status
value of the struct. Similarly, "pi" and "pha", and the phas vector
are all stored as floats. Since the "pi", "pha", and "phas" variables
are declared as "writable" as well as "readable", they also will be
written to the output file. Note, however, that the "status" variable
is declared as "readable" only, and hence it will not be written to
an output file. Finally, the "energy" column is declared as
"writable" only, meaning it will not be read from the input file. In
this case, it can be assumed that "energy" will be calculated in the
program before being output along with the other values.
In these simple cases, only the columns specified as "writable" will
be output using
FunTableRowPut(). However,
it often is the case that you want to merge the user columns back in
with the input columns, even in cases where not all of the input
column names are explicitly read or even known. For this important
case, the B<merge=[type]> keyword is provided in the plist string.
The B<merge=[type]> keyword tells Funtools to merge the columns from
the input file with user columns on output. It is normally used when
an input and output file are opened and the input file provides the
Funtools reference handle
for the output file. In this case, each time
FunTableRowGet() is called, the
raw input rows are saved in a special buffer. If
FunTableRowPut() then is called
(before another call to
FunTableRowGet()), the contents
of the raw input rows are merged with the user rows according to the
value of B<type> as follows:
=over 4
=item *
B<update>: add new user columns, and update value of existing ones (maintaining the input data type)
=item *
B<replace>: add new user columns, and replace the data type
and value of existing ones. (Note that if tlmin/tlmax values are not
specified in the replacing column, but are specified in the original
column being replaced, then the original tlmin/tlmax values are used
in the replacing column.)
=item *
B<append>: only add new columns, do not "replace" or "update" existing ones
=back
Consider the example above. If B<merge=update> is specified in the
plist string, then "energy" will be added to the input columns, and
the values of "pi", "pha", and "phas" will be taken from the user
space (i.e., the values will be updated from the original values, if
they were changed by the program). The data type for "pi", "pha", and
"phas" will be the same as in the original file. If
B<merge=replace> is specified, both the data type and value of
these three input columns will be changed to the data type and value
in the user structure. If B<merge=append> is specified, none of
these three columns will be updated, and only the "energy" column will
be added. Note that in all cases, "status" will be written from the
input data, not from the user record, since it was specified as read-only.
Standard applications will call
FunColumnSelect()
to define user columns. However, if this routine is not called, the
default behavior is to transfer all input columns into user space. For
this purpose a default record structure is defined such that each data
element is properly aligned on a valid data type boundary. This
mechanism is used by programs such as fundisp and funtable to process
columns without needing to know the specific names of those columns.
It is not anticipated that users will need such capabilities (contact
us if you do!)
By default, FunColumnSelect()
reads/writes rows to/from an "array of structs", where each struct contains
the column values for a single row of the table. This means that the
returned values for a given column are not contiguous. You can
set up the IO to return a "struct of arrays" so that each of the
returned columns are contiguous by specifying B<org=structofarrays>
(abbreviation: B<org=soa>) in the plist.
(The default case is B<org=arrayofstructs> or B<org=aos>.)
For example, the default setup to retrieve rows from a table would be
to define a record structure for a single event and then call
FunColumnSelect()
as follows:
typedef struct evstruct{
short region;
double x, y;
int pi, pha;
double time;
} *Ev, EvRec;
got = FunColumnSelect(fun, sizeof(EvRec), NULL,
"x", "D:10:10", mode, FUN_OFFSET(Ev, x),
"y", "D:10:10", mode, FUN_OFFSET(Ev, y),
"pi", "J", mode, FUN_OFFSET(Ev, pi),
"pha", "J", mode, FUN_OFFSET(Ev, pha),
"time", "1D", mode, FUN_OFFSET(Ev, time),
NULL);
Subsequently, each call to
FunTableRowGet()
will return an array of structs, one for each returned row. If instead you
wanted to read columns into contiguous arrays, you specify B<org=soa>:
typedef struct aevstruct{
short region[MAXROW];
double x[MAXROW], y[MAXROW];
int pi[MAXROW], pha[MAXROW];
double time[MAXROW];
} *AEv, AEvRec;
got = FunColumnSelect(fun, sizeof(AEvRec), "org=soa",
"x", "D:10:10", mode, FUN_OFFSET(AEv, x),
"y", "D:10:10", mode, FUN_OFFSET(AEv, y),
"pi", "J", mode, FUN_OFFSET(AEv, pi),
"pha", "J", mode, FUN_OFFSET(AEv, pha),
"time", "1D", mode, FUN_OFFSET(AEv, time),
NULL);
Note that the only modification to the call is in the plist string.
Of course, instead of using staticly allocated arrays, you also can specify
dynamically allocated pointers:
/* pointers to arrays of columns (used in struct of arrays) */
typedef struct pevstruct{
short *region;
double *x, *y;
int *pi, *pha;
double *time;
} *PEv, PEvRec;
got = FunColumnSelect(fun, sizeof(PEvRec), "org=structofarrays",
"$region", "@I", mode, FUN_OFFSET(PEv, region),
"x", "@D:10:10", mode, FUN_OFFSET(PEv, x),
"y", "@D:10:10", mode, FUN_OFFSET(PEv, y),
"pi", "@J", mode, FUN_OFFSET(PEv, pi),
"pha", "@J", mode, FUN_OFFSET(PEv, pha),
"time", "@1D", mode, FUN_OFFSET(PEv, time),
NULL);
Here, the actual storage space is either allocated by the user or by the
FunColumnSelect() call).
In all of the above cases, the same call is made to retrieve rows, e.g.:
buf = (void *)FunTableRowGet(fun, NULL, MAXROW, NULL, &got);
However, the individual data elements are accessed differently.
For the default case of an "array of structs", the
individual row records are accessed using:
for(i=0; i<got; i++){
ev = (Ev)buf+i;
fprintf(stdout, "%.2f\t%.2f\t%d\t%d\t%.4f\t%.4f\t%21.8f\n",
ev->x, ev->y, ev->pi, ev->pha, ev->dx, ev->dy, ev->time);
}
For a struct of arrays or a struct of array pointers, we have a single struct
through which we access individual columns and rows using:
aev = (AEv)buf;
for(i=0; i<got; i++){
fprintf(stdout, "%.2f\t%.2f\t%d\t%d\t%.4f\t%.4f\t%21.8f\n",
aev->x[i], aev->y[i], aev->pi[i], aev->pha[i],
aev->dx[i], aev->dy[i], aev->time[i]);
}
Support for struct of arrays in the
FunTableRowPut()
call is handled analogously.
See the evread example code
and
evmerge example code
for working examples of how
FunColumnSelect() is used.
=head1 SEE ALSO
See funtools(n) for a list of Funtools help pages
=cut
|