diff options
Diffstat (limited to 'doc/files.html')
| -rw-r--r-- | doc/files.html | 627 |
1 files changed, 627 insertions, 0 deletions
diff --git a/doc/files.html b/doc/files.html new file mode 100644 index 0000000..0d903a4 --- /dev/null +++ b/doc/files.html @@ -0,0 +1,627 @@ +<!-- =defdoc funfiles funfiles n --> +<HTML> +<HEAD> +<TITLE>Funtools Data Files</TITLE> +</HEAD> +<BODY> + +<!-- =section funfiles NAME --> +<H2><A NAME="files">FunFiles: Funtools Data Files</A></H2> + +<!-- =section funfiles SYNOPSIS --> +<H2>Summary</H2> +This document describes the data file formats (FITS, array, raw +events) as well as the file types (gzip, socket, etc.) supported +by Funtools. + +<!-- =section funfiles DESCRIPTION --> +<H2>Description</H2> +<P> +Funtools supports FITS images and binary tables, and binary files +containing array (homogeneous) data or event (heterogeneous) data. +IRAF-style brackets are appended to the filename to specify various +kinds of information needed to characterize these data: +<PRE> + file[ext|ind|ARRAY()|EVENTS(),section][filters] + or + file[ext|ind|ARRAY()|EVENTS(),section,filters] +</PRE> +where: +<UL> +<LI> <B>file</B> is the Funtools file name +<LI> <B>ext</B> is the FITS extension name +<LI> <B>ind</B> is the FITS extension number +<LI> <B>ARRAY()</B> is an array specification +<LI> <B>EVENTS()</B> is an event specification +<LI> <B>section</B> is the image section specification +<LI> <B>filters</B> are spatial region and table (row) filters +</UL> + +<H2><A NAME="formats">Supported Data Formats</A></H2> +<P> +Funtools programs (and the underlying libraries) support the +following data file formats: +<UL> +<LI> FITS images (and image extensions) +<LI> FITS binary tables +<LI> binary files containing an array of homogeneous data +<LI> binary files containing events, i.e. records of heterogeneous data +<LI> column-based text files, which are documented <A HREF="./text.html">here</A> +<LI> non-disk files and lists of files +</UL> +Information needed to identify and characterize +the event or image data can be specified on the command line +using IRAF-style bracket notation appended to the filename: +<PRE> + foo.fits # open FITS default extension + image.fits[3] # open FITS extension #3 + events.fits[EVENTS] # open EVENTS extension + array.file[ARRAY(s1024)] # open 1024x1024 short array + events.file[EVENTS(x:1024,y:1024...)] # open non-FITS event list +</PRE> +Note that in many Unix shells (e.g., csh and tcsh), filenames must +be enclosed in quotes to protect the brackets from shell processing. + +<H2><A NAME="fits">FITS Images and Binary Tables</A></H2> +<P> +When <A HREF="./library.html#funopen">FunOpen()</A> opens a FITS file +without a bracket specifier, the default behavior is to look for a +valid image in the primary HDU. In the absence of a primary image, +Funtools will try to open an extension named either <B>EVENTS</B> or +<B>STDEVT</B>, if one of these exists. This default behavior supports +both FITS image processing and standard X-ray event list processing +(which, after all, is what we at SAO/HEAD do). + +<P> +In order to open a FITS binary table or image extension explicitly, it +is necessary to specify either the extension name or the extension +number in brackets: +<PRE> + foo.fits[1] # open extension #1: the primary HDU + foo.fits[3] # open extension #3 of a FITS file + foo.fits[GTI] # open GTI extension of a FITS file +</PRE> +The ext argument specifies the name of the FITS extension (i.e. the +value of the EXTENSION header parameter in a FITS extension), while +the index specifies the value of the FITS EXTVER header parameter. +Following FITS conventions, extension numbers start at 1. + +<P> +When a FITS data file is opened for reading using +<A HREF="./library.html#funopen">FunOpen()</A>, the specified extension +is automatically located and is used to initialize the Funtools internal +data structures. + +<H2><A NAME="events">Non-FITS Raw Event Files</A></H2> + +In addition to FITS tables, Funtools programs and libraries can operate +on non-FITS files containing heterogeneous event records. To specify +such an event file, use: + +<UL> +<LI> file[EVENTS(event-spec)] +<LI> file[EVENTS()] +</UL> +where <B>event-spec</B> is a string that specified the names, data +types, and optional image dimensions for each element of the event +record: +<UL> +<LI> [name]:[n][type]:[(lodim:)hidim] +</UL> + +<P> +Data types follow standard conventions for FITS binary tables, but include +two extra unsigned types ('U' and 'V'): +<UL> +<LI> <B>B</B> -- unsigned 8-bit char +<LI> <B>I</B> -- signed 16-bit int +<LI> <B>J</B> -- signed 32-bit int +<LI> <B>K</B> -- signed 64-bit int +<LI> <B>E</B> -- 32-bit float +<LI> <B>D</B> -- 64-bit float +<LI> <B>U</B> -- unsigned 16-bit int +<LI> <B>V</B> -- unsigned 32-bit int +</UL> +An optional integer value <B>n</B> can be prefixed to the type to indicate +that the element is an array of n values. For example: +<PRE> + foo.fits[EVENTS(x:I,y:I,status:4J)] +</PRE> +defines x and y as 16-bit ints and status as an array of 4 32-bit ints. + +<P> +Furthermore, image dimensions can be attached to the event specification +in order to tell Funtools how to bin the events into an image. They +follow the conventions for the FITS TLMIN/TLMAX keywords. If the low +image dimension is not specified, it defaults to 1. Thus: + +<UL> +<LI> RAWX:J:1:100 +<LI> RAWX:J:100 +</UL> +both specify that the dimension of this column runs from 1 to 100. + +<P> +NB: it is required that all padding be specified in the record +definition. Thus, when writing out whole C structs instead of +individual record elements, great care must be taken to include +the compiler-added padding in the event definition. + +<P> +For example, suppose a FITS binary table has the following set of column +definitions: +<PRE> + TTYPE1 = 'X ' / Label for field + TFORM1 = '1I ' / Data type for field + TLMIN1 = 1 / Min. axis value + TLMAX1 = 10 / Max. axis value + TTYPE2 = 'Y ' / Label for field + TFORM2 = '1I ' / Data type for field + TLMIN2 = 2 / Min. axis value + TLMAX2 = 11 / Max. axis value + TTYPE3 = 'PHA ' / Label for field + TFORM3 = '1I ' / Data type for field + TTYPE4 = 'PI ' / Label for field + TFORM4 = '1J ' / Data type for field + TTYPE5 = 'TIME ' / Label for field + TFORM5 = '1D ' / Data type for field + TTYPE6 = 'DX ' / Label for field + TFORM6 = '1E ' / Data type for field + TLMIN6 = 1 / Min. axis value + TLMAX6 = 10 / Max. axis value + TTYPE7 = 'DY ' / Label for field + TFORM7 = '1E ' / Data type for field + TLMIN7 = 3 / Min. axis value + TLMAX7 = 12 / Max. axis value +</PRE> + +An raw event file containing these same data would have the event +specification: +<PRE> + EVENTS(X:I:10,Y:I:2:11,PHA:I,PI:J,TIME:D,DX:E:10,DY:E:3:12) +</PRE> + +<P> +If no event specification string is included within the EVENTS() operator, +then the event specification is taken from the <B>EVENTS</B> environment +variable: +<PRE> + setenv EVENTS "X:I:10,Y:I:10,PHA:I,PI:J,TIME:D,DX:E:10,DY:E:10" +</PRE> + +<P> +In addition to knowing the data structure, it is necessary to know the +<EM>endian</EM> ordering of the data, i.e., whether or not the data is +in <EM>bigendian</EM> format, so that we can convert to the native +format for this platform. This issue does not arise for FITS Binary +Tables because all FITS files use big-endian ordering, regardless of +platform. But for non-FITS data, big-endian data produced on a Sun +workstation but read on a Linux PC needs to be byte-swapped, since PCs +use little-endian ordering. To specify an ordering, use the +<EM>bigendian=</EM> or <EM>endian=</EM> keywords on the command-line +or the EVENTS_BIGENDIAN or EVENTS_ENDIAN environment variables. The +value of the <EM>bigendian</EM> variables should be "true" or "false", +while the value of the <EM>endian</EM> variables should be "little" or +"big". + +<P> +For example, a PC can access data produced by a Sun using: +<PRE> + hrc.nepr[EVENTS(),bigendian=true] +or + hrc.nepr[EVENTS(),endian=big] +or + setenv EVENTS_BIGENDIAN true +or + setenv EVENTS_ENDIAN big +</PRE> +If none of these are specified, the data are assumed to follow the +format for that platform and no byte-swapping is performed. + +<H2><A NAME="arrays">Non-FITS Array Files</A></H2> + +In addition to FITS images, Funtools programs and libraries can operate +on non-FITS files containing arrays of homogeneous data. To specify +an array file, use: +<UL> +<LI> file[ARRAY(array-spec)] +<LI> file[ARRAY()] +</UL> + +where array-spec is of the form: +<UL> +<LI> [type][dim1][.dim2][:skip][endian] +</UL> + +and where [type] is: +<UL> +<LI> b (8-bit unsigned char) +<LI> s (16-bit short int) +<LI> u (16-bit unsigned short int) +<LI> i (32-bit int) +<LI> r,f (32-bit float) +<LI> d (64-bit float) +</UL> + +<P> +The dim1 specification is required, but dim2 is optional and defaults +to dim1. The skip specification is optional and defaults to 0. The +optional endian specification can be 'l' or 'b' and defaults to the +endian type for the current machine. + +<P> +If no array specification is included within the ARRAY() operator, +then the array specification is taken from the <B>ARRAY</B> environment +variable. For example: + +<PRE> + foo.arr[ARRAY(r512)] # bitpix=-32 dim1=512 dim2=512 + foo.arr[ARRAY(r512.400)] # bitpix=-32 dim1=512 dim2=400 + foo.arr[ARRAY(r512.400]) # bitpix=-32 dim1=512 dim2=400 + foo.arr[ARRAY(r512.400:2880)] # bitpix=-32 dim1=512 dim2=400 skip=2880 + foo.arr[ARRAY(r512l)] # bitpix=-32 dim1=512 dim2=512 endian=little + setenv ARRAY "r512.400:2880" + foo.arr[ARRAY()] # bitpix=-32 dim1=512 dim2=400 skip=2880 +</PRE> + +<H2><A NAME="sections">Specifying Image Sections</A></H2> + +Once a data file (and possibly, a FITS extension) has been specified, +the next (optional) part of a bracket specification can be used to +select image <B>section</B> information, i.e., to specify the x,y +limits of an image section, as well as the blocking factor to apply to +that section. This information can be added to any file specification but +only is used by Funtools image processing routines. + +<P> +The format of the image section specification is one of the following: +<UL> +<LI> file[xy0:xy1,block] +<LI> file[x0:x1,y0:y1,block] +<LI> file[x0:x1,*,block] +<LI> file[*,y0:y1,block] +<LI> file[*,block] +</UL> +where the limit values can be ints or "*" for default. A single "*" +can be used instead of val:val, as shown. Note that blocking is +applied to the section after it is extracted. + +<P> +In addition to image sections specified by the lo and hi x,y limits, image +sections using center positions can be specified: +<UL> +<LI> file[dim1@xcen,dim2@ycen] +<LI> file[xdim2@xcen@ycen] +<LI> file[dim1@xcen,dim2@ycen,block] +<LI> file[dim@xcen@ycen,block] +</UL> +Note that the (float) values for dim, dim1, dim2, xcen, ycen must be +specified or else the expression does not make sense! + +<P> +In all cases, block is optional and defaults to 1. An 's' or 'a' can +be appended to signify "sum" or "average" blocking (default is "sum"). +Section specifications are given in image coordinates by default. If you +wish to specify physical coordinates, add a 'p' as the last character +of the section specification, before the closing bracket. +For example: +<PRE> +<UL> +<LI> file[-8:-7,-8:-7p] +<LI> file[-8:-7,-8:-7,2p] +</UL> +</PRE> +A section can be specified in any Funtools file name. If the operation +to be applied to that file is an imaging operation, then the +specification will be utilized. If the operation is purely a table +operation, then the section specification is ignored. + +<P> +Do not be confused by: +<PRE> + foo.fits[2] + foo.fits[*,2] +</PRE> +The former specifies opening the second extension of the FITS file. +The latter specifies application of block 2 to the image section. + +<P> +Note that the section specification must come after +any of FITS <B>ext</B> name or <B>ind</B> number, +but all sensible defaults are supported: +<UL> +<LI> file[ext] +<LI> file[ext,index] +<LI> file[index] +<LI> file[ext,section] +<LI> file[ext,index,section] +<LI> file[index,section] +<LI> file[section] +</UL> + +<H2><A NAME="binning">Binning FITS Binary Tables and Non-FITS Event Files</H2> + +If a FITS binary table or a non-FITS raw event file is to be binned +into a 2D image (e.g., using the +<A HREF="./programs.html#funimage">funimage</A> +program), it is necessary to specify the two columns to be used for the +binning, as well as the dimensions of the image. Funtools first looks +for a specifier of the form: +<PRE> + bincols=([xnam[:tlmin[:tlmax:[binsiz]]]],[ynam[:tlmin[:tlmax[:binsiz]]]]) +</PRE> +in bracket syntax, and uses the column names thus specified. The tlmin, tlmax, +and binsiz specifiers determine the image binning dimensions using: +<PRE> + dim = (tlmax - tlmin)/binsiz (floating point data) + dim = (tlmax - tlmin)/binsiz + 1 (integer data) +</PRE> +These tlmin, tlmax, and binsiz specifiers can be omitted if TLMIN, +TLMAX, and TDBIN header parameters are present in the FITS binary +table header, respectively. If only one parameter is specified, it is +assumed to be tlmax, and tlmin defaults to 1. If two parameters are +specified, they are assumed to be tlmin and tlmax. + +For example, to bin an HRC event list columns "VPOS" and "UPOS", use: +<PRE> + hrc.nepr[bincols=(VPOS,UPOS)] +</PRE> +or +<PRE> + hrc.nepr[bincols=(VPOS:49152,UPOS:4096)] +</PRE> +Note that you can optionally specify the dimensions of these columns +to cover cases where neither TLMAX keywords are defined in +the header. If either dimension is specified, then both must be specified. + +<P> +You can set the FITS_BINCOLS or EVENTS_BINCOLS environment variable as +an alternative to adding the "bincols=" specifier to each file name +for FITS binary tables and raw event files, respectively. If no +binning keywords or environment variables are specified, or if the +specified columns are not in the binary table, the Chandra parameters +CPREF (or PREFX) are searched for in the FITS binary table header. +Failing this, columns named "X" and "Y" are sought. If these are not +found, the code looks for columns containing the characters "X" and +"Y". Thus, you can bin on "DETX" and "DETX" columns without +specifying them, if these are the only column names containing the "X" +and "Y" characters. + +<p> +Ordinarily, each event or row contributes one count to an image pixel +during the 2D binning process. Thus, if five events all have the same +(x,y) position, the image pixel value for that position will have a +value of five. It is possible to specify a variable contribution +for each event by using the vcol=[colname] filter spec: +<PRE> + vcol=[colname] +</PRE> +The vcol colname is a column containing a numeric value in each event row +that will be used as the contribution of the given event to its image +pixel. For example, consider an event file that has the following content: +<PRE> + x:e:4 y:e:4 v:e + ------ ------ ---- + 1 1 1.0 + 2 2 2.0 + 3 3 3.0 + 4 4 0.0 + 1 1 1.0 + 2 2 2.0 + 3 3 3.0 + 4 4 4.0 +</PRE> +There are two events with x,y value of (1,1) so ordinarily a 2D image will +have a value of 2 in the (1,1) pixel. If the v column is specified as the +value column: +<PRE> + foo.fits'[vcol=v]' +</PRE> +then each pixel will contain the additive sum of the associated (x,y) +column values from the v column. For example, image pixel (1,1) will +contain 1. + 1. = 2, image pixel (2,2) will contain (2 + 2) = 4, etc. + +<p> +An important variation on the use of a value column to specify the +contribution an event makes to an image pixel is when the value column +contains the reciprocal of the event contribution. For this case, the +column name should be prefixed with a / (divide sign) thus: +<PRE> + foo.fits'[vcol=/v]' +</PRE> +Each image pixel value will then be the sum of the reciprocals of the value +column. A zero in the value column results in NaN (not a number). +Thus, in the above example, image pixel (1.1) will contain 1/1 + 1/1 = 2, +image pixel (2,2) will contain (1/2 + 1/2) = 1, etc. Image pixel (4,4) +will contain (1/0 + 1/4) = NaN. + +<p> +You can set the FITS_VCOL or EVENTS_VCOL environment variable as +an alternative to adding the "vcol=" specifier to each file name +for FITS binary tables and raw event files, respectively. + +<P> +Finally, when binning events, the data type of the resulting 2D image +must be specified. This can be done with the "bitpix=[n]" keyword in +the bracket specification. For example: +<PRE> + events.fits[bincols=(VPOS,UPOS),bitpix=-32] +</PRE> +will create a floating point image binned on columns VPOS and UPOS. +If no bitpix keyword is specified, bitpix=32 is assumed. As with +bincols values, you also can use the FITS_BITPIX and EVENTS_BITPIX +environment variables to set this value for FITS binary tables and +raw event files, respectively. + +<P> +The <b>funimage</b> program also allows you to create a 1D image projection +along any column of a table by using the <b>bincols=[column]</b> +filter specification and specifying a single column. +For example, the following command projects a 1D image along +the chipx column of a table: +<PRE> + funimage ev.fits'[bincols=chipx]' im.fits +</PRE> +See <A HREF="./programs.html#funimage">funimage</A> for more +information about creating 1D and 2D images. + +<P> +Finally, please note that Funtools supports most FITS standards. +We will add missing support as required by the community. In general, +however, we do not support non-standard extensions. For example, we +sense the presence of the binary table 'variable length array' +proposed extension and we pass it along when copying and filtering +files, but we do not process it. We will add support for new standards +as they become official. + +<H2><A NAME="filters">Table and Spatial Region Filters</H2> +<P> +Note that, in addition extensions and image sections, Funtools bracket +notation can be used to specify table and spatial region filters. These +filters are always placed after the image section information. They +can be specified in the same bracket or in a separate bracket +immediately following: +<UL> +<LI> file[ext|ind|ARRAY()|EVENTS(),section][filters] +<LI> file[ext|ind|ARRAY()|EVENTS(),section,filters] +</UL> +where: +<UL> +<LI> <B>file</B> is the Funtools file name +<LI> <B>ARRAY()</B> is an array specification +<LI> <B>EVENTS()</B> is an event list specification +<LI> <B>ext</B> is the FITS extension name +<LI> <B>ind</B> is the FITS extension number +<LI> <B>section</B> is the image section to extract +<LI> <B>filters</B> are spatial region and table (row) filters to apply +</UL> + +The topics of table and region filtering are covered in detail in: +<UL> +<LI><A HREF="./filters.html">Table Filtering</A> +<LI><A HREF="./regions.html">Spatial Region Filtering</A> +</UL> + +<H2><A NAME="types">Disk Files and Other Supported File Types</A></H2> +<P> +The specified <B>file</B> usually is an ordinary disk file. In +addition, gzip'ed files are supported in Funtools: gzip'ed input files +are automatically uncompressed as they are read, and gzip'ed output +files are compressed as they are written. NB: if a FITS binary table +is written in gzip format, the number of rows in the table will be set +to -1. Such a file will work with Funtools programs but will not work +with other FITS programs such as ds9. + +<P> +The special keywords "stdin" and "stdout" designate Unix standard +input and standard output, respectively. The string "-" (hyphen) will +be taken to mean "stdin" if the file is opened for reading and +"stdout" if the file is opened for writing. + +<P> +A file also can be an INET socket on the same or another machine using +the syntax: +<PRE> + machine:port +</PRE> +Thus, for example: +<PRE> + karapet:1428 +</PRE> +specifies that I/O should be performed to/from port 1428 on the +machine karapet. If no machine name is specified, the default is to +use the current machine: +<PRE> + :1428 +</PRE> +This means to open port 1428 on the current machine. Socket support +allows you to generate a distributed pipe: +<PRE> + on karapet: funtask1 in.fits bynars:1428 + on bynars: funtask2 :1428 out.fits +</PRE> +The socket mechanism thus supports simple parallel processing using +<B>process decomposition</B>. Note that parallel processing using +<B>data decomposition</B> is supported via the <B>section</B> specifier (see +below), and the <B>row#</B> specifier, which is part of +<A HREF="./filters.html">Table Filtering</A>. + +<P> +A file also can be a pointer to shared memory using the syntax: +<PRE> + shm:[id|@key][:size] +</PRE> +A shared memory segment is specified with a <B>shm:</B> prefix, +followed by either the shared memory id or the shared memory key +(where the latter is prefixed by the '@' character). The size (in +bytes) of the shared memory segment can then be appended (preceded by +the ':' character). If the size specification is absent, the code will +attempt to determine the length automatically. + +If the open mode contains the string "w+", then the memory segment will be +created if it does not exist. (It also will be released and deleted when the +file is closed.) In the case where a memory segment is being created, the +length of the segment is required. + +<P> +A file also can be Unix piped command (i.e. a program to run) using the syntax: +<PRE> + "pipe: command arg1 ... argn" +</PRE> +The output from the command must be a valid FITS file. It is important +to use quotes to protect spaces so that command arguments are passed +correctly. A silly example is: +<PRE> + fundisp "pipe: funtable 'foo.fits[cir 512 512 .1]' stdout" +</PRE> +This seemed like a good idea at the time ... + +<H2><A NAME="types">Lists of Files</A></H2> + +<P> +Funtools also will process a list of files as a single file using the +syntax: +<PRE> + "list: file1 file2 ... filen" +</PRE> +The files in the list are separated by whitespace. Any of the +above file types can be used. For example, if two files, foo1.fits and +foo2.fits, are part of the same observation, they can be processed as +a single file (using their own filters): +<PRE> + fundisp "list: foo1.fits[cir(512,512,10)] foo2.fits[cir(511,511,10)]" + X Y PHA PI TIME DX DY + -------- -------- -------- -------- --------------------- -------- -------- + 512 512 6 7 79493997.45854475 578 574 + 512 512 8 9 79494575.58943175 579 573 + 512 512 5 6 79493631.03866175 578 575 + 512 512 5 5 79493290.86521725 578 575 + 512 512 8 9 79493432.00990875 579 573 + 511 511 5 5 79488631.09462625 580 575 + 511 511 10 11 79488780.60006675 580 573 + 511 511 4 4 79494562.35474326 580 575 + 511 511 6 6 79488203.01561825 580 575 + 511 511 6 6 79488017.99730176 580 575 + 511 511 4 4 79494332.45355175 580 575 + 511 511 9 10 79492685.94014275 581 574 + 511 511 5 5 79487708.71298325 580 575 + 511 511 8 9 79493719.00160225 581 573 +</PRE> +Again, note that it is important to avoid spaces in the filters +because the list separator also is whitespace. To protect whitespace +in a filter, enclose the file specification in quotes: +<PRE> + fundisp "list: 'foo1.fits[cir 512 512 .1]' foo2.fits[cir(511,511,.1)]" +</PRE> + +<!-- =section funfiles SEE ALSO --> +<!-- =text See funtools(n) for a list of Funtools help pages --> +<!-- =stop --> + +<P> +<A HREF="./help.html">Go to Funtools Help Index</A> + +<H5>Last updated: February 15, 2006</H5> + +</BODY> +</HTML> |