1 files changed, 1019 insertions, 0 deletions
diff --git a/funtools/doc/pod/funfiles.pod b/funtools/doc/pod/funfiles.pod
new file mode 100644
index 0000000..7efb464
--- /dev/null
+++ b/funtools/doc/pod/funfiles.pod
@@ -0,0 +1,1019 @@
+=pod
+
+=head1 NAME
+
+
+
+B<FunFiles: Funtools Data Files>
+
+
+
+=head1 SYNOPSIS
+
+
+
+
+This document describes the data file formats (FITS, array, raw
+events) as well as the file types (gzip, socket, etc.) supported
+by Funtools.
+
+
+
+=head1 DESCRIPTION
+
+
+
+
+
+Funtools supports FITS images and binary tables, and binary files
+containing array (homogeneous) data or event (heterogeneous) data.
+IRAF-style brackets are appended to the filename to specify various
+kinds of information needed to characterize these data:
+
+  file[ext|ind|ARRAY()|EVENTS(),section][filters]
+  or
+  file[ext|ind|ARRAY()|EVENTS(),section,filters]
+
+where:
+
+
+=over 4
+
+
+
+
+=item *
+
+B<file> is the Funtools file name
+
+
+=item *
+
+B<ext> is the FITS extension name
+
+
+=item *
+
+B<ind> is the FITS extension number
+
+
+=item *
+
+B<ARRAY()> is an array specification
+
+
+=item *
+
+B<EVENTS()> is an event specification
+
+
+=item *
+
+B<section> is the image section specification
+
+
+=item *
+
+B<filters> are spatial region and table (row) filters
+
+
+=back
+
+
+
+B<Supported Data Formats>
+
+Funtools programs (and the underlying libraries) support the
+following data file formats:
+
+
+=over 4
+
+
+
+
+=item *
+
+FITS images (and image extensions)
+
+
+=item *
+
+FITS binary tables
+
+
+=item *
+
+binary files containing an array of homogeneous data
+
+
+=item *
+
+binary files containing events, i.e. records of heterogeneous data
+
+
+=item *
+
+column-based text files, which are documented here
+
+
+=item *
+
+non-disk files and lists of files
+
+
+=back
+
+
+Information needed to identify and characterize
+the event or image data can be specified on the command line 
+using IRAF-style bracket notation appended to the filename:
+
+  foo.fits                              # open FITS default extension
+  image.fits[3]                         # open FITS extension #3
+  events.fits[EVENTS]                   # open EVENTS extension
+  array.file[ARRAY(s1024)]              # open 1024x1024 short array
+  events.file[EVENTS(x:1024,y:1024...)] # open non-FITS event list
+
+Note that in many Unix shells (e.g., csh and tcsh), filenames must
+be enclosed in quotes to protect the brackets from shell processing.
+
+B<FITS Images and Binary Tables>
+
+When FunOpen() opens a FITS file
+without a bracket specifier, the default behavior is to look for a
+valid image in the primary HDU. In the absence of a primary image,
+Funtools will try to open an extension named either B<EVENTS> or
+B<STDEVT>, if one of these exists. This default behavior supports
+both FITS image processing and standard X-ray event list processing
+(which, after all, is what we at SAO/HEAD do).
+
+
+In order to open a FITS binary table or image extension explicitly, it
+is necessary to specify either the extension name or the extension
+number in brackets:
+
+  foo.fits[1]                      # open extension #1: the primary HDU
+  foo.fits[3]                      # open extension #3 of a FITS file
+  foo.fits[GTI]                    # open GTI extension of a FITS file
+
+The ext argument specifies the name of the FITS extension (i.e. the
+value of the EXTENSION header parameter in a FITS extension), while
+the index specifies the value of the FITS EXTVER header parameter.
+Following FITS conventions, extension numbers start at 1.
+
+
+When a FITS data file is opened for reading using 
+FunOpen(), the specified extension
+is automatically located and is used to initialize the Funtools internal
+data structures.
+
+B<Non-FITS Raw Event Files>
+
+In addition to FITS tables, Funtools programs and libraries can operate
+on non-FITS files containing heterogeneous event records. To specify
+such an event file, use:
+
+
+
+=over 4
+
+
+
+
+=item *
+
+file[EVENTS(event-spec)]
+
+
+=item *
+
+file[EVENTS()]
+
+
+=back
+
+
+where B<event-spec> is a string that specified the names, data
+types, and optional image dimensions for each element of the event
+record:
+
+
+=over 4
+
+
+
+
+=item *
+
+[name]:[n][type]:[(lodim:)hidim]
+
+
+=back
+
+
+
+
+Data types follow standard conventions for FITS binary tables, but include
+two extra unsigned types ('U' and 'V'):
+
+
+=over 4
+
+
+
+
+=item *
+
+B<B> -- unsigned 8-bit char
+
+
+=item *
+
+B<I> -- signed 16-bit int
+
+
+=item *
+
+B<J> -- signed 32-bit int
+
+
+=item *
+
+B<K> -- signed 64-bit int
+
+
+=item *
+
+B<E> -- 32-bit float
+
+
+=item *
+
+B<D> -- 64-bit float
+
+
+=item *
+
+B<U> -- unsigned 16-bit int
+
+
+=item *
+
+B<V> -- unsigned 32-bit int
+
+
+=back
+
+
+An optional integer value B<n> can be prefixed to the type to indicate
+that the element is an array of n values. For example:
+
+  foo.fits[EVENTS(x:I,y:I,status:4J)]
+
+defines x and y as 16-bit ints and status as an array of 4 32-bit ints.
+
+
+Furthermore, image dimensions can be attached to the event specification
+in order to tell Funtools how to bin the events into an image. They
+follow the conventions for the FITS TLMIN/TLMAX keywords. If the low
+image dimension is not specified, it defaults to 1.  Thus:
+
+
+
+=over 4
+
+
+
+
+=item *
+
+RAWX:J:1:100
+
+
+=item *
+
+RAWX:J:100
+
+
+=back
+
+
+both specify that the dimension of this column runs from 1 to 100.
+
+
+NB: it is required that all padding be specified in the record
+definition. Thus, when writing out whole C structs instead of
+individual record elements, great care must be taken to include
+the compiler-added padding in the event definition.
+
+
+For example, suppose a FITS binary table has the following set of column
+definitions:
+
+  TTYPE1  = 'X                 ' / Label for field
+  TFORM1  = '1I                ' / Data type for field
+  TLMIN1  =                    1 / Min. axis value
+  TLMAX1  =                   10 / Max. axis value
+  TTYPE2  = 'Y                 ' / Label for field
+  TFORM2  = '1I                ' / Data type for field
+  TLMIN2  =                    2 / Min. axis value
+  TLMAX2  =                   11 / Max. axis value
+  TTYPE3  = 'PHA               ' / Label for field
+  TFORM3  = '1I                ' / Data type for field
+  TTYPE4  = 'PI                ' / Label for field
+  TFORM4  = '1J                ' / Data type for field
+  TTYPE5  = 'TIME              ' / Label for field
+  TFORM5  = '1D                ' / Data type for field 
+  TTYPE6  = 'DX                ' / Label for field
+  TFORM6  = '1E                ' / Data type for field
+  TLMIN6  =                    1 / Min. axis value
+  TLMAX6  =                   10 / Max. axis value
+  TTYPE7  = 'DY                ' / Label for field
+  TFORM7  = '1E                ' / Data type for field
+  TLMIN7  =                    3 / Min. axis value
+  TLMAX7  =                   12 / Max. axis value
+
+
+An raw event file containing these same data would have the event
+specification:
+
+  EVENTS(X:I:10,Y:I:2:11,PHA:I,PI:J,TIME:D,DX:E:10,DY:E:3:12)
+
+
+
+If no event specification string is included within the EVENTS() operator,
+then the event specification is taken from the B<EVENTS> environment
+variable:
+
+  setenv EVENTS "X:I:10,Y:I:10,PHA:I,PI:J,TIME:D,DX:E:10,DY:E:10"
+
+
+
+In addition to knowing the data structure, it is necessary to know the
+I<endian> ordering of the data, i.e., whether or not the data is
+in I<bigendian> format, so that we can convert to the native
+format for this platform. This issue does not arise for FITS Binary
+Tables because all FITS files use big-endian ordering, regardless of
+platform. But for non-FITS data, big-endian data produced on a Sun
+workstation but read on a Linux PC needs to be byte-swapped, since PCs
+use little-endian ordering. To specify an ordering, use the
+I<bigendian=> or I<endian=> keywords on the command-line
+or the EVENTS_BIGENDIAN or EVENTS_ENDIAN environment variables.  The
+value of the I<bigendian> variables should be "true" or "false",
+while the value of the I<endian> variables should be "little" or
+"big".
+
+
+For example, a PC can access data produced by a Sun using:
+
+  hrc.nepr[EVENTS(),bigendian=true]
+or
+  hrc.nepr[EVENTS(),endian=big]
+or
+  setenv EVENTS_BIGENDIAN true
+or
+  setenv EVENTS_ENDIAN big
+
+If none of these are specified, the data are assumed to follow the
+format for that platform and no byte-swapping is performed.
+
+B<Non-FITS Array Files>
+
+In addition to FITS images, Funtools programs and libraries can operate
+on non-FITS files containing arrays of homogeneous data. To specify
+an array file, use:
+
+
+=over 4
+
+
+
+
+=item *
+
+file[ARRAY(array-spec)]
+
+
+=item *
+
+file[ARRAY()]
+
+
+=back
+
+
+
+where array-spec is of the form:
+
+
+=over 4
+
+
+
+
+=item *
+
+[type][dim1][.dim2][:skip][endian]
+
+
+=back
+
+
+
+and where [type] is:
+
+
+=over 4
+
+
+
+
+=item *
+
+b   (8-bit unsigned char)
+
+
+=item *
+
+s   (16-bit short int)
+
+
+=item *
+
+u   (16-bit unsigned short int)
+
+
+=item *
+
+i   (32-bit int)
+
+
+=item *
+
+r,f (32-bit float)
+
+
+=item *
+
+d   (64-bit float)
+
+
+=back
+
+
+
+
+The dim1 specification is required, but dim2 is optional and defaults
+to dim1.  The skip specification is optional and defaults to 0.  The
+optional endian specification can be 'l' or 'b' and defaults to the
+endian type for the current machine.
+
+
+If no array specification is included within the ARRAY() operator,
+then the array specification is taken from the B<ARRAY> environment
+variable. For example:
+
+
+  foo.arr[ARRAY(r512)]          # bitpix=-32 dim1=512 dim2=512
+  foo.arr[ARRAY(r512.400)]      # bitpix=-32 dim1=512 dim2=400
+  foo.arr[ARRAY(r512.400])      # bitpix=-32 dim1=512 dim2=400
+  foo.arr[ARRAY(r512.400:2880)] # bitpix=-32 dim1=512 dim2=400 skip=2880
+  foo.arr[ARRAY(r512l)]         # bitpix=-32 dim1=512 dim2=512 endian=little
+  setenv ARRAY "r512.400:2880"
+  foo.arr[ARRAY()]              # bitpix=-32 dim1=512 dim2=400 skip=2880
+
+
+B<Specifying Image Sections>
+
+Once a data file (and possibly, a FITS extension) has been specified,
+the next (optional) part of a bracket specification can be used to
+select image B<section> information, i.e., to specify the x,y
+limits of an image section, as well as the blocking factor to apply to
+that section. This information can be added to any file specification but
+only is used by Funtools image processing routines.
+
+
+The format of the image section specification is one of the following:
+
+
+=over 4
+
+
+
+
+=item *
+
+file[xy0:xy1,block]
+
+
+=item *
+
+file[x0:x1,y0:y1,block]
+
+
+=item *
+
+file[x0:x1,*,block]
+
+
+=item *
+
+file[*,y0:y1,block]
+
+
+=item *
+
+file[*,block]
+
+
+=back
+
+
+where the limit values can be ints or "*" for default. A single "*"
+can be used instead of val:val, as shown.  Note that blocking is
+applied to the section after it is extracted.
+
+
+In addition to image sections specified by the lo and hi x,y limits, image
+sections using center positions can be specified:
+
+
+=over 4
+
+
+
+
+=item *
+
+file[dim1@xcen,dim2@ycen]
+
+
+=item *
+
+file[xdim2@xcen@ycen]
+
+
+=item *
+
+file[dim1@xcen,dim2@ycen,block]
+
+
+=item *
+
+file[dim@xcen@ycen,block]
+
+
+=back
+
+
+Note that the (float) values for dim, dim1, dim2, xcen, ycen must be
+specified or else the expression does not make sense!
+
+
+In all cases, block is optional and defaults to 1. An 's' or 'a' can
+be appended to signify "sum" or "average" blocking (default is "sum").
+Section specifications are given in image coordinates by default. If you
+wish to specify physical coordinates, add a 'p' as the last character
+of the section specification, before the closing bracket.
+For example:
+
+
+
+=over 4
+
+
+
+
+=item *
+
+file[-8:-7,-8:-7p]
+
+
+=item *
+
+file[-8:-7,-8:-7,2p]
+
+
+=back
+
+
+
+A section can be specified in any Funtools file name. If the operation
+to be applied to that file is an imaging operation, then the
+specification will be utilized. If the operation is purely a table
+operation, then the section specification is ignored.
+
+
+Do not be confused by:
+
+  foo.fits[2]
+  foo.fits[*,2]
+
+The former specifies opening the second extension of the FITS file.
+The latter specifies application of block 2 to the image section.
+
+
+Note that the section specification must come after
+any of FITS B<ext> name or B<ind> number,
+but all sensible defaults are supported:
+
+
+=over 4
+
+
+
+
+=item *
+
+file[ext]
+
+
+=item *
+
+file[ext,index]
+
+
+=item *
+
+file[index]
+
+
+=item *
+
+file[ext,section]
+
+
+=item *
+
+file[ext,index,section]
+
+
+=item *
+
+file[index,section]
+
+
+=item *
+
+file[section]
+
+
+=back
+
+
+
+B<Binning FITS Binary Tables and Non-FITS Event Files>
+
+If a FITS binary table or a non-FITS raw event file is to be binned
+into a 2D image (e.g., using the 
+funimage
+program), it is necessary to specify the two columns to be used for the
+binning, as well as the dimensions of the image.  Funtools first looks
+for a specifier of the form:
+
+ bincols=([xnam[:tlmin[:tlmax:[binsiz]]]],[ynam[:tlmin[:tlmax[:binsiz]]]])
+
+in bracket syntax, and uses the column names thus specified. The tlmin, tlmax,
+and binsiz specifiers determine the image binning dimensions using:
+
+  dim = (tlmax - tlmin)/binsiz     (floating point data)
+  dim = (tlmax - tlmin)/binsiz + 1 (integer data)
+
+These tlmin, tlmax, and binsiz specifiers can be omitted if TLMIN,
+TLMAX, and TDBIN header parameters are present in the FITS binary
+table header, respectively. If only one parameter is specified, it is
+assumed to be tlmax, and tlmin defaults to 1. If two parameters are
+specified, they are assumed to be tlmin and tlmax.
+
+For example, to bin an HRC event list columns "VPOS" and "UPOS", use:
+
+  hrc.nepr[bincols=(VPOS,UPOS)]
+
+or
+
+  hrc.nepr[bincols=(VPOS:49152,UPOS:4096)]
+
+Note that you can optionally specify the dimensions of these columns
+to cover cases where neither TLMAX keywords are defined in
+the header.  If either dimension is specified, then both must be specified.
+
+
+You can set the FITS_BINCOLS or EVENTS_BINCOLS environment variable as
+an alternative to adding the "bincols=" specifier to each file name
+for FITS binary tables and raw event files, respectively.  If no
+binning keywords or environment variables are specified, or if the
+specified columns are not in the binary table, the Chandra parameters
+CPREF (or PREFX) are searched for in the FITS binary table header.
+Failing this, columns named "X" and "Y" are sought.  If these are not
+found, the code looks for columns containing the characters "X" and
+"Y".  Thus, you can bin on "DETX" and "DETX" columns without
+specifying them, if these are the only column names containing the "X"
+and "Y" characters.
+
+
+Ordinarily, each event or row contributes one count to an image pixel
+during the 2D binning process. Thus, if five events all have the same
+(x,y) position, the image pixel value for that position will have a
+value of five. It is possible to specify a variable contribution
+for each event by using the vcol=[colname] filter spec:
+
+ vcol=[colname]
+
+The vcol colname is a column containing a numeric value in each event row
+that will be used as the contribution of the given event to its image
+pixel. For example, consider an event file that has the following content:
+
+  x:e:4    y:e:4    v:e
+  ------   ------   ----
+  1        1        1.0
+  2        2        2.0
+  3        3        3.0
+  4        4        0.0
+  1        1        1.0
+  2        2        2.0
+  3        3        3.0
+  4        4        4.0
+
+There are two events with x,y value of (1,1) so ordinarily a 2D image will
+have a value of 2 in the (1,1) pixel. If the v column is specified as the 
+value column:
+
+  foo.fits'[vcol=v]'
+
+then each pixel will contain the additive sum of the associated (x,y)
+column values from the v column.  For example, image pixel (1,1) will
+contain 1. + 1. = 2, image pixel (2,2) will contain (2 + 2) = 4, etc.
+
+
+An important variation on the use of a value column to specify the
+contribution an event makes to an image pixel is when the value column
+contains the reciprocal of the event contribution. For this case, the
+column name should be prefixed with a / (divide sign) thus:
+
+  foo.fits'[vcol=/v]'
+
+Each image pixel value will then be the sum of the reciprocals of the value
+column. A zero in the value column results in NaN (not a number).
+Thus, in the above example, image pixel (1.1) will contain 1/1 + 1/1 = 2,
+image pixel (2,2) will contain (1/2 + 1/2) = 1, etc. Image pixel (4,4)
+will contain (1/0 + 1/4) = NaN.
+
+
+You can set the FITS_VCOL or EVENTS_VCOL environment variable as
+an alternative to adding the "vcol=" specifier to each file name
+for FITS binary tables and raw event files, respectively.
+
+
+Finally, when binning events, the data type of the resulting 2D image
+must be specified. This can be done with the "bitpix=[n]" keyword in
+the bracket specification.  For example:
+
+  events.fits[bincols=(VPOS,UPOS),bitpix=-32]
+
+will create a floating point image binned on columns VPOS and UPOS.
+If no bitpix keyword is specified, bitpix=32 is assumed.  As with
+bincols values, you also can use the FITS_BITPIX and EVENTS_BITPIX
+environment variables to set this value for FITS binary tables and
+raw event files, respectively.
+
+
+The B<funimage> program also allows you to create a 1D image projection
+along any column of a table by using the B<bincols=[column]>
+filter specification and specifying a single column.
+For example, the following command projects a 1D image along
+the chipx column of a table:
+
+  funimage ev.fits'[bincols=chipx]' im.fits
+
+See funimage for more
+information about creating 1D and 2D images.
+
+
+Finally, please note that Funtools supports most FITS standards.
+We will add missing support as required by the community. In general,
+however, we do not support non-standard extensions. For example, we
+sense the presence of the binary table 'variable length array'
+proposed extension and we pass it along when copying and filtering
+files, but we do not process it. We will add support for new standards
+as they become official.
+
+B<Table and Spatial Region Filters>
+
+Note that, in addition extensions and image sections, Funtools bracket
+notation can be used to specify table and spatial region filters.  These
+filters are always placed after the image section information.  They
+can be specified in the same bracket or in a separate bracket
+immediately following:
+
+
+=over 4
+
+
+
+
+=item *
+
+file[ext|ind|ARRAY()|EVENTS(),section][filters]
+
+
+=item *
+
+file[ext|ind|ARRAY()|EVENTS(),section,filters]
+
+
+=back
+
+
+where:
+
+
+=over 4
+
+
+
+
+=item *
+
+B<file> is the Funtools file name
+
+
+=item *
+
+B<ARRAY()> is an array specification
+
+
+=item *
+
+B<EVENTS()> is an event list specification
+
+
+=item *
+
+B<ext> is the FITS extension name
+
+
+=item *
+
+B<ind> is the FITS extension number
+
+
+=item *
+
+B<section> is the image section to extract
+
+
+=item *
+
+B<filters> are spatial region and table (row) filters to apply
+
+
+=back
+
+
+
+The topics of table and region filtering are covered in detail in:
+
+
+=over 4
+
+
+
+
+=item *
+
+Table Filtering
+
+
+=item *
+
+Spatial Region Filtering
+
+
+=back
+
+
+
+B<Disk Files and Other Supported File Types>
+
+The specified B<file> usually is an ordinary disk file. In
+addition, gzip'ed files are supported in Funtools: gzip'ed input files
+are automatically uncompressed as they are read, and gzip'ed output
+files are compressed as they are written. NB: if a FITS binary table
+is written in gzip format, the number of rows in the table will be set
+to -1. Such a file will work with Funtools programs but will not work
+with other FITS programs such as ds9.
+
+
+The special keywords "stdin" and "stdout" designate Unix standard
+input and standard output, respectively. The string "-" (hyphen) will
+be taken to mean "stdin" if the file is opened for reading and
+"stdout" if the file is opened for writing.
+
+
+A file also can be an INET socket on the same or another machine using
+the syntax:
+
+  machine:port
+
+Thus, for example:
+
+  karapet:1428
+
+specifies that I/O should be performed to/from port 1428 on the
+machine karapet.  If no machine name is specified, the default is to
+use the current machine:
+
+  :1428
+
+This means to open port 1428 on the current machine. Socket support
+allows you to generate a distributed pipe:
+
+  on karapet:       funtask1 in.fits bynars:1428
+  on bynars:        funtask2 :1428 out.fits
+
+The socket mechanism thus supports simple parallel processing using
+B<process decomposition>. Note that parallel processing using
+B<data decomposition> is supported via the B<section> specifier (see
+below), and the B<row#> specifier, which is part of 
+Table Filtering.
+
+
+A file also can be a pointer to shared memory using the syntax:
+
+  shm:[id|@key][:size]
+
+A shared memory segment is specified with a B<shm:> prefix,
+followed by either the shared memory id or the shared memory key
+(where the latter is prefixed by the '@' character).  The size (in
+bytes) of the shared memory segment can then be appended (preceded by
+the ':' character). If the size specification is absent, the code will
+attempt to determine the length automatically. 
+
+If the open mode contains the string "w+", then the memory segment will be
+created if it does not exist. (It also will be released and deleted when the
+file is closed.) In the case where a memory segment is being created, the
+length of the segment is required.
+
+
+A file also can be Unix piped command (i.e. a program to run) using the syntax:
+
+  "pipe: command arg1 ... argn"
+
+The output from the command must be a valid FITS file. It is important
+to use quotes to protect spaces so that command arguments are passed
+correctly. A silly example is:
+
+  fundisp "pipe: funtable 'foo.fits[cir 512 512 .1]' stdout"
+
+This seemed like a good idea at the time ...
+
+B<Lists of Files>
+
+
+Funtools also will process a list of files as a single file using the
+syntax:
+
+  "list: file1 file2 ... filen"
+
+The files in the list are separated by whitespace. Any of the
+above file types can be used. For example, if two files, foo1.fits and
+foo2.fits, are part of the same observation, they can be processed as
+a single file (using their own filters):
+
+  fundisp "list: foo1.fits[cir(512,512,10)] foo2.fits[cir(511,511,10)]"
+         X        Y      PHA       PI                  TIME       DX       DY
+  -------- -------- -------- -------- --------------------- -------- --------
+       512      512        6        7     79493997.45854475      578      574
+       512      512        8        9     79494575.58943175      579      573
+       512      512        5        6     79493631.03866175      578      575
+       512      512        5        5     79493290.86521725      578      575
+       512      512        8        9     79493432.00990875      579      573
+       511      511        5        5     79488631.09462625      580      575
+       511      511       10       11     79488780.60006675      580      573
+       511      511        4        4     79494562.35474326      580      575
+       511      511        6        6     79488203.01561825      580      575
+       511      511        6        6     79488017.99730176      580      575
+       511      511        4        4     79494332.45355175      580      575
+       511      511        9       10     79492685.94014275      581      574
+       511      511        5        5     79487708.71298325      580      575
+       511      511        8        9     79493719.00160225      581      573
+
+Again, note that it is important to avoid spaces in the filters
+because the list separator also is whitespace. To protect whitespace
+in a filter, enclose the file specification in quotes:
+
+  fundisp "list: 'foo1.fits[cir 512 512 .1]' foo2.fits[cir(511,511,.1)]" 
+
+
+
+
+=head1 SEE ALSO
+
+
+
+See funtools(n) for a list of Funtools help pages
+
+
+
+=cut