CGS4 data files under ORAC

Raw files in $ORAC_DATA_IN

The CGS4 raw data files are now stored as starlink HDS containers. Each file is equivalent to 1 observation, and as such contains a header component and 1 or more (actually NINT) integrations. Each integration is stored as an NDF component of the HDS file.

The filenames are thus: cYYYYMMDD_NNNNN.sdf, where YYYYMMDD is the numeric UT date (year, month and day) and NNNNN is the observation number, padded with leading zeros when necessary.

Reduced single frame files in $ORAC_DATA_OUT

First, some conventions:

The filename structure is: (PREFIX)(UTDATE)_(FRAME NUMBER)[_(EXTENSION)].sdf, where (THIS) is allways there and [THIS] is optional.

(PREFIX) is the letter 'c' if the file contains data from a single observation. It is 'gc' if the file contains data from a number of observations - ie a group.

_(EXTENSION) is used by individual primitives (think of a primitive as a single step within a recipe) for their output files. The pipeline its self keeps track of passing these files between primitives, though all the useful ones are left on the disk at the end so you can look at intermediate data products if you wish.

For example, c20000410_00123_ff.sdf would be data from a single observation, number 123, that has been flat fielded (the table later tells you the _ff means flat fielded).

So is that and HDS or an NDF?

Well, often, it depends; the pipeline passes files between primitives as either HDS or NDF, depending on which is most appropriate. For example, with 2x2 sampled data, the raw file will be an HDS container containg the 4 NDF data arrays from the 4 sample positions. If we're using a 1x1 sampled flat field, the flat field primitive will flat field each component, and write out an HDS container containg the four flat fielded data arrays as NDFs. On the otherhand, if the flat field is taken with the same sampling as the data, the pipeline will interleave the samples of both the data and the flat field before carrying out the flatfielding, thus the _ff file would be a single NDF.

Of course, some primitives will only write out single-NDF files; for example, the primitive that interleaves and/or coadds the samples into one larger image writes out _inc files, and these are allways single component NDFs.

An additional factor is that later on in the processing, the pipeline may go back to using HDS containers. This happens for example when we start extracting beams and spectra from the reduced group image - HDS containers are used, containing information for each beam - for example opt-extract profiles, and the spectra themselves, whilst they pass through the primitives that operate on the individual extracted spectra before these are combined to give the final spectrum - for example de-rippling.

This figure illustrates the data flow between primitives for the reduction of a typical single frame.

Reduced group files in $ORAC_DATA_OUT

The pipeline adds the individual frames into the group as they are processed. The group number is usually the frame number of the first frame in the group. Group files are allways single NDFs.