Sky Subtraction and the Propagation of Variance

There are two strategies for combining observations into groups depending upon the value of the logical flag add_in_pairs. If TRUE the reduction routines combine reduced observations belonging to a particular group into pairs of observations. Each pair must consist of one observation of type OBJECT and one of type SKY. Other combinations are not allowed.

If an observation is the first of the pair, it is simply remembered until next time around. If it is the second observation, the SKY member of the pair may be multiplied by a weighting factor indicated by the sky_wt parameter before being subtracted from the OBJECT member, and the result co-added to the group. Pairs of observations may also be weighted according to their variance.

The variance in the group may be propagated by combining together the variances contained in the individual reduced observations i.e. FROM_INT according to the familiar variance rule (Bevington 1969):

or, if these are poorly determined, by estimating the standard error, S, in the variations between the pairs of reduced observations when they are co-added into the group i.e. FROM_OBS according to the formula:

The latter will be necessary if the observations consist of only one exposure per integration and one detector scan, and therefore contain no error information themselves.

It is important to grasp that variance weighting of errors can only be used when the errors are already determined. It makes no sense, therefore, to enable variance weighting and attempt to propagate errors using the FROM_OBS regime.

Indeed, at the present time, although the data reduction provides both methods for error propagation i.e. viz. FROM_INT and FROM_OBS, the user is advised to default to FROM_OBS as, aside from the aforementioned caveat, a bug exists somewhere in the FROM_INT code that has never been traced. For completeness, however, both are documented here.

If add_in_pairs is FALSE, the routines apply a reduced observation belonging to a particular group to a reduced group file. Reduced observations of type OBJECT are added to the contents of the reduced group file whilst reduced observations of type SKY are subtracted from the contents of the reduced group file, after being optionally multiplied by a weighting factor indicated by the sky_wt parameter. Observations may also be weighted according to their variance if observing conditions dictate. Observation types other than OBJECT and SKY are not allowed.

You might well ask, then, what is the difference between add_in_pairs set to FALSE as opposed to TRUE since both appear to produce sky subtracted reduced groups?

First, if an equal number of OBJECT and SKY observations are taken, the running mean should be calculated using the number of pairs of observations, rather than the total number of observations, or the mean signal calculated will be a factor of 2 smaller than it should be. The signal will be calculated correctly for a series of OBJECT observations with no SKYs.

Second, if there are unequal numbers of bad pixels at a given location from the SKY and OBJECT observations, that particular point will not be sky-subtracted properly.

Third, the variance weighting algorithm will work properly only for a series of consecutive OBJECT observations. If both OBJECT and SKY observations are reduced, different weights may be applied to OBJECT and SKY, and the data will not be properly sky-subtracted.

For these reasons, it is better to add such observations in pairs i.e. with add_in_pairs set to TRUE.

This is only part of the story, however, as CGS4 can take data in several modes the two most common of which are CHOP and STARE. For CHOP mode sky subtraction, phase A -- B is calculated within the acquisition electronics and groups are formed as a sum of objects. If chopping and nodding are enabled then OBJECT---SKY pairs are formed as for STARE mode (essentially already described above but, of course, there should be an equal number of OBJECT and SKY frames taken during the acquisition phase).

Note that when observations are combined into reduced groups, a running average is maintained, with hooks built in for first and second observations so that a sensible mean and variance are generated, broadly according to the formula:

where is the data value for group, object and sky and n is the number of coadds. Some observers have noted that coadding many sky subtracted frames into reduced groups ( i.e. n > 50 ) slows the data reduction down by a significant factor and have attributed this to re-averaging the whole stack. That is incorrect. The probable cause of the slow down is the many FITS items and Portable--CGS4DR specific structures that must be included in the reduced group as part of the reduction. Extending such structures is, unfortunately, a costly process.