Polynomial fits to CSO tau data

We have made a concentrated effort to better characterize the extinction corrections in the different SCUBA filters and update the previous work on the relationships between these and the CSO Tau value. A full description of this analysis and other documentation regarding calibration can be found here.

This has led to a change in the way we recommend reducing SCUBA data. Our preferred method of calculating sky opacities at the SCUBA wavelengths of 450 and 850µm is to fit a polynomial to the CSO data for the night concerned, and for any observation infer the 450 and 850 opacities from the polynomial fit using the relationships derived above. This has two advantages - i) CSO tau is sampled much more frequently than skydips, ii) fitting a smooth curve removes noise associated with individual data points.

Fits have been produced for the time since SCUBA has become operative - primarily in order to automatically produce a database of reduced SCUBA observations which are made available to the public. But scientists can also use these fits to reduce their own data. Here you can find postscript plots of the fits produced for each night, as well as files containing the numerical fit parameters. An example plot is given below. The green data points are CSO tau measurements, the red data points are 850 micron skydips converted to the CSO values using the correlations derived. The solid line is the fit used. The x axis is in units of fraction of the 24 period past midnight UT. The residual between the data and the fit is also shown.

"1.

For 850-µm data, both the skydips (with the updated values of T_HOT, T_COLD and ) and the revised CSO Tau relations are trustworthy. Given the infrequency of skydip measurements, we recommend that when the CSO Tau monitor is working, the polynomial fits be used in conjunction with the revised relations. However, if observers are concerned that the CSO Tau measurements are obtained at a fixed azimuth, the skydips will be more than sufficient for data reduction.

2.

For 450-µm data, individual skydips cannot be trusted. However, the revised CSO Tau relations were constructed with extreme care (Section 5), and are believed to be solid. To reduce 450-µm data, we suggest using either the polynomial fits to the CSO Tau or the 850-µm skydips, in conjunction with the new relations.

3.

For post-upgrade narrow band data, there are not enough data to construct CSO Tau relations. There is no a priori reason for these relations to differ from the pre-upgrade relations. The pre-upgrade relations should be a good approximation until more data are available to confirm this."

(from Calculating Sky Opacities: a re-analysis for SCUBA data by Elese N. Archibald, Jeff W. Wagg & Tim Jenness).

Oracdr can be told to use these fits in its reduction of your data. Simply use oracdr -calib tausys=csofit . For each observation the value of sky opacity is derived at the beginning and end using the polynomial fit, and the relationships between CSO tau and sky opacities. If your date is recent it is quite possible that the version of oracdr installed at your institute does not have the most up to date fits. Each month has a nejpolys.txt file with the fits which can be appended to the file ORACDR uses:

i) If your oracdr environment variables are not set type oracdr_scuba
ii) type setenv | grep ORAC_DATA_CAL this will tell you which directory oracdr keeps its calibration files in.
iii-a.) replace your current file csofit.dat in this directory with the file csofit.dat from these web pages (which contains all the fits in the archive) to bring your version of oracdr up to date.
iii-b.) or you can just append the nejpolys.txt file to your csofit.dat file.
You may not have permission to do steps iii-a or iii-b, and may need to get you your site manager to do it. Alternatively you can copy the directory to your own space and set the environment variable to point to it like this:

1.) Type oracdr_scuba

2.) Type setenv | grep ORAC_DATA_CAL This gives you the directory where oracdr looks for calibration related data. i.e.: /jcmt_sw/oracdr_cal/scuba/

3.) Make a directory where you can keep calibration data and have writing permission for. i.e
mkdir /myhome/mycalibration/

4.) Copy all the data from one to the other by typing (in this example)
cp -r /jcmt_sw/oracdr_cal/scuba/* /myhome/mycalibration/

5.) Append all the nejpolys.txt you want to the file /myhome/mycalibration/csofit.dat

6.) Type setenv ORAC_DATA_CAL /myhome/mycalibration/
If you run oracdr it now looks up calibration data in /myhome/mycalibration.

7.) Type oracdr -calib tausys=csofit

Here is a general reference for the use of different calibration mode in ORAC-DR.

If you use surf you are going to have to handle the polynomial fit yourself.This can either be done by a visual read-out of the graphic files or more precisely by evaluating the fits numerically. You can get the numerical parameters from the file nejpolys.txt which contains the fits for a full month, or from the file csofit.dat which contains all fits to date.. Each line in these files is a separate fit. The first column is the utdate. The second and third columns are the range of time over which the fit holds, given in terms of the fraction of the 24 hour period starting at midnight UT. The fourth column is the order of the fit (n). The next n+1 columns are the polynomial coefficients x0 to xn, where tau=x0+x1.t+x2.t^2....xn.t^n. These are all the data you should need, and it should be pretty straight forward to use the fit with minimal programming effort. A cgi-script to interface with these files is on the wish list - but may take some time. For the curious: the next two numbers are measures of the mean residual of the fit.The third last number is the value of sigma clipping used to throw-out bad cso-tau values, and the second and third last numbers are the tau_min, and tau_max clipping values used.

The following guidelines have been applied in the production of the polynomial fits to the CSO-Tau data since October 1 2001. The fits before that date generally obey these guidelines but might not always.

Skydips which are shown in the plots and used in the production of the fits have been screened for their quality. Only 850-µm skydips with the wideband filter have been used. The following cuts have been applied to DISCRETE SKYDIPS: 0.84 < < 0.86 ; 0.7 < Bandwithfactor < 0.89 ; Residual < 15 ; for RASTER SKYDIPS the cuts are : 0.84 < < 0.93 ; 0.7 < Bandwithfactor < 0.89 ; Residual < 220. These cuts have been choosen to assure that only trustworthy skydips are used and are a compromise between the statistics of the skydips, results of manual reductions and consistency of the archive. Approximately 80 percent of the Rasterdips and 95 percent of the Discrete Dips are used. (However, this does not reflect a lower trustworthiness of the Rasterdips.) The cuts applied for other filters can be requested if of interest.

The fitting of the CSO data is a deliberately subjective process, preferring a human judgment above an automated one, The guidelines above can therefore be disputed and the quality and applicability of any given fit can be challenged. This fact should be taken as a reminder that each calibration is a scientific task in itself and the responsibility finally rests with the observer. Any comments however, regarding the archived fits are very welcome.