CADC ARCHIVE SUCCESS STORY
Reprinted by permission of HIA Lights, the weekly
internal newsletter of the Herzberg Institute of Astrophysics.
The Canadian Astronomy Data Center (CADC) has successfully written
data to a 4.3 Gigabyte DVD-R (recordable Digital Versatile Disc). The
DVD-R that was written contains 261107 files of radio astronomy data
from the James Clark Maxwell Telescope. This contents of the DVD-R
constitutes the data generated by the James Clark Maxwell Telescope
from August 1987 to August 1995.
The data that were written onto this first DVD-R came from six
CD-Rs thus, the storage space required to store these data has been
reduced by a factor of six.
The file format used to store these date is the Universal Disk
Format (UDF) which is a relatively new file system which is expected
to be the industry standard for DVD-Rs. This file system differs from
the file system commonly used on CD-R (recordable CD) which is the ISO
9660 standard available from American National Standards Institute
(ANSI).
The Archive Storage (ASTO) software, which was developed by the
CADC, was used to create the DVD-R. This software, in-turn, uses
software from Tracer Technologies (www.tracertech.com) to do the
actually burning (writing) of the DVD-R with a Pioneer
(www.pioneerusa.com) DVD-R writer. Software from Tracer Technologies
is also being used to manage our Pioneer DVD-R jukebox where our first
DVD-R created currently resides. This DVD-R has already been joined
by two others, and more are expected to join it soon.
We at the CADC are very excited by this achievement. Moving our
data to DVD-R will greatly reduce the storage space needed to keep all
of the data we currently have in the archive. It also provides a
solution to the storing of large data files (approx. 210 Megabytes
each) that are being produced by the Canada France Hawaii
Telescope.
| The JCMT Archive is a collaboration between the
JCMT and CADC, and behind the scenes some hard work is being
undertaken to make this archive a world-class facility. CADC selected
the JCMT data to be the first to be converted to their new DVD-R
storage media as highlighted in the weekly publication of HIA,
reprinted on the left. The next step will be to try to push the
envelope of scientific usability of archival data by installing the
SCUBA reduction pipeline (ORAC-DR:
www.jach.hawaii.edu/stardocs/sun231.htx/sun231.html) at CADC. The
pipeline will initially be used to generate on-the-fly preview images
for single observations, optionally in FITS format. However,
ultimately we would like to be able to combine any compatible
observations, i.e. of the same object or the same region on the sky,
and produce a result with the best possible calibration for an
automatic process. I am a bit cautious here since we don't know at
present how good this calibration will be in general. A lot of work
has been done using the pipeline to look at the stability of SCUBA
and calibration for photometry by Jeff Wagg (UVic), Tim Jenness (JAC)
and Ian Robson (JAC). A subsequent investigation of SCUBA calibration
characteristics and optimal calibration procedures by Jeff, Elese
Archibald (JAC), and Iain Coulson (JAC) is nearing completion. The
results look promising. Their conclusions will be used to derive
representative SCUBA calibration parameters for each night which in
turn will reduce the overhead of a pipeline reduction sufficiently to
make it feasible to start combining data from one or more nights.
Whether or not the resulting images will be reliable enough for
scientific use without the need for the archive user to re-reduce the
data, remains to be seen. The interactive nature of ground-based
astronomy with limited centralized control over the observations may
prove to be too much of a challenge resulting in too many images with
systematic problems. Regardless, any investigator using archival data
in this fashion will be faced with a learning curve on how to guard
against errors. Standard quality checks will have to become an
accepted practice, i.e. comparing the results of logical subsets of the
observations. It will be interesting where these initiatives will lead
to in the end.
Since becoming available at CADC in 1997, the JCMT spectral line
Archive has enjoyed a continuously increasing popularity. To date
there have been about 400 requests for data from users in 11 different
countries who have downloaded a total of about 7,500 observations from
the Archive (after correcting for the 10,000 files downloaded by a
single user this year!). The figure below shows a breakdown per
year. On average each user has downloaded 20 files. The usage of the
JCMT Archive is especially remarkable given that the spectral line
data files have a non-FITS data format and require the use of the Specx software
package for their reduction. The expectation is that with the
release of the SCUBA observations the usage of the JCMT Archive at
CADC will increase markedly.
Acknowledgement: with thanks to the hardworking staff at CADC,
especially David Bohlender and Séverin Gaudet, for their efforts in
support of the JCMT Archive. The use of the CADC astronomical archives
is free: there is no charge to browse through the different
catalogues, to use the preview system, or to retrieve data. However,
prior to retrieving archive data, users must register.
|
