Overview: On The Fly Recalibration ST-ECF Archive

Automatic Recalibration of HST Archival Data A new paradigm for archives Dennis Crabree *, Daniel Durand *, Norman Hill *, Séverin Gaudet *, Benoît Pirenne ** *) Canadian Astronomy Data Centre, Dominion Astrophysical Observatory, Herzberg Institute of Astrophysics, National Research Council of Canada, Victoria, BC **) Space Telescope -- European Coordinating Facility, European Southern Observatory, Garching, Germany ...

HST science data is automatically calibrated when it is received at the STScI and this calibrated data is included in the archive. The calibration software, which is contained in the IRAF/STSDAS hst_calib package, takes as input the raw data and any necessary calibration reference images or tables if they are already available. The software determines which calibration steps to perform by checking the values of the calibration switches in the header. It selects which reference files to use in the calibration by examining the reference file keywords. The values of these switches and keywords depend upon the exact configuration of the instrument, the date of the observation and any other constraints. The values are set in the headers of the raw data in the RSDP (Routine Science Data Pipeline).

Until now, when users requested calibrated data from the HST archive, they received the data produced by the RSDP pipeline. However, with instrumental properties changing with time, better approaches for calibration of some instruments have been introduced and there have been other general improvements to the calibration of HST data. So what is a user to do? Fortunately, the same, or improved, software which runs in the calibration pipeline at STScI is also available in the released version of IRAF/STSDAS. One can recalibrate data from the archive by starting with the raw data, editing the appropriate header keywords to reflect the new calibration files and running the appropriate software. The Institute maintains a database which contains the recommended calibration reference files for each observation. However, this is not the most convenient approach for users, and this led us to develop an automatic recalibration process for HST data which essentially duplicates what a user would do manually.

1. The raw science data needs to be on-line
2. The needed calibration reference files and tables need to be on-line
3. A pipeline to recalibrate the data needs to be in place

In order for the recalibration to work properly, all of the necessary calibration files and tables need to be on-line. The CADC maintains a directory structure containing all of the necessary files by querying the 'bestref' tables each night to identify the distinct reference files which are needed, comparing this list to the ones currently on-line and then reporting which files need to be installed. Currently, there are approximately 9 GB of reference files on-line. Often a reference file is not available at the CADC/ST-ECF, i.e. it has not yet been distributed, and must be retrieved from STScI using Starview. A process at the ST-ECF copies the reference files regularly over the network to ensure that both sites have all of the needed reference files on-line.

While there are tasks within IRAF/STSDAS tasks for changing the header keywords associated with recalibration, they must be run manually. We developed a small IRAF/STSDAS pipeline to automate this process and to perform the complete 'recalibration pipeline'. One of the key steps in this process is a Perl script which queries the database to identify the recommended calibration reference files and tables. The output from this Perl script is a small IRAF script which updates the necessary IRAF parameters needed to update the raw data headers.

The process which occurs when a request is received is the following:

• The raw data is copied from CD-ROM to a staging area and decompressed
• An IRAF process is started which runs the recalibration pipeline. This pipeline does the following:
  • reads the data into IRAF/STSDAS (converts from FITS to internal format)
  • runs the Perl script to find out the recommended reference files
  • updates the raw headers to represent the new calibration information
  • runs the appropriate calibration pipeline and writes the results back as FITS files
• Copies the recalibrated data to the anonymous ftp area for retrieval

• inclusion of the pipelines for the new instruments (STIS and NICMOS);
• addition of extra calibration steps for the WFPC2 (like cosmic ray removal, co-addition of frames). This however, entails the need to create the currently unavailable notion of "products" or groups of observations;
• addition of more 'experimental' calibration steps with e.g., improved background treatment (FOS);
• the instrument physical models now being developed at the ST-ECF for FOS and STIS.
• interactive recalibration for expert users where processing steps can be turned off or on, and specific calibration files used.

The relevant URLs are:

http://cadcwww.dao.nrc.ca/hst

and

http://archive.eso.org/wdb/wdb/hst/science/form.