New media for archive data distribution

Benoît Pirenne, ESO DMD & ST-ECF

The ESO/ST-ECF archive has, in its 13 years of existence, gone through a fairly large amount of data storage media and systems. For data distribution in particular and besides electronic (FTP) transfer, we have been offering our users the choice between a number of media: from the now defunct 9-track tape to the most recent DVD-R. In between, all sorts of formats have come and gone: Quarter-inch cartridge (QIC), DAT-DDS 1 & 2, Exabyte. Still available today are the DAT-DSS 3 and the DLT 4000 and DLT 7000, as well as CD-R.

The reasons for the changes are of course the adoption of the most economical technologies and the adaptation to data volumes. The former is important for our users who must be able to afford reading the data we provide. The latter is essential if we want to limit the total amount of volumes to be written, shipped to the users and later read by him or her. So our choices have always been, to a large extent, cost- and user-driven.

We have now come to a point where a technology change is again necessary: the tape formats we are using are outdated and still suffer from the same lack of user-friendliness as they did in the past. The DVD-R is still fine and economical but is less and less attractive when it comes to deliver data volumes such as those produced by the large mosaic cameras of ground-based telescope (e.g., the Wide Field Imager on ESO's 2.2m telescope) or data from HST's ACS with its many by-products. And the future promises an even worse situation on ESO's side: OmegaCAM will produce data by the "truck load" and VISTA and ALMA are already on the horizon. Based on those considerations, we now realize the need for something as convenient as a DVD, with a better price per GB and a low manipulation cost for the archive site, without requiring on the user's side large investment in media reading equipment. A study was carried out last year with those requirements as a driving force.

The study led to the constitution of a table (reproduced below) listing the current devices in use as well as the new promising technologies.

Table 1: Comparison of currently available digital media suitable for data transport. Besides the media type, the capacity, practical transfer rate, cost per volume and manpower cost per volume are provided. The re-use factor indicates whether a medium would be given away or requested back from the user. TCO stands for "Total cost of ownership". In this case, it indicates how much a typical data package of about 1 TB would cost on a per GB basis.

The conclusion is obviously that the emphasis should be put more and more on network data transfer, as it is by far the cheapest method available. However, pure electronic data transfer cannot address all possible case as very large data packages will not conveniently copy across, given the current bandwidth available. Moreover, the user or the archive center might have good reasons to want a physical medium to be delivered, even for smaller datasets (e.g., Principal Investigator data).

If we consider the other arguments that determine the suitability of a given medium for data transfer, it is quite clear that some of the tape systems available today such as the Super AIT from Sony or the "Super DLT" come quite cheap. However, next to the inconvenience linked to the sequential access and the loss of the file names due to the use of the FITS tape format, the tape drives required to read the media on the user side are exceedingly expensive (between 5 and 10 K€). This is a burden that would not make recipients of our tapes very happy. This is why we have investigated a new technology: the USB/FireWire external magnetic disk. The disk is still quite expensive but much less so if re-used a number of times. The burden and manual cost of following up disks' whereabouts is compensated by the ease of use for both the archive center and the user. As a matter of fact, those disks are mostly "plug-and-play" and are detected by the computer as soon as they are plugged in. For the largest possible compatibility across operating systems and hardware platforms, a ISO9660-compatible file system is written on the disks such that it will look like a very large CD to the operating system. The disks we are considering to use are either 80GB or 300GB units with USB and/or FireWire interfaces. They have been tested for compatibility with several platforms. Table 2 below summarizes what can be offered.

At the time of this writing, the necessary software and procedures to manipulate, write and follow up those magnetic disks have been prepared and tested. Further tests with a few "power users" are currently taking place. The new service will be offered starting April 1^st, 2004 only for request whose total volume would be in excess of several dozen gigabytes. The recipient of the disks are requested to return them to ESO within about 10 working days.

Parallel to the introduction of the new magnetic disks we will also stop offering tapes as archive data distribution medium. This means that neither DAT nor DLT 4000 and 7000 will be available in the media options of the request submission procedure.