CCLRC / RUTHERFORD APPLETON LABORATORY
Particle Physics & Astronomy Research Council
Starlink Project
Clive Davenhall
Institute for Astronomy, University of Edinburgh & Starlink
27th October 1996
Notes on Writing an AstroCatalog
Server
(DRAFT)
Version 1.0
The ESO AstroCatalog library allows applications to access remote databases, catalogues and archives across the internet. It is used by the ESO skycat image display tool and various other applications. In order to make a database accessible to the AstroCatalog library a `server' must be provided for it. This server will accept remote queries sent in a standardised format, interrogate the database and return a table of results corresponding to the objects in the database which satisfied the query. These notes are intended to assist in writing such a server. They should be used in conjunction with listings for one or more existing servers.
The communication between the remote application and the server uses the Hyper-Text Transfer Protocol (HTTP) developed for the World Wide Web. The servers are, strictly speaking, gateways using the Common Gateway Interface (CGI). One way of thinking of the remote application is as a very specialised Web browser. One consequence of this approach is that if a site is to provide an AstroCatalog server it must also be running a Web server. The HTTP and CGI protocols are, of course, enormously flexible and there are additional protocols and conventions for communicating between an AstroCatalog application and server. These protocols and conventions are described in subsequent sections. The format for queries is described in Section 3 and that for the returned results in Section 4.
Though the protocols for communicating between an AstroCatalog application and server are (reasonably) well defined, there are an unlimited number of ways in which a server could be written to accept standard queries and return standard results. Thus, these notes cannot prescribe how a server must be written, but merely suggest some alternatives. An AstroCatalog server is a CGI script and CGI scripts are usually (but not necessarily) written in Perl. For the rest of these notes I shall assume that the server is to be implemented as a Perl script. The entire server could be written in Perl, with the database being accessed directly from the Perl script. However, it is more likely that the Perl script will invoke a Database Management System (DBMS) or special-purpose program to interrogate the database.
This document assumes that you have some familiarity with the AstroCatalog library or, at least, skycat. The AstroCatalog library is documented in Astronomical Catalog Library Interface Specification by Allan Brighton. It also assumes that you are familiar with HTTP, writing CGI scripts and Perl. There are numerous books describing the HTTP and CGI protocols. The HTML Source Book by Ian Graham seems to be a particularly good one. Similarly there are various books on Perl. Learning Perl by Randall Schwartz and Programming Perl by Larry Wall and Randall Schwartz are comprehensive and easy to use.
It might seem perverse to describe how to access a new server with the AstroCatalog library before describing how to write the server. However, access to the new server will often be the first thing that you set up, so that you can try it out during development and debugging.
The servers for catalogues, databases and archives are identified to the AstroCatalog library using a syntax of the form:
name@location
where name is an abbreviation for the name of the catalogue and location an abbreviation for the institution where it is held. By convention both name and location are usually quite short. For example:
ppm@eso
is the abbreviation for the version of Bastian and Roeser's accents? Positions and Proper Motion catalogue available at ESO.
A list of all the catalogues, databases and archives which the AstroCatalog library can access is held in a configuration file. You must take a copy of this configuration file, add details for your new server and then instruct your copy of the AstroCatalog library to access this modified file.
By default the AstroCatalog library uses a configuration file kept at ESO. You can retrieve a copy of it from URL:
http://archive.eso.org/skycat/skycat2.0.cfg
To access your own copy of this file you must place it in a directory accessible to your local HTTP server (that is, one which is `visible to the Web'). Then set environment variable SKYCAT_CONFIG to a URL which points to this file.
You must now add an entry for your catalogue to this file. The file is simply a text file which can be modified with a text editor. It contains extensive comments which should be read in conjunction with these notes.
For example, the entry for the PPM catalogue is:
serv_type: catalog long_name: PPM at ESO short_name: ppm@eso url: http://archive.eso.org/skycat/servers/ppm-server... symbol: mag circle 15-$mag
The purposes of the various items are as follows.
symbol is optional; the other items are mandatory.
Finally, if you want your server to become generally available then remember to send either the details or your modified configuration file to the skycat/AstroCatalog library team at ESO.
The parameters at the end of the server URL pass values which define the query which objects in the catalogue must satisfy if they are to be selected. The types of queries which are currently supported are mostly concerned with selecting objects within a specified region of sky. The region may be either:
The query parameters are as follows.
col1,col2,...coln
It is not always necessary (or possible) for a server to support all sorts of queries. For example, the minimum and maximum magnitude has no meaning in the case of the UK Schmidt Plate Catalogue.
The query parameters to select objects within 120 minutes of arc of 10:00:00, +30:00:00 in the PPM catalogue was:
http://archive.eso.org/skycat/servers/ppm-server
?ra=10:00:0.00&dec=+30:00:0.00&radius=120&nout=101
The query parameters used by the AstroCatalog library are special cases of a proposed general format for exchanging information between remote astronomical information services. A working document describing this proposal is available at URL:
http://vizier.u-strasbg.fr/doc/asu.html
Note that there seem to be differences between the query parameters currently implemented and the proposal.
The server must return the list of selected objects as a tab-separated list. The tab-separted list format is described in Section 2.3 of Astronomical Catalog Library Interface Specification. Briefly, it is a file or other stream of bytes, in which the values for individual fields in the list are separated by a tab character (ASCII character 9). A file in tab-separated list format can be modified with a text editor. The details of the format are as follows.
Content-type: text/plain
The optional special columns MORE and PREVIEW can be used within information and image servers (see Section ???)
A tab-separated list corresponding to a some columns in a subset of the PPM catalogue is listed below.
Content-type: text/plain Id ra dec mag -- -- --- --- +29 1956 148.028620833333 29.2643666666667 9.2 +31 2059 148.081295833333 30.3859694444444 9.5 +29 1958p 148.092170833333 28.5661469444444 9.8 +32 1947 148.121254166667 31.5426083333333 8.6 +29 1959 148.179058333333 29.0041111111111 9.6
~/.skycat/log
If you are using some application of your own rather than skycat then the AstroCatalog library is easily modified to echo the query when it sends it. Proceed as follows.
// send the query
result_buf = http_.get(buf, nlines);
cout << buf << endl;
between these two lines.
http://vizier.u-strasbg.fr/doc/asu.html
There are numerous books about the HTTP and CGI protocols and the Perl language. I have found the following useful, though they are not necessarily the best. They are included for completeness.