Programmatic Access

Overview

Users can gain programmatic and tool access to the archive science portal utilising the standard interfaces provided by the International Virtual Observatory Alliance protocols described here below.  

The Table Access Protocol (TAP) is a RESTful web-service protocol that gives access to collections of tabular data referred to collectively as a tableset. TAP services accept queries posed against the tableset available via the service and return the query response as another table, in accord with the relational model. Queries to the ESO TAP services are submitted using the Astronomical Data Query Language ADQL (Ortiz and Lusted et al., 2008, IVOA Recommendation) and may execute synchronously or asynchronously. The result of a TAP query is another table, normally returned as a VOTable, though other output formats are available (FITS, json, votable/base64). The table collections made accessible via TAP are stored in relational database management systems. A TAP service exposes the database schema to client applications so that queries can be posed directly against arbitrary data tables available via the service.

  • In particular, ESO offers two TAP services:  tap_obs to query both the database tables describing the observed raw and reduced data obtained at the La Silla Paranal Observatory, and the database table containing the ESO ambient conditions and meteorological measurements (seeing, isoplanatic angle, precipitable water, turbulence profiles, etc.); tap_cat to query the scientific catalogues provided by the principal investigators of ESO observing programmes.
  •  

  • Powerful spatially-indexed queries can be formulated using the spatial extensions of ADQL, which is otherwise based on the SQL 92 standard. For example, it is possible to query the reduced data products by their footprints: does the footprint contain or intersect a specific sky region? which images or source tables overlap for a minimum extent of 0.8 square degrees? Note: while tap_obs offers the full power of spatially-indexed queries, tap_cat supports only the cone search formulated as in the following example:  SELECT ... FROM pessto_tran_cat_fits_V2 WHERE CONTAINS(point('', transient_raj2000, transient_decj2000), circle('', 41.2863, -55.7406, 0.04))=1 AND ...
  •  

  • Asynchronous queries allow for long running queries to complete without the client maintaining a connection to the service. Results are stored by the service for later retrieval by the client. Asynchronous query execution is generally more robust and not susceptible to time-outs or other transient failures. They are especially suited to queries that run for a long time before producing output (e.g. queries that compute or aggregate values). See (TAP 1.0, P. Dowler et al, 2010, IVOA Recommendation) for the full TAP specification, but also the (UWS REST Note, G. Rixon, 2007) for a primer on how to interact with the TAP asynchronous jobs. Please note that in this first release of the ESO Programmatic Access, the TAP UPLOAD feature is not implemented, as it will become available in a later release.
  •  

  • Synchronous queries execute immediately and the client must wait for the query to finish. Synchronous query execution is generally simpler and provides a faster (low latency) response and should be adequate when the query will execute and start returning results quickly. Even with large query results, synchronous queries are a good approach as long as the service can stream the output and consume modest internal resources.
  •  

  • Via tap_obs, users can query the standard ivoa.ObsCore table (ObsCore, M. Louys et al, 2017, IVOA Recommendation)) to gather access to all the ESO reduced data, the dbo.raw table to gather access to all the ESO raw data, and all the asm.* tables to gather access to the ambient conditions and meteorological measurements of the La Silla, Paranal and Chajnantor sites.

 

The Simple Spectral Access Protocol of the VO (SSAP, D. Tody et al., 2012, IVOA Recommendation) defines a uniform interface to remotely discover and access one dimensional spectra. The ESO Simple Spectral Access service provides access to the 1D reduced spectra generated either by the principal investigators of ESO observations, or by an unattended ESO processing-pipeline that makes use of certified master calibrations. The spectra are FITS files adhering to the ESO Science Data Product standard (PDF), based on the Virtual Observatory Spectral Data Model standard (v1.0 for some (simple) spectra, and v2.0 for some others (more complex): use the VOCLASS FITS keyword to discern between the two). A dedicated page provides help in reading and displaying the 1D spectral format, also using IRAF, IDL, and python.

Example: In its simplest form, a query to the ESO SSAP for 1d science spectra in a cone of 0.2 degrees around a given equatorial J2000 position, looks like this:
http://archive.eso.org/ssap?REQUEST=queryData&POS=123.269560,-34.57804&SIZE=0.2

 

The DataLink service (P. Dowler et al, 2015, IVOA Recommendation) is linked to from within the responses received from the ObsCore service. For example, a query to the ivoa.ObsCore table can return a column called access_url; such URL can be recognised (by its UCD) as a datalink resource. Following a datalink URL, the user can gather the access points to either, the main science file described in the selected ObsCore record, any ancillary file associated to the main science file, the file(s) out of which the main science file originated, the preview of the main science file, the data description associated to the main science file, and any product that could have been generated out of the present science file. Basically, the DataLink is a service that finds all files that are somehow related to the selected science file, and presents a table with the access points of all those files. The output result of a DataLink invocation is a VOTable, unless the parameter RESPONSFORMAT=json is added to the request.

Not all services/tables return the datalink; in this cases (e.g. in the dbo.raw table case) the URL is easily computed by appending the content of the dp_id field to the fixed string 'http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?'.
Example: http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?ADP.2014-10-01T10:19:21.580

Since 2019-Nov-15, the datalink response for a raw science frame includes links to the service that returns the calibration files needed to calibrate it. More information is available within the programmatic section of the page: Automatic selection of calibration files

 

 

Service End Point
tap_obs http://archive.eso.org/tap_obs
tap_cat http://archive.eso.org/tap_cat
ssap http://archive.eso.org/ssap
datalink http://archive.eso.org/datalink

 

A demo page is available, to learn how to use all the above (and more). Therein, live examples are provided; the user can execute them, or can modify them to learn how to use the system, can learn how to formulate valid URLs that can be used to script the archive access. Some simple python scripts are also provided, to illustrate how to script the access to the ESO science archive.

The TAP Query Management page can be used to control, or to learn how to control from a script, the TAP asynchronous jobs.