Programmatic Access

Overview

Users can gain programmatic and tool access to the archive science portal utilising the standard interfaces provided by the International Virtual Observatory Alliance protocols described here below.  

News

Since April 2020, the following new features are available:
Content:
  • support for ALMA data within the same ivoa.ObsCore table that already hosts Paranal La Silla and APEX data, for an integrated data discovery experience via the Tabular Access Protocol (tap_obs service)
Functionality:
  • authenticated access to proprietary (and other) data:
    • authentication via token (OAuth2.0, JSON Web Token)
    • basic authentication
  • positional and spectral cutout of science data products and their ancillary files
    • via the IVOA Server-side Operations for Data Access (SODA) standard
  • datalink: support for proprietary data
  • datalink: support for SODA (cutout)
Documentation:
  • A (growing) collection of HOWTOs, in form of jupiter notebooks, is available to illustrate the ESO programmatic interfaces.
Older but related news:
  • Since November 2019, the datalink service supports calSelector to associate calibrations to raw frames.

Read more...

 

The Table Access Protocol (TAP) is a web-service protocol that gives access to collections of tabular data referred to collectively as a tableset. TAP services accept queries posed against the tableset available via the service and return the query response as another table, in accord with the relational model. Queries to the ESO TAP services are submitted using the Astronomical Data Query Language ADQL (Ortiz and Lusted et al., 2008, IVOA Recommendation) and may execute synchronously or asynchronously. The result of a TAP query is another table, normally returned as a VOTable, though other output formats are available (FITS, json, votable/base64). The table collections made accessible via TAP are stored in relational database management systems. A TAP service exposes the database schema to client applications so that queries can be posed directly against arbitrary data tables available via the service.

  • In particular, ESO offers two TAP services:  tap_obs to query both the database tables describing the observed raw and reduced data obtained at the La Silla Paranal Observatory, and the database table containing the ESO ambient conditions and meteorological measurements (seeing, isoplanatic angle, precipitable water, turbulence profiles, etc.); tap_cat to query the scientific catalogues provided by the principal investigators of ESO observing programmes.
  •  

  • Powerful spatially-indexed queries can be formulated using the spatial extensions of ADQL, which is otherwise based on the SQL 92 standard. For example, it is possible to query the reduced data products by their footprints: does the footprint contain or intersect a specific sky region? which images or source tables overlap for a minimum extent of 0.8 square degrees? Note: while tap_obs offers the full power of spatially-indexed queries, tap_cat supports only the cone search formulated as in the following example:  SELECT ... FROM pessto_tran_cat_fits_V2 WHERE CONTAINS(point('', transient_raj2000, transient_decj2000), circle('', 41.2863, -55.7406, 0.04))=1 AND ...
  •  

  • Asynchronous queries allow for long running queries to complete without the client maintaining a connection to the service. Results are stored by the service for later retrieval by the client. Asynchronous query execution is generally more robust and not susceptible to time-outs or other transient failures. They are especially suited to queries that run for a long time before producing output (e.g. queries that compute or aggregate values). See (TAP 1.0, P. Dowler et al, 2010, IVOA Recommendation) for the full TAP specification, but also the (UWS REST Note, G. Rixon, 2007) for a primer on how to interact with the TAP asynchronous jobs. Please note that in this first release of the ESO Programmatic Access, the TAP UPLOAD feature is not implemented, as it will become available in a later release.
  •  

  • Synchronous queries execute immediately and the client must wait for the query to finish. Synchronous query execution is generally simpler and provides a faster (low latency) response and should be adequate when the query will execute and start returning results quickly. Even with large query results, synchronous queries are a good approach as long as the service can stream the output and consume modest internal resources.
  •  

  • Via tap_obs, users can query the standard ivoa.ObsCore table (ObsCore, M. Louys et al, 2017, IVOA Recommendation)) to gather access to all the ESO reduced data, the dbo.raw table to gather access to all the ESO raw data, and all the asm.* tables to gather access to the ambient conditions and meteorological measurements of the La Silla, Paranal and Chajnantor sites.

 

The Simple Spectral Access Protocol of the VO (SSAP, D. Tody et al., 2012, IVOA Recommendation) defines a uniform interface to remotely discover and access one dimensional spectra. The ESO Simple Spectral Access service provides access to the 1D reduced spectra generated either by the principal investigators of ESO observations, or by an unattended ESO processing-pipeline that makes use of certified master calibrations. The spectra are FITS files adhering to the ESO Science Data Product standard (PDF), based on the Virtual Observatory Spectral Data Model standard (v1.0 for some (simple) spectra, and v2.0 for some others (more complex): use the VOCLASS FITS keyword to discern between the two). A dedicated page provides help in reading and displaying the 1D spectral format, also using IRAF, IDL, and python.

Example: In its simplest form, a query to the ESO SSAP for 1d science spectra in a cone of 0.2 degrees around a given equatorial J2000 position, looks like this:
http://archive.eso.org/ssap?REQUEST=queryData&POS=123.269560,-34.57804&SIZE=0.2

 

The DataLink service (P. Dowler et al, 2015, IVOA Recommendation) is linked to from within the responses received from: the ivoa.ObsCore table for reduced data, and the dbo.raw table for raw data. For example, a query to the ivoa.ObsCore table can return a column called access_url; such URL can be recognised (by its UCD) as a datalink resource. Following a datalink URL, the user can gather the access points to either, the main science file described in the selected ObsCore record, any ancillary file associated to the main science file, the file(s) out of which the main science file originated (progenitors), the preview of the main science file, the data description associated to the main science file, and any product that could have been generated out of the present science file (derivation). Similarly, for the datalink_url column of the dbo.raw table. Basically, the DataLink is a service that finds all files that are somehow related to the selected science file, and presents a table with the access points of all those files. The output result of a DataLink invocation is a VOTable, unless the parameter RESPONSFORMAT=json is added to the request.

It is also possible to compute a datalink URL, instead of querying for it, by appending the ESO identifier dp_id field to the fixed string 'http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?', for example:

  • (reduced data): http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?ADP.2014-10-01T10:19:21.580
  • (raw data): http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?GIRAF.2003-09-26T23:43:54.741

Since 2019-Nov-15, the datalink response for a raw science frame includes links to the service that returns the calibration files needed to calibrate it. More information is available within the programmatic section of the page: Automatic selection of calibration files

Since April 2020, the DataLink service supports the SODA Service Descriptor (refer to the section 4.2 SODA Service Descriptor from DataLink), providing all the metadata necessary for the user to invoke the ESO SODA cutout service onto the specific dataset at hand.

Since April 2020, the DataLink service supports proprietary data, on top of the already supported public data. Special access to commisioning and project-specific files is not yet supported.

 

The Cutout service, compliant to the Server-side Operations for Data Access (SODA) standard (F. Bonnarel et al, 2017 IVOA Recommendation) is available since April 2020. It offers to the users the possibility to download just only the positional or spectral data chunks (including errors and quality arrays) they are interested in, without any loss in accuracy (no resampling, no rescaling) for any science or ancillary file with suitable positional and/or spectral dimensions. Currently supported are positional cutouts for images, cubes, source tables, and catalog tiles, and spectral cutouts for cubes and spectra, and their accompanying ancillary files like weightmaps, etc.

 

Authentication via either basic authentication or via OAuth2.0 OpenID_Connect is supported since April 2020. Users can now programmatically authenticate and download their proprietary data using those standard protocols, without the need to handshake with the archive in a complex Archive Request process. When requesting via HTTP a non-public dataset:

With basic authentication, the user attaches to the HTTP header the directive: "Authorization: Basic xxx" where xxx is the base64-encoding of the byte sequence composed of her "username:password";

With token-based authentications, the user, when authenticating through the OAuth2.0 endpoint (see below), receives a token that must be inserted in the HTTP header when requesting non-public data. The token remains active and valid for 8 hours (the validity range might change in the future).

Examples of the above are provided in a demo page (see here below).  

 

Service End Point (link to capabilities)
tap_obs http://archive.eso.org/tap_obs
tap_cat http://archive.eso.org/tap_cat
ssap http://archive.eso.org/ssap
datalink http://archive.eso.org/datalink
SODA/cutout https://dataportal.eso.org/dataportal_new/soda
OAuth2.0 OpenID_Connect https://www.eso.org/sso/oidc/token

 

A demo page is available, to learn how to use all the above (and more). Therein, live examples are provided; the user can execute them, or can modify them to learn how to use the system, can learn how to formulate valid URLs that can be used to script the archive access. Some simple python scripts are also provided, including, since April 2020, some jupyter notebooks, to illustrate how to script the access to the ESO science archive.

The TAP Query Management page can be used to control, or to learn how to control from a script, the TAP asynchronous jobs.