Programmatic Access

Overview

Users can gain programmatic and tool access to the archive science portal utilising the standard interfaces provided by the International Virtual Observatory Alliance protocols described here below.  

News

Since August 2021, the following new features are available:
  • Authorised TAP queries (via token authentication) allow to search through observations with private metadata: useful only to users who have been granted specific permissions, for example to instrument and operation teams, or to PIs of confidential observing programmes (or their delagates); not useful to PIs of regular programmes, nor to normal archive users (tap_obs service)
  • Authorised DataLink access (via token authentication) allows to find data files (e.g. calibration files) related to an observation whose metadata are private.
  • To know more, visit the new Authentication & Authorisation documentation page and from therein the related python jupyter notebook.
  • New columns added to the tables of reduced data (ivoa.ObsCore); for details check the change log page.
  • Ability to search for products whose metadata have changed (curation) using the new last_mod_date column (ivoa.ObsCore table of the tap_obs service, while dbo.raw already supported this)
Since April 2020, the following new features are available:
Content:
  • support for ALMA data within the same ivoa.ObsCore table that already hosts Paranal La Silla and APEX data, for an integrated data discovery experience via the Tabular Access Protocol (tap_obs service)
Functionality:
  • authenticated access to proprietary (and other) data:
    • authentication via token (OAuth2.0, JSON Web Token)
    • basic authentication
  • positional and spectral cutout of science data products and their ancillary files
    • via the IVOA Server-side Operations for Data Access (SODA) standard
  • datalink: support for proprietary data
  • datalink: support for SODA (cutout)
Documentation:
  • A (growing) collection of HOWTOs, in form of jupiter notebooks, is available to illustrate the ESO programmatic interfaces.
Older but related news:
  • Since November 2019, the datalink service supports calSelector to associate calibrations to raw frames.

Read more...

 

The Table Access Protocol (TAP) is a web-service protocol that gives access to tabular data. The tables made accessible via TAP are stored in relational database management systems. A TAP service opens and exposes the database to client applications so that queries can be posed directly against the avaialble tables. Queries to the ESO TAP services are formulated using the Astronomical Data Query Language ADQL (Ortiz and Lusted et al., 2008, IVOA Recommendation) and may execute synchronously or asynchronously. Query results are normally returned as a VOTable, though other output formats are available (FITS, json, txt, etc.).

  • In particular, ESO offers two TAP services:  tap_obs to query both the database tables describing the observed raw and reduced data obtained at the La Silla Paranal Observatory, and the database tables containing the ESO ambient conditions and meteorological measurements (seeing, isoplanatic angle, precipitable water, turbulence profiles, etc.); tap_cat to query the scientific catalogues provided by the principal investigators [PIs] of ESO public surveys and observing programmes.
  •  

  • Powerful spatially-indexed queries can be formulated using the spatial extensions of ADQL, which is otherwise based on the SQL 92 standard. For example, it is possible to query the reduced data products by their footprints: does the footprint contain or intersect a specific sky region? which images or source tables overlap for a minimum extent of 0.8 square degrees? Note: while tap_obs offers the full power of spatially-indexed queries, tap_cat supports only the cone search formulated as in the following example:  SELECT ... FROM pessto_tran_cat_fits_V2 WHERE CONTAINS(point('', transient_raj2000, transient_decj2000), circle('', 41.2863, -55.7406, 0.04))=1 AND ...
  •  

  • Asynchronous queries allow for long running queries to complete without the client maintaining a connection to the service. Results are stored by the service for later retrieval by the client. Asynchronous query execution is generally more robust and not susceptible to time-outs or other transient failures. They are especially suited to queries that run for a long time before producing output (e.g. queries that compute or aggregate values). See (TAP 1.0, P. Dowler et al, 2010, IVOA Recommendation) for the full TAP specification, but also the (UWS REST Note, G. Rixon, 2007) for a primer on how to interact with the TAP asynchronous jobs. Please note that the TAP UPLOAD feature is not implemented, though it will become available in a later release.
  •  

  • Synchronous queries execute immediately and the client must wait for the query to finish. Synchronous query execution is generally simpler and provides a faster (low latency) response and should be adequate when the query will execute and start returning results quickly. Even with large query results, synchronous queries are a good approach as long as the service can stream the output and consume modest internal resources.
  •  

  • Via tap_obs, users can query the standard ivoa.ObsCore table (ObsCore, M. Louys et al, 2017, IVOA Recommendation)) to gather access to all the ESO reduced data, the dbo.raw table to gather access to all the ESO raw data, and all the asm.* tables to gather access to the ambient conditions and meteorological measurements of the La Silla, Paranal and Chajnantor sites.
 
  • Authorised tap_obs queries (via token authentication) allow to search through all raw observations the user has been granted (metadata) access to, including the ones with private metadata not accessible anonymously.

Useful only to: users who have been granted specific permissions (on a run, or on an instrument), for example to instrument and operation teams, or to PIs of confidential observing programmes (or their delagates);

Not useful to: PIs of regular programmes, nor to normal archive users; these users can still run authenticated queries (provided they have an ESO account), but:

                - their queries will return exactly the same records as if unauthenticated,

                - their queries will run slower than if unathenticated.

 

The Simple Spectral Access Protocol of the VO (SSAP, D. Tody et al., 2012, IVOA Recommendation) defines a uniform interface to remotely discover and access one dimensional spectra. The ESO Simple Spectral Access service provides access to the 1D reduced spectra generated either by the principal investigators of ESO observations, or by an unattended ESO processing-pipeline that makes use of certified master calibrations. The spectra are FITS files adhering to the ESO Science Data Product standard (PDF), based on the Virtual Observatory Spectral Data Model standard (v1.0 for some (simple) spectra, and v2.0 for some others (more complex): use the VOCLASS FITS keyword to discern between the two). A dedicated page provides help in reading and displaying the 1D spectral format, also using IRAF, IDL, and python.

Example: In its simplest form, a query to the ESO SSAP for 1d science spectra in a cone of 0.2 degrees around a given equatorial J2000 position, looks like this:
http://archive.eso.org/ssap?REQUEST=queryData&POS=123.269560,-34.57804&SIZE=0.2

 

The DataLink service (P. Dowler et al, 2015, IVOA Recommendation) is a service that finds all files that are somehow related to the selected science file, and presents a table with the access points of all those files. The output result of a DataLink invocation is a VOTable, unless the parameter RESPONSFORMAT=json is added to the request. The ESO datalink service is linked to from within the responses received from queries to the ivoa.ObsCore table for reduced data, and the dbo.raw table for raw data. For example, a query to the ivoa.ObsCore table can return a column called access_url; such URL can be recognised (by its UCD) as a datalink resource. Following a datalink URL, the user can gather the access points (for download) of the main science file described in the selected ObsCore record, of any ancillary file associated to the main science file, of the file(s) out of which the main science file originated (progenitors), of the preview of the main science file, of the data release description associated to the main science file, and of any product that could have been generated out of the selected science file (derivation). Similarly, for the datalink_url column of the dbo.raw table.

It is also possible to compute a datalink URL, instead of querying for it, by appending the ESO identifier dp_id field to the fixed string 'http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?', for example:

  • (reduced data): http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?ADP.2014-10-01T10:19:21.580
  • (raw data): http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?GIRAF.2003-09-26T23:43:54.741

Since 2019-Nov-15, the datalink response for a raw science frame includes links to the service that returns the calibration files needed to calibrate it. More information is available within the programmatic section of the page: Automatic selection of calibration files

Since April 2020, the DataLink service supports the SODA Service Descriptor standard (refer to the section 4.2 SODA Service Descriptor from DataLink), providing all the metadata necessary for the user to invoke the ESO cutout service onto the specific dataset at hand (see below).

Since April 2020, the DataLink service supports proprietary data, on top of the already supported public data.

Since August 2021, the service supports token authentication, allowing operations teams and users with specific granted permissions, to run datalink also on their own private observations (not visible anonymously, e.g., commissioning and project-specific files).

 

The Cutout service, compliant to the Server-side Operations for Data Access (SODA) standard (F. Bonnarel et al, 2017 IVOA Recommendation) is available since April 2020. It offers to the users the possibility to download just only the positional or spectral data chunks (including errors and quality arrays) they are interested in, without any loss in accuracy (no resampling, no rescaling) for any science or ancillary file with suitable positional and/or spectral dimensions. Currently supported are positional cutouts for images, cubes, source tables, and catalog tiles, and spectral cutouts for cubes and spectra, and their accompanying ancillary files like weightmaps, etc.

 

Authentication to the ESO data portal, to download files or their cutouts, via either basic authentication or via OAuth2.0 OpenID_Connect is supported since April 2020. Users can programmatically authenticate and download their proprietary data (whether full download, or SODA/cutout) using those standard authentication protocols, without the need to handshake with the archive in a complex Archive Request process.

Authentication to the tap_obs and datalink via OAuth2.0 OpenID_Connect (basic authentication not supported): As of July 2021, it is possible to invoke tap_obs and datalink with token authentication, with the purpose of exercising specific user's permissions that allow to access data whose metadata are not accessible anonymously. For a full explanation and examples of this, please visit a dedicated ESO Programmatic Authentication & Authorisation and, from therein, a related python jupyter notebook.

With basic authentication, the user attaches to the HTTP header the directive: "Authorization: Basic xxx" where xxx is the base64-encoding of the byte sequence composed of her "username:password";

With token-based authentications, the user, when authenticating through the OAuth2.0 endpoint (see below), receives a token that must be inserted in the HTTP header. The token remains active and valid for 8 hours (the validity range might change in the future). In that case the HTTP header directive becomes: "Authorization: Bearer xxx" where xxx is the received token.

 

Service End Point (link to capabilities)
tap_obs https://archive.eso.org/tap_obs
tap_cat https://archive.eso.org/tap_cat
ssap https://archive.eso.org/ssap
datalink https://archive.eso.org/datalink
SODA/cutout https://dataportal.eso.org/dataPortal/soda
OAuth2.0 OpenID_Connect https://www.eso.org/sso/oidc/token

 

A demo page is available, with live examples, to learn how to use all the above (and more). Therein, live examples are provided; the user can execute them, or can modify them to learn how to use the system, can learn how to formulate valid URLs that can be used to script the archive access. Some simple python scripts are also provided, including, since April 2020, some jupyter notebooks, to illustrate how to script the access to the ESO science archive.

The TAP Query Management page can be used to control, or to learn how to control from a script, the TAP asynchronous jobs.