ESO Programmatic Authentication & Authorisation

How to access private data and metadata

This jupyter notebook complements with some python examples what described in the ESO Programmatic Authentication & Authorisation documentation page.

It drives you through the process of:

  1. Authenticating to receive a token
  2. Performing authorised archive searches on raw data via TAP (using your token to exercise your permissions)
  3. Downloading science raw data with authorisation
  4. Finding the associated calibration reference files (via DataLink and calSelector)
  5. Downloading the calibration reference files and the association tree

This notebook is based on a little utility module called  eso_programmatic.py  and downloadable here, which contains, among others, the method to get a token (getToken).


Note: a live version of this notebook can be run using this MyBinder page (allow some time for the repository to start).


Initialisations

1 Authenticating

Get an ESO token using your ESO credential

With your ESO username and password you can get an authorization token (the id_token) using the getToken() method (see it here), part of the eso_programmatic.py module.

2 Authorised archive searches

Remember what written in documentation page, at §1.2.1 Which users should (not) perform authorised data searches? before performing authorised archive searches! Authorised queries are slower than anonymous queries and only few users will really need that functionality.

2.1 Setup a python requests session with an Authorization header

Create a python requests session and add your token to its header. You will pass this session to an ESO service when you want to ensure that your own permissions are taken into consideration.

2.2 Execute authorised queries

Any query you send to the tap service so initialised will be "authorised", in the sense that your permissions will be taken into consideration.

To achieve this, your query gets modified on-the-fly by the TAP software; the resulting SQL query ensures that you retrieve all the records you have granted access to, including the public ones, and only those. Such modified query (which you do not see) is more complex than the one you actually typed, and cannot be as fast.

For this reason we suggest to run authorised queries asynchronously, so to give it more execution time and not waiting for its results, hence avoiding http or application timeouts and possible intervening transient failures.

How? Using a TAP job.

3 Downloading the selected science files using their access_url

4 Finding and downloading the associated calibration reference files

The datalink service (implementing the VO DataLink protocol) helps you find out files related to an input science file (whether raw or product, in this case a raw). Let's call THIS the science file at hand. Datalink in particular can give you back two lists of calibration reference files that can be used to process THIS:

As a side note, Datalink can also offer access to other related files, e.g.:

The datalink_url field of the dbo.raw table provides you the link that can be used to find files associated to the selected science frame.

As shown above, the Datalink result is a table; each of its records provides a pointer (access_url) to an associated file, or to a service that returns associated files (like the calibration reference files); to distinguish among the records, the semantics column can be used.

In this case there are 4 records:

To know more:
For the two different flavours of calibration files (raw and processed), please refer to the documentation page of the calSelector service.
For the description of all possible semantics values, please refer to:

Here we want to get the processed calibration files, hence:

4.2 Getting the list of processed calibration reference files (using calSelector and DataLink)

The automatic selection of calibration files (raw or processed) is performed by the above-mentioned calSelector service, exposed also programmatically.

One of the calSelector interfaces (the responseformat=votable param must be present), is fully compatible with the datalink VO protocol. This means that the same pyvo DatalinkResults.from_result_url() method can be used, e.g., to get the list of associated raw2master files.

4.2.1 Check calibration cascade qualities

Check if calibration cascade is complete, if it is certified, and if it is actually for processed calib files

Beware: When executing a request for processed calibrations, you might get back the raw calibrations instead! This is happening when no processed calibrations exists for the given raw frame, in which case the service, not to leave you empty-handed, gives back the raw calibrations instead. It is possible to check this, by reading the calibration cascade description, as shown here below.

4.3 Downloading the calibration reference files

To download the calibration files we use again the downloadURL method of the eso_programmatic.py module.

All ESO calibration files are open to the public, hence there is no need to pass your token/session.

4.4 Getting the Association Tree describing the relations among the science frame and calibration files

You might have spotted above, that the associated_calib_files, generated invoking the raw2master_url, provides not only the #calibrator entries, but also an entry for the association tree.

Association Tree :== file describing the relations among the input raw frame(s) and the calibration files (in custom XML format)

You can use its semantics to find its access_url, as shown here below.