ESO Programmatic Authentication & Authorisation¶

How to access private data and metadata¶

This jupyter notebook complements with some python examples what described in the ESO Programmatic Authentication & Authorisation documentation page.

It drives you through the process of:

  1. Authenticating to receive a token
  2. Performing authorised archive searches on raw data via TAP (using your token to exercise your permissions)
  3. Downloading science raw data with authorisation
  4. Finding the associated calibration reference files (via DataLink and calSelector)
  5. Downloading the calibration reference files and the association tree

This notebook is based on a little utility module called  eso_programmatic.py  and downloadable here, which contains, among others, the method to get a token (getToken).

Note: a live version of this notebook can be run using this MyBinder page (allow some time for the repository to start).
Initialisations¶
In [1]:
TAP_URL = "http://archive.eso.org/tap_obs"

# Note: The TAP_CAT service (used to query public catalogues) does not need to support authentication

# Importing useful packages
import os 
import sys
import requests
#import cgi
import json
import time

import pyvo
from pyvo.dal import tap
from pyvo.auth.authsession import AuthSession
    
# Verify the version of pyvo 
from pkg_resources import parse_version
pyvo_version = parse_version(pyvo.__version__) 
test_pyvo_version = (pyvo_version == parse_version('1.1') or pyvo_version > parse_version('1.2.1') )
if not test_pyvo_version:
    print('You are using a not supported version of pyvo (version={version}).\nPlease use pyvo v1.1, v1.3, or higher, not v1.2* [ref. pyvo github issue #298]'.format(version=pyvo.__version__))
    raise ImportError('The pyvo version you are using is not supported, use 1.3+ or 1.1.')

print('\npyvo version {version} \n'.format(version=pyvo.__version__))

import eso_programmatic as eso

pyvo version 1.8 


/var/folders/bn/l4qv8ncs6cdcscgsb2h1p6hr0000n5/T/ipykernel_41979/1342949828.py:18: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import parse_version

1 Authenticating¶

Get an ESO token using your ESO credential¶

With your ESO username and password you can get an authorization token (the id_token) using the getToken() method (see it here), part of the eso_programmatic.py module.

In [2]:
# Prompt for user's credentials and get a token
import getpass

username = input("Type your ESO username: ")
password=getpass.getpass(prompt="%s's password: "%(username), stream=None)

token = eso.getToken(username, password)
if token != None:
    print('token: ' + token)
else:
    sys.exit(-1)

token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNzbyJ9.eyJqdGkiOiJUR1QtMzIwNzQ5LVRoS1BhMGs0WU05WXZIYWFMY0ZZRVRVcHBPMDdhZGxRNjZ5Qi1waGM0M09KVmJ3RENNY2t6akhoS1d1cEtka3p1eVktc3NvIiwic2lkIjoiM2Y4MWJhZDQxZWVhODFiM2EwMjIxZmJhMTA1Njg5ZmRiMGIyYzJiMCIsImlzcyI6Imh0dHBzOi8vd3d3LmVzby5vcmcvc3NvL29pZGMiLCJhdWQiOiJjbGllbnRpZCIsImV4cCI6MTc3NTA2NzM3MywiaWF0IjoxNzc1MDM4NTczLCJuYmYiOjE3NzUwMzgyNzMsInN1YiI6IkFNSUNPTCIsImNsaWVudF9pZCI6ImNsaWVudGlkIiwiYXV0aF90aW1lIjoxNzc1MDM4NTczLCJhdF9oYXNoIjoiMU0tRTlEQjdxbXNBT1BjUUYteFdYdyIsImFjY291bnRfaWQiOjUyMjQ5LCJlbWFpbCI6IkFsYmVydG8uTWljb2xAZXNvLm9yZyIsImZhbWlseV9uYW1lIjoiTWljb2wiLCJnaXZlbl9uYW1lIjoiQWxiZXJ0byIsImlzX2ltcGVyc29uYXRpbmciOmZhbHNlLCJuYW1lIjoiRHIuIE1pY29sIiwicm9sZXMiOlsidXNlciIsIkdOTFRfVVNFUiJdLCJzZXgiOiIiLCJ0aXRsZSI6IkRyLiIsInByZWZlcnJlZF91c2VybmFtZSI6IkFNSUNPTCJ9.eDZbb_bRAP4wXMXP4Pi6ij-zTtAn90JqCbF-W_bNBYysGL3O4a4bfB3EBZcggiRvyJ3XtGV_C-kgfW2YsDfIA-1tReL8M79dYCvnmP7CSWaBLEm8D372eU9TCIVI96YUQY6RjLy7XmGak2rGN00IoAZ_sinXWRrDwJL2WiP0XmjTRLqpsAbCz1Gj7-zP4T9qOTaRB7JK4acvNefSg333VZepL0fgtZIvD4QL3ZK7d7Znx_KyzGig9jqCjT3YFvcALPGImwOwPDbfxTsHnrcQUApf_wLydFcPi-Hua-zf0cyHFuNKKpeJqi16QALViSqeY0dIad1imYM_L9FmLwR-PA

2 Authorised archive searches¶

Remember what written in documentation page, at §1.2.1 Which users should (not) perform authorised data searches? before performing authorised archive searches! Authorised queries are slower than anonymous queries and only few users will really need that functionality.

  • authorised archive searches are useful only to users with special permissions
  • a PI of a regular observing programme normally does not possess special permissions
  • authorised queries are slower than anonymous queries, so use them only if you really need them!

2.1 Setup a python requests session with an Authorization header¶

Create a python requests session and add your token to its header. You will pass this session to an ESO service when you want to ensure that your own permissions are taken into consideration.

In [3]:
session = requests.Session()
session.headers['Authorization'] = "Bearer " + token

# Initialise a tap service for authorised queries
# passing the created "tokenised" session
# Remember: passing a non tokenised-session, or no session at all, 
# will result in tap performing anonymous queries:
# none of your permissions will be used, hence the queryies will run faster,
# and you will not be able to find any file with protected metadata.

tap = pyvo.dal.TAPService(TAP_URL, session=session)

# for comparison, use: 
# tap = pyvo.dal.TAPService(TAP_URL) 
# to execute your queries anonymously

2.2 Execute authorised queries¶

Any query you send to the tap service so initialised will be "authorised", in the sense that your permissions will be taken into consideration.

To achieve this, your query gets modified on-the-fly by the TAP software; the resulting SQL query ensures that you retrieve all the records you have granted access to, including the public ones, and only those. Such modified query (which you do not see) is more complex than the one you actually typed, and cannot be as fast.

For this reason we suggest to run authorised queries asynchronously, so to give it more execution time and not waiting for its results, hence avoiding http or application timeouts and possible intervening transient failures.

How? Using a TAP job.

In [4]:
# define the query you want to run, e.g.:
query = "select top 2 * from dbo.raw where dp_cat='SCIENCE' and prog_id = 'your-protected-observing-run' "

# well, in this example we use a non-protected run, 
# but please pretend it is actually a protected one given the purpose of this notebook!

# let's consider only 2 of its science frames:
query = "select top 2 * from dbo.raw where dp_cat='SCIENCE' and prog_id = '098.C-0739(C)' "


results = None

# define a job that will run the query asynchronously 
job = tap.submit_job(query)

# extending the maximum duration of the job to 300s (default 60 seconds)
job.execution_duration = 300 # max allowed: 3600s

# job initially is in phase PENDING; you need to run it and wait for completion: 
job.run()

try:
    job.wait(phases=["COMPLETED", "ERROR", "ABORTED"], timeout=600.)
except pyvo.DALServiceError:
    print('Exception on JOB {id}: {status}'.format(id=job.job_id, status=job.phase))

print("Job: %s %s" %(job.job_id, job.phase))

if job.phase == 'COMPLETED':
    # When the job has completed, the results can be fetched:
    results = job.fetch_result()

# the job can be deleted (always a good practice to release the disk space on the ESO servers)
job.delete()

# Let's print the results to examine the content:
# check out the access_url and the datalink_url
if results:
    print("query results:")
    eso.printTableTransposedByTheRecord(results.to_table()) 
else:
    print("!" * 42)
    print("!                                        !")
    print("!       No results could be found.       !")
    print("!       ? Perhaps no permissions ?       !")
    print("!       Aborting here.                   !")
    print("!                                        !")
    print("!" * 42)
    quit()

Job: d98a8ab8-6262-48ef-a302-332cbdfe2d9f COMPLETED
query results:
===================================================================================================================
    access_estsize = 58631
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308
    datalink_url   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    date_obs       = 2016-09-26T03:04:09.3085
    dec            = 1.60597
    dec_pnt        = 1.60597
    det_chip1id    = ESO-Hawaii2RG
    det_chop_ncycles = --
    det_dit        = --
    det_expid      = --
    det_ndit       = 8
    dp_cat         = SCIENCE
    dp_id          = SPHER.2016-09-26T03:04:09.308
    dp_tech        = IFU
    dp_type        = OBJECT,FLUX
    ecl_lat        = 3.955495
    ecl_lon        = 185.098217
    exp_start      = 2016-09-26T03:04:09.307Z
    exposure       = 8.0
    filter_path    = 
    gal_lat        = -55.878393
    gal_lon        = 87.249379
    grat_path      = 
    gris_path      = 
    ins_mode       = 
    instrument     = SPHERE
    lambda_max     = --
    lambda_min     = --
    last_mod_date  = 2016-09-26T03:15:57.943Z
    mjd_obs        = 57657.12788551
    ob_id          = 1427665
    ob_name        = sphere_irdifs_HIP_116384_-AML.obx
    object         = HIP_116384
    obs_mode       = s
    origfile       = SPHERE_IRDIFS_IFS_OBS270_0001.fits
    period         = 98
    pi_coi         = LAGRANGE/ KEPPLER/ BORGNIET/ DESIDERA/ MORDASINI/ AIGRAIN/ BEUST/ BONAVITA/ MEUNIER/ CHAUVIN/ MESSINA/ WORTERS/ LO CURTO/ DELORME/ LANNIER/ STERZIK/ GRATTON
    prog_id        = 098.C-0739(C)
    prog_title     = TOWARDS A FULL EXPLORATION OF THE JOVIAN PLANETS AROUND YOUNG STARS
    prog_type      = 0
    ra             = 353.75279888
    ra_pnt         = 353.752799
    release_date   = 2017-09-26T03:15:57.840Z
    s_region       = POSITION J2000 353.752799 1.60597
    slit_path      = 
    target         = HIP_116384
    tel_airm_end   = 1.1430000066757202
    tel_airm_start = 1.1449999809265137
    tel_alt        = 60.82099914550781
    tel_ambi_fwhm_end = 0.4399999976158142
    tel_ambi_fwhm_start = 0.4300000071525574
    tel_ambi_pres_end = 744.8499755859375
    tel_ambi_pres_start = 744.8800048828125
    tel_ambi_rhum  = 9.5
    tel_az         = 207.42999267578125
    telescope      = ESO-VLT-U3
    tpl_expno      = 1
    tpl_id         = SPHERE_irdifs_obs
    tpl_name       = IFS and IRDIS observations
    tpl_nexp       = 50
    tpl_seqno      = 2
    tpl_start      = 2016-09-26T03:03:40
    ---------------------------------------------------------------------------------------------------------
    access_estsize = 50908
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.496
    datalink_url   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.496
    date_obs       = 2016-09-26T03:04:09.4953
    dec            = 1.60597
    dec_pnt        = 1.60597
    det_chip1id    = ESO-Hawaii2RG
    det_chop_ncycles = --
    det_dit        = --
    det_expid      = --
    det_ndit       = 16
    dp_cat         = SCIENCE
    dp_id          = SPHER.2016-09-26T03:04:09.496
    dp_tech        = IMAGE,DUAL,CORONOGRAPHY
    dp_type        = OBJECT,FLUX
    ecl_lat        = 3.955495
    ecl_lon        = 185.098217
    exp_start      = 2016-09-26T03:04:09.497Z
    exposure       = 4.0
    filter_path    = B_H,D_H23
    gal_lat        = -55.878393
    gal_lon        = 87.249379
    grat_path      = 
    gris_path      = 
    ins_mode       = 
    instrument     = SPHERE
    lambda_max     = 1770.0
    lambda_min     = 1480.0
    last_mod_date  = 2016-09-26T03:15:57.950Z
    mjd_obs        = 57657.12788768
    ob_id          = 1427665
    ob_name        = sphere_irdifs_HIP_116384_-AML.obx
    object         = HIP_116384
    obs_mode       = s
    origfile       = SPHERE_IRDIFS_IRDIS_OBS270_0001.fits
    period         = 98
    pi_coi         = LAGRANGE/ KEPPLER/ BORGNIET/ DESIDERA/ MORDASINI/ AIGRAIN/ BEUST/ BONAVITA/ MEUNIER/ CHAUVIN/ MESSINA/ WORTERS/ LO CURTO/ DELORME/ LANNIER/ STERZIK/ GRATTON
    prog_id        = 098.C-0739(C)
    prog_title     = TOWARDS A FULL EXPLORATION OF THE JOVIAN PLANETS AROUND YOUNG STARS
    prog_type      = 0
    ra             = 353.75279888
    ra_pnt         = 353.752799
    release_date   = 2017-09-26T03:15:57.840Z
    s_region       = POSITION J2000 353.752799 1.60597
    slit_path      = 
    target         = HIP_116384
    tel_airm_end   = 1.1430000066757202
    tel_airm_start = 1.1449999809265137
    tel_alt        = 60.823001861572266
    tel_ambi_fwhm_end = 0.4399999976158142
    tel_ambi_fwhm_start = 0.4300000071525574
    tel_ambi_pres_end = 744.8499755859375
    tel_ambi_pres_start = 744.8800048828125
    tel_ambi_rhum  = 9.5
    tel_az         = 207.42300415039062
    telescope      = ESO-VLT-U3
    tpl_expno      = 2
    tpl_id         = SPHERE_irdifs_obs
    tpl_name       = IFS and IRDIS observations
    tpl_nexp       = 50
    tpl_seqno      = 2
    tpl_start      = 2016-09-26T03:03:40
    ---------------------------------------------------------------------------------------------------------

3 Downloading the selected science files using their access_url¶

In [5]:
# The access_url field of the dbo.raw table
# provides the link that can be used to download the file

# Here we pass that link together with your session
# to the downloadURL method of the eso_programmatic.py module
# (similarly to the authorised queries, if no session is passed, 
#  downloadURL will attempt to download the file anonymously)

print("Start downloading...")
for raw in results:
    access_url = raw['access_url'] # the access_url is the link to the raw file
    status, filepath = eso.downloadURL(access_url, session=session, dirname="/tmp")
    if status==200:
        print("      RAW: %s downloaded  "  % (filepath))
    else:
        print("ERROR RAW: %s NOT DOWNLOADED (http status:%d)"  % (filepath, status))

Start downloading...
      RAW: /tmp/SPHER.2016-09-26T03:04:09.308.fits.Z downloaded  
      RAW: /tmp/SPHER.2016-09-26T03:04:09.496.fits.Z downloaded

4 Finding and downloading the associated calibration reference files¶

The datalink service (implementing the VO DataLink protocol) helps you find out files related to an input science file (whether raw or product, in this case a raw). Let's call THIS the science file at hand. Datalink in particular can give you back two lists of calibration reference files that can be used to process THIS:

  • the list of raw calibration reference files (mode: raw2raw)
  • the list of processed calibration reference files (mode: raw2master)

As a side note, Datalink can also offer access to other related files, e.g.:

  • products generated out of THIS,
  • provenance files, i.e., the science files that were used to generate THIS
  • preview file, a quick look of THIS (for products only)
  • ancillary files of THIS (e.g. a weightmap of an imaging product) (for products only)
  • data documentation describing the science aim and the processing applied to THIS (for products only)
  • night log (for raw files only)
  • processed quicklook (for raw files only)

4.1 Find the link to the associated calibration reference files (using DataLink)¶

The datalink_url field of the dbo.raw table provides you the link that can be used to find files associated to the selected science frame.

In [6]:
# A python datalink object is created running
# the pyvo DataLinkResults.from_result_url() method onto the datalink_url.

# When dealing with files whose metadata are protected, we need to be authorised:
# for that we need to pass to the from_result_url() also the above-created python requests session.

# For the sake of this example, let's just consider the first science raw frame:
first_record = results[0]
datalink_url = first_record['datalink_url']

datalink = pyvo.dal.adhoc.DatalinkResults.from_result_url(datalink_url, session=session)

# The resulting datalink object contains the table of files associated
# to SPHER.2016-09-26T03:04:09.308
# Note: Were this input file a metadata protected file (it is not, but suppose...),
# and had you not passed your session, or had you no permission to see this file,
# DataLink would have given you back only a laconic table with the message 
# that that you do not have access permissions or that the file does not exist.

# let's print the resulting datalink table:
eso.printTableTransposedByTheRecord(datalink.to_table())

===================================================================================================================
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308
    service_def    = 
    error_message  = 
    semantics      = #this
    description    = Requested file
    content_type   = application/fits
    content_length = 58631135
    eso_origfile   = SPHER.2016-09-26T03:04:09.308.fits.Z
    eso_category   = 
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Raw&responseformat=votable
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2raw
    description    = List of access points of all the raw calibrations associated to the provided input raw file, and siblings (if any)
    content_type   = application/x-votable+xml
    content_length = 500000
    eso_origfile   = 
    eso_category   = 
    eso_datalink   = 
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Master&responseformat=votable
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master
    description    = List of access points of all the master calibrations associated to the provided input raw file, and siblings (if any)
    content_type   = application/x-votable+xml
    content_length = 500000
    eso_origfile   = 
    eso_category   = 
    eso_datalink   = 
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308.NL
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#night_log
    description    = The Night Log report contains comments on any issues that could have happened during the observation (e.g. instrument problems), as well as the information about the ambient conditions (airmass, seeing, transparency, etc.). The content_length provides only an order of magnitude of the file size.
    content_type   = text/plain
    content_length = 10000
    eso_origfile   = SPHER.2016-09-26T03:04:09.308.NL.txt
    eso_category   = Night Log
    eso_datalink   = 
    ---------------------------------------------------------------------------------------------------------

As shown above, the Datalink result is a table; each of its records provides a pointer (access_url) to an associated file, or to a service that returns associated files (like the calibration reference files); to distinguish among the records, the semantics column can be used.

In this case there are 4 records:

  • semantics = #this :

    • the first record in any datalink response always describes the input file (THIS)


  • semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2raw :

    • provides a link (access_url) to the the associated raw calibration files


  • semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master :

    • provides a link (access_url) to the associated processed calibration files


  • semantics = http://archive.eso.org/rdf/datalink/eso#night_log :

    • provide a link (access_url) to the associated Night Log report


To know more:
For the two different flavours of calibration files (raw and processed), please refer to the documentation page of the calSelector service.
For the description of all possible semantics values, please refer to:
  • the ESO semantics
  • the DataLink VO standard semantics

Here we want to get the processed calibration files, hence:

In [7]:
# Let's get the link to the processed calibration files (raw2master)

semantics = 'http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master'

raw2master_url = next(datalink.bysemantics( semantics )).access_url

# which returns the calSelector (see next box) link:
# https://archive.eso.org/calselector/v1/associations?dp_id=\
#SPHER.2016-09-26T03:04:09.308&mode=Raw2Master&responseformat=votable

4.2 Getting the list of processed calibration reference files (using calSelector and DataLink)¶

The automatic selection of calibration files (raw or processed) is performed by the above-mentioned calSelector service, exposed also programmatically.

One of the calSelector interfaces (the responseformat=votable param must be present), is fully compatible with the datalink VO protocol. This means that the same pyvo DatalinkResults.from_result_url() method can be used, e.g., to get the list of associated raw2master files.

In [8]:
# Don't forget to pass your session in case the science file has protected metadata!

associated_calib_files = pyvo.dal.adhoc.DatalinkResults.from_result_url(raw2master_url, session=session)

eso.printTableTransposedByTheRecord(associated_calib_files.to_table())

# create and use a mask to get only the #calibration entries,
# given that other entries, like #this or ...#sibiling_raw, could be present:
calibrator_mask = associated_calib_files['semantics'] == '#calibration'
calib_urls = associated_calib_files.to_table()[calibrator_mask]['access_url','eso_category']

#eso.printTableTransposedByTheRecord(calib_urls)

===================================================================================================================
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308
    service_def    = 
    error_message  = 
    semantics      = #this
    description    = category="IFS_SCI_FLUX" certified="true" complete="true" mode="Raw2Master" type="main" messages=""
    content_type   = application/fits
    content_length = 58631135
    eso_category   = IFS_SCIENCE_DR_RAW
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:05:51.768
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#sibling_raw
    description    = Any raw science file that needs to/can be calibrated together with the science raw file provided in input
    content_type   = application/fits
    content_length = 58748437
    eso_category   = IFS_SCIENCE_DR_RAW
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:05:51.768
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T04:06:44.796
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#sibling_raw
    description    = Any raw science file that needs to/can be calibrated together with the science raw file provided in input
    content_type   = application/fits
    content_length = 58751635
    eso_category   = IFS_SCIENCE_DR_RAW
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T04:06:44.796
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T04:08:27.017
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#sibling_raw
    description    = Any raw science file that needs to/can be calibrated together with the science raw file provided in input
    content_type   = application/fits
    content_length = 58562497
    eso_category   = IFS_SCIENCE_DR_RAW
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T04:08:27.017
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:14:14.686
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 54616320
    eso_category   = IFS_CAL_BACKGROUND
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:14:14.686
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:15:18.126
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 2721600
    eso_category   = IFS_IFU_FLAT_FIELD
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:15:18.126
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:17:07.116
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 54630720
    eso_category   = IFS_MASTER_DFF_LONG1
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:07.116
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:17:20.893
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 54630720
    eso_category   = IFS_MASTER_DFF_LONG2
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:20.893
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:17:33.533
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 54630720
    eso_category   = IFS_MASTER_DFF_LONG3
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:33.533
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:17:52.300
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 54630720
    eso_category   = IFS_MASTER_DFF_LONGBB
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:52.300
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:18:06.383
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 54624960
    eso_category   = IFS_PREAMP_FLAT
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:18:06.383
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://dataportal.eso.org/dataPortal/file/M.SPHERE.2016-09-27T08:18:40.706
    service_def    = 
    error_message  = 
    semantics      = #calibration
    description    = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
    content_type   = application/fits
    content_length = 176267520
    eso_category   = IFS_WAVECALIB
    eso_datalink   = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:18:40.706
    ---------------------------------------------------------------------------------------------------------
    ID             = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
    access_url     = https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Master
    service_def    = 
    error_message  = 
    semantics      = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master
    description    = Calibration cascade showing relations among all science and the master calibration files
    content_type   = application/xml
    content_length = 1000000
    eso_category   = ASSOCIATION_TREE
    eso_datalink   = 
    ---------------------------------------------------------------------------------------------------------

4.2.1 Check calibration cascade qualities¶

Check if calibration cascade is complete, if it is certified, and if it is actually for processed calib files

Beware: When executing a request for processed calibrations, you might get back the raw calibrations instead! This is happening when no processed calibrations exists for the given raw frame, in which case the service, not to leave you empty-handed, gives back the raw calibrations instead. It is possible to check this, by reading the calibration cascade description, as shown here below.
In [9]:
# Given the above list of "associated_calib_files"
# and knowing that we requested...
mode_requested = "raw2master"

# ... let's print out some important info and warnings on the received calibration cascade: 
# - is the cascade complete? 
# - is the cascade certified?
# - has the cascade being generated for the mode you requested (processed calibrations) or not?

# That info is embedded in the description field of the #this record.
# We use the printCalselectorInfo of the eso_programmatic.py to parse/make sense of it.

this_description=next(associated_calib_files.bysemantics('#this')).description

# *** New verbose parameter; without it no calibration info is shown
alert, mode_warning, certified_warning = eso.printCalselectorInfo(this_description, mode_requested, verbose=1)

if alert!="":
    print("%s" % (alert))
if mode_warning!="":
    print("%s" % (mode_warning))
if certified_warning!="":
    print("%s" % (certified_warning))
    
question = None
answer = None
if len(calib_urls):
    print()
    if alert or mode_warning or certified_warning:    
        question = "Given the above warning(s), do you still want to download these %d calib files [y/n]? " %(len(calib_urls))
    else:
        question = "No warnings reported, do you want to download these %d calib files [y/n]? " %(len(calib_urls))

while answer != 'y' and answer != 'n':
    answer = input(question)
    

    calibration info:
    ------------------------------------
    science category=IFS_SCI_FLUX
    cascade complete=true
    cascade messages=
    cascade certified=true
    cascade executed mode=raw2master

4.3 Downloading the calibration reference files¶

To download the calibration files we use again the downloadURL method of the eso_programmatic.py module.

All ESO calibration files are open to the public, hence there is no need to pass your token/session.

In [10]:
if answer == 'y':
    print("Downloading the %d calibration reference files..." % (len(calib_urls)) )

    i_calib=0
    for url,category in calib_urls:
        i_calib+=1
        status, filename = eso.downloadURL(url)
        if status==200:
            print("    CALIB: %4d/%d dp_id: %s (%s) downloaded"  % (i_calib, len(calib_urls), filename, category))
        else:
            print("    CALIB: %4d/%d dp_id: %s (%s) NOT DOWNLOADED (http status:%d)"  % (i_calib, len(calib_urls), filename, category, status))

Downloading the 8 calibration reference files...
    CALIB:    1/8 dp_id: ./M.SPHERE.2016-09-27T08:14:14.686.fits (IFS_CAL_BACKGROUND) downloaded
    CALIB:    2/8 dp_id: ./M.SPHERE.2016-09-27T08:15:18.126.fits (IFS_IFU_FLAT_FIELD) downloaded
    CALIB:    3/8 dp_id: ./M.SPHERE.2016-09-27T08:17:07.116.fits (IFS_MASTER_DFF_LONG1) downloaded
    CALIB:    4/8 dp_id: ./M.SPHERE.2016-09-27T08:17:20.893.fits (IFS_MASTER_DFF_LONG2) downloaded
    CALIB:    5/8 dp_id: ./M.SPHERE.2016-09-27T08:17:33.533.fits (IFS_MASTER_DFF_LONG3) downloaded
    CALIB:    6/8 dp_id: ./M.SPHERE.2016-09-27T08:17:52.300.fits (IFS_MASTER_DFF_LONGBB) downloaded
    CALIB:    7/8 dp_id: ./M.SPHERE.2016-09-27T08:18:06.383.fits (IFS_PREAMP_FLAT) downloaded
    CALIB:    8/8 dp_id: ./M.SPHERE.2016-09-27T08:18:40.706.fits (IFS_WAVECALIB) downloaded

4.4 Getting the Association Tree describing the relations among the science frame and calibration files¶

You might have spotted above, that the associated_calib_files, generated invoking the raw2master_url, provides not only the #calibrator entries, but also an entry for the association tree.

Association Tree :== file describing the relations among the input raw frame(s) and the calibration files (in custom XML format)

You can use its semantics to find its access_url, as shown here below.

In [11]:
association_tree_semantics = 'http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master'

# Notice that the datalink service and the calselector service use the same semantics
# to indicate two different things:
# - in datalink: it points to the distinct list of calibration reference files (responseformat=votable);
#                its eso_category is not defined
# - in calselector: it points to the calibration cascade description (format still XML but not votable);
#                its eso_category is set to "ASSOCIATION_TREE"

association_tree_mask = associated_calib_files['semantics'] == association_tree_semantics
association_tree = associated_calib_files.to_table()[association_tree_mask]['access_url','eso_category']

for url, category in association_tree:
    # the url points to the calselector service, which, for metadata protected files, needs a tokenised-session
    status, filename = eso.downloadURL(url, session=session)
    print(url)
    if status == 200:
        print("  Association tree: %s (%s) downloaded"  % (filename, category))
    else:
        print("  Association tree: %s (%s) NOT DOWNLOADED (http status:%d)"  % (filename, category, status))

https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Master
  Association tree: ./SPHER.2016-09-26T03:04:09.308_raw2master.xml (ASSOCIATION_TREE) downloaded