This jupyter notebook complements with some python examples what described in the ESO Programmatic Authentication & Authorisation documentation page.
It drives you through the process of:
This notebook is based on a little utility module called eso_programmatic.py and downloadable here, which contains, among others, the method to get a token (getToken).
TAP_URL = "http://archive.eso.org/tap_obs"
# Note: The TAP_CAT service (used to query public catalogues) does not need to support authentication
# Importing useful packages
import os
import sys
import requests
#import cgi
import json
import time
import pyvo
from pyvo.dal import tap
from pyvo.auth.authsession import AuthSession
# Verify the version of pyvo
from pkg_resources import parse_version
pyvo_version = parse_version(pyvo.__version__)
test_pyvo_version = (pyvo_version == parse_version('1.1') or pyvo_version > parse_version('1.2.1') )
if not test_pyvo_version:
print('You are using a not supported version of pyvo (version={version}).\nPlease use pyvo v1.1, v1.3, or higher, not v1.2* [ref. pyvo github issue #298]'.format(version=pyvo.__version__))
raise ImportError('The pyvo version you are using is not supported, use 1.3+ or 1.1.')
print('\npyvo version {version} \n'.format(version=pyvo.__version__))
import eso_programmatic as eso
pyvo version 1.1
With your ESO username and password you can get an authorization token (the id_token) using the getToken() method (see it here), part of the eso_programmatic.py module.
# Prompt for user's credentials and get a token
import getpass
username = input("Type your ESO username: ")
password=getpass.getpass(prompt="%s's password: "%(username), stream=None)
token = eso.getToken(username, password)
if token != None:
print('token: ' + token)
else:
sys.exit(-1)
Type your ESO username: AMICOL AMICOL's password: ········ token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InNzbyJ9.eyJqdGkiOiJUR1QtODA2MS02RTgwMG9FMXlXWS1EQkk1NHdyeXdNdE9OOFVneDJyR3NJcmZoZ3hOeURweDJaM2JKVkx0cGZnaFJiS1lZUmhQSFRVLXNzbyIsInNpZCI6ImUzYWEzODViMGZmNWZhZjQ4ZTRhZjE5MGNhY2Y5ZTBkZjAxNzUyZWEiLCJpc3MiOiJodHRwczovL3d3dy5lc28ub3JnL3Nzby9vaWRjIiwiYXVkIjoiY2xpZW50aWQiLCJleHAiOjE2NDU3NDQ2ODMsImlhdCI6MTY0NTcxNTg4MywibmJmIjoxNjQ1NzE1NTgzLCJzdWIiOiJBTUlDT0wiLCJjbGllbnRfaWQiOiJjbGllbnRpZCIsImF1dGhfdGltZSI6MTY0NTcxNTg4MywiYXRfaGFzaCI6Ii1SV09sLWdKbHVXYjViMGlSRUw0bVEiLCJhY2NvdW50X2lkIjo1MjI0OSwiZW1haWwiOiJBbGJlcnRvLk1pY29sQGVzby5vcmciLCJmYW1pbHlfbmFtZSI6Ik1pY29sIiwiZ2l2ZW5fbmFtZSI6IkFsYmVydG8iLCJpc19pbXBlcnNvbmF0aW5nIjpmYWxzZSwibmFtZSI6IkRyLiBNaWNvbCIsInJvbGVzIjpbInVzZXIiLCJHTkxUX1VTRVIiXSwic2V4IjoiIiwidGl0bGUiOiJEci4iLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJBTUlDT0wifQ.ECXknNqna5v3WoL13KbCCoMi0TxRL4IjMeVNNyN-SFl2V1AJ6cwXsANNCKLwxOBALNsli_yJD0OBRqpY-6MQviJEfddHJWIVGBVjyQIZhQc8LYnaImElXVY9G3TkzJ1prqgkYUYYQmWoqxqLcnaLanwrlfzyixrD2YtjMLsqR929oD60ytYzTPOR-YQ89KkNHTYZGml2JMCXGhX_XFTsigK_UhCKuyt5nmG1LpGFsIurCVqnvTc75scypmbpXn0v4vCrMspYJHzJZ2oA3R_HqSbvuF062fXTb0jiy9l6iZ26wql0mm6n0aCoss6cBc-WQDkFfWyGf3Ce89sZbxfZlw==
Remember what written in documentation page, at §1.2.1 Which users should (not) perform authorised data searches? before performing authorised archive searches! Authorised queries are slower than anonymous queries and only few users will really need that functionality.
Create a python requests session and add your token to its header. You will pass this session to an ESO service when you want to ensure that your own permissions are taken into consideration.
session = requests.Session()
session.headers['Authorization'] = "Bearer " + token
# Initialise a tap service for authorised queries
# passing the created "tokenised" session
# Remember: passing a non tokenised-session, or no session at all,
# will result in tap performing anonymous queries:
# none of your permissions will be used, hence the queryies will run faster,
# and you will not be able to find any file with protected metadata.
tap = pyvo.dal.TAPService(TAP_URL, session=session)
# for comparison, use:
# tap = pyvo.dal.TAPService(TAP_URL)
# to execute your queries anonymously
Any query you send to the tap service so initialised will be "authorised", in the sense that your permissions will be taken into consideration.
To achieve this, your query gets modified on-the-fly by the TAP software; the resulting SQL query ensures that you retrieve all the records you have granted access to, including the public ones, and only those. Such modified query (which you do not see) is more complex than the one you actually typed, and cannot be as fast.
For this reason we suggest to run authorised queries asynchronously, so to give it more execution time and not waiting for its results, hence avoiding http or application timeouts and possible intervening transient failures.
How? Using a TAP job.
# define the query you want to run, e.g.:
query = "select top 2 * from dbo.raw where dp_cat='SCIENCE' and prog_id = 'your-protected-observing-run' "
# well, in this example we use a non-protected run,
# but please pretend it is actually a protected one given the purpose of this notebook!
# let's consider only 2 of its science frames:
query = "select top 2 * from dbo.raw where dp_cat='SCIENCE' and prog_id = '098.C-0739(C)' "
results = None
# define a job that will run the query asynchronously
job = tap.submit_job(query)
# extending the maximum duration of the job to 300s (default 60 seconds)
job.execution_duration = 300 # max allowed: 3600s
# job initially is in phase PENDING; you need to run it and wait for completion:
job.run()
try:
job.wait(phases=["COMPLETED", "ERROR", "ABORTED"], timeout=600.)
except pyvo.DALServiceError:
print('Exception on JOB {id}: {status}'.format(id=job.job_id, status=job.phase))
print("Job: %s %s" %(job.job_id, job.phase))
if job.phase == 'COMPLETED':
# When the job has completed, the results can be fetched:
results = job.fetch_result()
# the job can be deleted (always a good practice to release the disk space on the ESO servers)
job.delete()
# Let's print the results to examine the content:
# check out the access_url and the datalink_url
if results:
print("query results:")
eso.printTableTransposedByTheRecord(results.to_table())
else:
print("!" * 42)
print("! !")
print("! No results could be found. !")
print("! ? Perhaps no permissions ? !")
print("! Aborting here. !")
print("! !")
print("!" * 42)
quit()
Job: 359b5826-b98c-4dd8-8f8b-6d98196c397c COMPLETED
query results:
===================================================================================================================
access_url = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308
datalink_url = http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
date_obs = 2016-09-26T03:04:09.3085
dec = 1.60597
dec_pnt = 1.60597
det_chip1id = ESO-Hawaii2RG
det_chop_ncycles = --
det_dit = --
det_expid = --
det_ndit = 8
dp_cat = SCIENCE
dp_id = SPHER.2016-09-26T03:04:09.308
dp_tech = IFU
dp_type = OBJECT,FLUX
ecl_lat = 3.955495
ecl_lon = 185.098217
exp_start = 2016-09-26T03:04:09.307Z
exposure = 8.0
filter_path =
gal_lat = -55.878393
gal_lon = 87.249379
grat_path =
gris_path =
ins_mode =
instrument = SPHERE
last_mod_date = 2016-09-26T03:15:57.943Z
mjd_obs = 57657.12890625
ob_id = 1427665
ob_name = sphere_irdifs_HIP_116384_-AML.obx
object = HIP_116384
obs_mode = s
origfile = SPHERE_IRDIFS_IFS_OBS270_0001.fits
period = 98
pi_coi = LAGRANGE/ KEPPLER/ BORGNIET/ DESIDERA/ MORDASINI/ AIGRAIN/ BEUST/ BONAVITA/ MEUNIER/ CHAUVIN/ MESSINA/ WORTERS/ LO CURTO/ DELORME/ LANNIER/ STERZIK/ GRATTON
prog_id = 098.C-0739(C)
prog_title = TOWARDS A FULL EXPLORATION OF THE JOVIAN PLANETS AROUND YOUNG STARS
prog_type = 0
ra = 353.75279888
ra_pnt = 353.752799
release_date = 2017-09-26T03:15:57.840Z
slit_path =
target = HIP_116384
tel_airm_end = 1.1430000066757202
tel_airm_start = 1.1449999809265137
tel_alt = 60.82099914550781
tel_ambi_fwhm_end = 0.4399999976158142
tel_ambi_fwhm_start = 0.4300000071525574
tel_ambi_pres_end = 744.8499755859375
tel_ambi_pres_start = 744.8800048828125
tel_ambi_rhum = 9.5
tel_az = 207.42999267578125
telescope = ESO-VLT-U3
tpl_expno = 1
tpl_id = SPHERE_irdifs_obs
tpl_name = IFS and IRDIS observations
tpl_nexp = 50
tpl_seqno = 2
tpl_start = 2016-09-26T03:03:40
---------------------------------------------------------------------------------------------------------
access_url = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:05:51.768
datalink_url = http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:05:51.768
date_obs = 2016-09-26T03:05:51.7679
dec = 1.60597
dec_pnt = 1.60597
det_chip1id = ESO-Hawaii2RG
det_chop_ncycles = --
det_dit = --
det_expid = --
det_ndit = 8
dp_cat = SCIENCE
dp_id = SPHER.2016-09-26T03:05:51.768
dp_tech = IFU
dp_type = OBJECT,FLUX
ecl_lat = 3.955495
ecl_lon = 185.098217
exp_start = 2016-09-26T03:05:51.767Z
exposure = 8.0
filter_path =
gal_lat = -55.878393
gal_lon = 87.249379
grat_path =
gris_path =
ins_mode =
instrument = SPHERE
last_mod_date = 2016-09-26T03:15:57.957Z
mjd_obs = 57657.12890625
ob_id = 1427665
ob_name = sphere_irdifs_HIP_116384_-AML.obx
object = HIP_116384
obs_mode = s
origfile = SPHERE_IRDIFS_IFS_OBS270_0002.fits
period = 98
pi_coi = LAGRANGE/ KEPPLER/ BORGNIET/ DESIDERA/ MORDASINI/ AIGRAIN/ BEUST/ BONAVITA/ MEUNIER/ CHAUVIN/ MESSINA/ WORTERS/ LO CURTO/ DELORME/ LANNIER/ STERZIK/ GRATTON
prog_id = 098.C-0739(C)
prog_title = TOWARDS A FULL EXPLORATION OF THE JOVIAN PLANETS AROUND YOUNG STARS
prog_type = 0
ra = 353.75279888
ra_pnt = 353.752799
release_date = 2017-09-26T03:15:57.840Z
slit_path =
target = HIP_116384
tel_airm_end = 1.1419999599456787
tel_airm_start = 1.1430000066757202
tel_alt = 60.999000549316406
tel_ambi_fwhm_end = 0.4099999964237213
tel_ambi_fwhm_start = 0.4399999976158142
tel_ambi_pres_end = 744.8300170898438
tel_ambi_pres_start = 744.8699951171875
tel_ambi_rhum = 9.5
tel_az = 206.625
telescope = ESO-VLT-U3
tpl_expno = 3
tpl_id = SPHERE_irdifs_obs
tpl_name = IFS and IRDIS observations
tpl_nexp = 50
tpl_seqno = 2
tpl_start = 2016-09-26T03:03:40
---------------------------------------------------------------------------------------------------------
# The access_url field of the dbo.raw table
# provides the link that can be used to download the file
# Here we pass that link together with your session
# to the downloadURL method of the eso_programmatic.py module
# (similarly to the authorised queries, if no session is passed,
# downloadURL will attempt to download the file anonymously)
print("Start downloading...")
for raw in results:
access_url = raw['access_url'] # the access_url is the link to the raw file
status, filepath = eso.downloadURL(access_url, session=session, dirname="/tmp")
if status==200:
print(" RAW: %s downloaded " % (filepath))
else:
print("ERROR RAW: %s NOT DOWNLOADED (http status:%d)" % (filepath, status))
Start downloading...
RAW: /tmp/SPHER.2016-09-26T03:04:09.308.fits.Z downloaded
RAW: /tmp/SPHER.2016-09-26T03:05:51.768.fits.Z downloaded
The datalink service (implementing the VO DataLink protocol) helps you find out files related to an input science file (whether raw or product, in this case a raw). Let's call THIS the science file at hand. Datalink in particular can give you back two lists of calibration reference files that can be used to process THIS:
As a side note, Datalink can also offer access to other related files, e.g.:
The datalink_url field of the dbo.raw table
provides you the link that can be used to find files associated
to the selected science frame.
# A python datalink object is created running
# the pyvo DataLinkResults.from_result_url() method onto the datalink_url.
# When dealing with files whose metadata are protected, we need to be authorised:
# for that we need to pass to the from_result_url() also the above-created python requests session.
# For the sake of this example, let's just consider the first science raw frame:
first_record = results[0]
datalink_url = first_record['datalink_url']
datalink = pyvo.dal.adhoc.DatalinkResults.from_result_url(datalink_url, session=session)
# The resulting datalink object contains the table of files associated
# to SPHER.2016-09-26T03:04:09.308
# Note: Were this input file a metadata protected file (it is not, but suppose...),
# and had you not passed your session, or had you no permission to see this file,
# DataLink would have given you back only a laconic table with the message
# that that you do not have access permissions or that the file does not exist.
# let's print the resulting datalink table:
eso.printTableTransposedByTheRecord(datalink.to_table())
===================================================================================================================
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308
service_def =
error_message =
semantics = #this
description = Requested file
content_type = application/fits
content_length = 58631135
eso_origfile = SPHER.2016-09-26T03:04:09.308.fits.Z
eso_category =
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Raw&responseformat=votable
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2raw
description = List of access points of all the raw calibrations associated to the provided input raw file, and siblings (if any)
content_type = application/x-votable+xml
content_length = 500000
eso_origfile =
eso_category =
eso_datalink =
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Master&responseformat=votable
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master
description = List of access points of all the master calibrations associated to the provided input raw file, and siblings (if any)
content_type = application/x-votable+xml
content_length = 500000
eso_origfile =
eso_category =
eso_datalink =
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://dataportal.eso.org/dataPortal/file/SPHER.2016-09-26T03:04:09.308.NL
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#night_log
description = The Night Log report contains comments on any issues that could have happened during the observation (e.g. instrument problems), as well as the information about the ambient conditions (airmass, seeing, transparency, etc.). The content_length provides only an order of magnitude of the file size.
content_type = text/plain
content_length = 10000
eso_origfile = SPHER.2016-09-26T03:04:09.308.NL.txt
eso_category = Night Log
eso_datalink =
---------------------------------------------------------------------------------------------------------
As shown above, the Datalink result is a table; each of its records provides a pointer (access_url) to an associated file, or to a service that returns associated files (like the calibration reference files); to distinguish among the records, the semantics column can be used.
In this case there are 4 records:
semantics = #this :
semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2raw :
semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master :
semantics = http://archive.eso.org/rdf/datalink/eso#night_log :
| To know more: For the two different flavours of calibration files (raw and processed), please refer to the documentation page of the calSelector service. |
| For the description of all possible semantics values, please refer to: |
Here we want to get the processed calibration files, hence:
# Let's get the link to the processed calibration files (raw2master)
semantics = 'http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master'
raw2master_url = next(datalink.bysemantics( semantics )).access_url
# which returns the calSelector (see next box) link:
# https://archive.eso.org/calselector/v1/associations?dp_id=\
#SPHER.2016-09-26T03:04:09.308&mode=Raw2Master&responseformat=votable
The automatic selection of calibration files (raw or processed) is performed by the above-mentioned calSelector service, exposed also programmatically.
One of the calSelector interfaces (the responseformat=votable param must be present), is fully compatible with the datalink VO protocol. This means that the same pyvo DatalinkResults.from_result_url() method can be used, e.g., to get the list of associated raw2master files.
# Don't forget to pass your session in case the science file has protected metadata!
associated_calib_files = pyvo.dal.adhoc.DatalinkResults.from_result_url(raw2master_url, session=session)
eso.printTableTransposedByTheRecord(associated_calib_files.to_table())
# create and use a mask to get only the #calibration entries,
# given that other entries, like #this or ...#sibiling_raw, could be present:
calibrator_mask = associated_calib_files['semantics'] == '#calibration'
calib_urls = associated_calib_files.to_table()[calibrator_mask]['access_url','eso_category']
#eso.printTableTransposedByTheRecord(calib_urls)
===================================================================================================================
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308&eso_download=file
service_def =
error_message =
semantics = #this
description = category="IFS_SCI_FLUX" certified="true" complete="true" mode="Raw2Master" type="main" messages=""
content_type = application/fits
content_length = 58631135
eso_category = IFS_SCIENCE_DR_RAW
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:05:51.768&eso_download=file
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#sibling_raw
description = Any raw science file that needs to/can be calibrated together with the science raw file provided in input
content_type = application/fits
content_length = 58748437
eso_category = IFS_SCIENCE_DR_RAW
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T03:05:51.768
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T04:06:44.796&eso_download=file
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#sibling_raw
description = Any raw science file that needs to/can be calibrated together with the science raw file provided in input
content_type = application/fits
content_length = 58751635
eso_category = IFS_SCIENCE_DR_RAW
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T04:06:44.796
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T04:08:27.017&eso_download=file
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#sibling_raw
description = Any raw science file that needs to/can be calibrated together with the science raw file provided in input
content_type = application/fits
content_length = 58562497
eso_category = IFS_SCIENCE_DR_RAW
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?SPHER.2016-09-26T04:08:27.017
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:14:14.686&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 54616320
eso_category = IFS_CAL_BACKGROUND
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:14:14.686
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:15:18.126&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 2721600
eso_category = IFS_IFU_FLAT_FIELD
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:15:18.126
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:07.116&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 54630720
eso_category = IFS_MASTER_DFF_LONG1
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:07.116
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:20.893&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 54630720
eso_category = IFS_MASTER_DFF_LONG2
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:20.893
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:33.533&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 54630720
eso_category = IFS_MASTER_DFF_LONG3
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:33.533
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:52.300&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 54630720
eso_category = IFS_MASTER_DFF_LONGBB
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:17:52.300
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:18:06.383&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 54624960
eso_category = IFS_PREAMP_FLAT
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:18:06.383
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:18:40.706&eso_download=file
service_def =
error_message =
semantics = #calibration
description = Calibration file associated to the input raw observation(s): #this (see above) and related siblings (if any)
content_type = application/fits
content_length = 176267520
eso_category = IFS_WAVECALIB
eso_datalink = https://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?M.SPHERE.2016-09-27T08:18:40.706
---------------------------------------------------------------------------------------------------------
ID = ivo://eso.org/ID?SPHER.2016-09-26T03:04:09.308
access_url = https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Master
service_def =
error_message =
semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master
description = Calibration cascade showing relations among all science and the master calibration files
content_type = application/xml
content_length = 1000000
eso_category = ASSOCIATION_TREE
eso_datalink =
---------------------------------------------------------------------------------------------------------
Check if calibration cascade is complete, if it is certified, and if it is actually for processed calib files
| Beware: When executing a request for processed calibrations, you might get back the raw calibrations instead! This is happening when no processed calibrations exists for the given raw frame, in which case the service, not to leave you empty-handed, gives back the raw calibrations instead. It is possible to check this, by reading the calibration cascade description, as shown here below. |
# Given the above list of "associated_calib_files"
# and knowing that we requested...
mode_requested = "raw2master"
# ... let's print out some important info and warnings on the received calibration cascade:
# - is the cascade complete?
# - is the cascade certified?
# - has the cascade being generated for the mode you requested (processed calibrations) or not?
# That info is embedded in the description field of the #this record.
# We use the printCalselectorInfo of the eso_programmatic.py to parse/make sense of it.
this_description=next(associated_calib_files.bysemantics('#this')).description
alert, mode_warning, certified_warning = eso.printCalselectorInfo(this_description, mode_requested)
if alert!="":
print("%s" % (alert))
if mode_warning!="":
print("%s" % (mode_warning))
if certified_warning!="":
print("%s" % (certified_warning))
question = None
answer = None
if len(calib_urls):
print()
if alert or mode_warning or certified_warning:
question = "Given the above warning(s), do you still want to download these %d calib files [y/n]? " %(len(calib_urls))
else:
question = "No warnings reported, do you want to download these %d calib files [y/n]? " %(len(calib_urls))
while answer != 'y' and answer != 'n':
answer = input(question)
calibration info:
------------------------------------
science category=IFS_SCI_FLUX
cascade complete=true
cascade messages=
cascade certified=true
cascade executed mode=raw2master
full description: category="IFS_SCI_FLUX" certified="true" complete="true" mode="Raw2Master" type="main" messages=""
No warnings reported, do you want to download these 8 calib files [y/n]? y
To download the calibration files we use again the downloadURL method of the eso_programmatic.py module.
All ESO calibration files are open to the public, hence there is no need to pass your token/session.
if answer == 'y':
print("Downloading the %d calibration reference files..." % (len(calib_urls)) )
i_calib=0
for url,category in calib_urls:
i_calib+=1
status, filename = eso.downloadURL(url)
if status==200:
print(" CALIB: %4d/%d dp_id: %s (%s) downloaded" % (i_calib, len(calib_urls), filename, category))
else:
print(" CALIB: %4d/%d dp_id: %s (%s) NOT DOWNLOADED (http status:%d)" % (i_calib, len(calib_urls), filename, category, status))
Downloading the 8 calibration reference files...
CALIB: 1/8 dp_id: ./M.SPHERE.2016-09-27T08:14:14.686.fits (IFS_CAL_BACKGROUND) downloaded
CALIB: 2/8 dp_id: ./M.SPHERE.2016-09-27T08:15:18.126.fits (IFS_IFU_FLAT_FIELD) downloaded
CALIB: 3/8 dp_id: ./M.SPHERE.2016-09-27T08:17:07.116.fits (IFS_MASTER_DFF_LONG1) downloaded
CALIB: 4/8 dp_id: ./M.SPHERE.2016-09-27T08:17:20.893.fits (IFS_MASTER_DFF_LONG2) downloaded
CALIB: 5/8 dp_id: ./M.SPHERE.2016-09-27T08:17:33.533.fits (IFS_MASTER_DFF_LONG3) downloaded
CALIB: 6/8 dp_id: ./M.SPHERE.2016-09-27T08:17:52.300.fits (IFS_MASTER_DFF_LONGBB) downloaded
CALIB: 7/8 dp_id: ./M.SPHERE.2016-09-27T08:18:06.383.fits (IFS_PREAMP_FLAT) downloaded
CALIB: 8/8 dp_id: ./M.SPHERE.2016-09-27T08:18:40.706.fits (IFS_WAVECALIB) downloaded
You might have spotted above, that the associated_calib_files, generated invoking the raw2master_url, provides not only the #calibrator entries, but also an entry for the association tree.
Association Tree :== file describing the relations among the input raw frame(s)
and the calibration files (in custom XML format)
You can use its semantics to find its access_url, as shown here below.
association_tree_semantics = 'http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master'
# Notice that the datalink service and the calselector service use the same semantics
# to indicate two different things:
# - in datalink: it points to the distinct list of calibration reference files (responseformat=votable);
# its eso_category is not defined
# - in calselector: it points to the calibration cascade description (format still XML but not votable);
# its eso_category is set to "ASSOCIATION_TREE"
association_tree_mask = associated_calib_files['semantics'] == association_tree_semantics
association_tree = associated_calib_files.to_table()[association_tree_mask]['access_url','eso_category']
for url, category in association_tree:
# the url points to the calselector service, which, for metadata protected files, needs a tokenised-session
status, filename = eso.downloadURL(url, session=session)
print(url)
if status == 200:
print(" Association tree: %s (%s) downloaded" % (filename, category))
else:
print(" Association tree: %s (%s) NOT DOWNLOADED (http status:%d)" % (filename, category, status))
https://archive.eso.org/calselector/v1/associations?dp_id=SPHER.2016-09-26T03:04:09.308&mode=Raw2Master Association tree: ./SPHER.2016-09-26T03:04:09.308_raw2master.xml (ASSOCIATION_TREE) downloaded