{
"cells": [
{
"cell_type": "markdown",
"id": "charming-breast",
"metadata": {},
"source": [
"# ESO Programmatic Authentication & Authorisation \n",
"## How to access private data and metadata \n",
"\n",
"\n",
"This jupyter notebook complements with some python examples what described in the ESO Programmatic Authentication & Authorisation documentation page.\n",
"\n",
"It drives you through the process of:\n",
"\n",
"1. Authenticating to receive a token\n",
"2. Performing authorised archive searches on raw data via TAP (using your token to exercise your permissions)\n",
"3. Downloading science raw data with authorisation\n",
"4. Finding the associated calibration reference files (via DataLink and calSelector)\n",
"5. Downloading the calibration reference files and the association tree\n",
"\n",
"\n",
"This notebook is based on a little utility module called eso_programmatic.py
and downloadable here, which contains, among others, the method to get a token (getToken). \n",
"\n",
"
datalink_url
field of the dbo.raw table\n",
"provides you the link that can be used to find files associated\n",
"to the selected science frame.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "described-ancient",
"metadata": {},
"outputs": [],
"source": [
"# A python datalink object is created running\n",
"# the pyvo DataLinkResults.from_result_url() method onto the datalink_url.\n",
"\n",
"# When dealing with files whose metadata are protected, we need to be authorised:\n",
"# for that we need to pass to the from_result_url() also the above-created python requests session.\n",
"\n",
"# For the sake of this example, let's just consider the first science raw frame:\n",
"first_record = results[0]\n",
"datalink_url = first_record['datalink_url']\n",
"\n",
"datalink = pyvo.dal.adhoc.DatalinkResults.from_result_url(datalink_url, session=session)\n",
"\n",
"# The resulting datalink object contains the table of files associated\n",
"# to SPHER.2016-09-26T03:04:09.308\n",
"# Note: Were this input file a metadata protected file (it is not, but suppose...),\n",
"# and had you not passed your session, or had you no permission to see this file,\n",
"# DataLink would have given you back only a laconic table with the message \n",
"# that that you do not have access permissions or that the file does not exist.\n",
"\n",
"# let's print the resulting datalink table:\n",
"eso.printTableTransposedByTheRecord(datalink.to_table())\n"
]
},
{
"cell_type": "markdown",
"id": "outdoor-spelling",
"metadata": {},
"source": [
"As shown above, the Datalink result is a table; each of its records provides a pointer (access_url) to an associated file, or to a service that returns associated files (like the calibration reference files); to distinguish among the records, the semantics
column can be used. \n",
"\n",
"In this case there are 4 records:\n",
" - semantics = #this
:
\n",
" \n",
" - semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2raw
:
\n",
" - provides a link (access_url) to the the associated raw calibration files
\n",
" \n",
" - semantics = http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master
:
\n",
" - provides a link (access_url) to the associated processed calibration files
\n",
" \n",
" - semantics = http://archive.eso.org/rdf/datalink/eso#night_log
:
\n",
" - provide a link (access_url) to the associated Night Log report
\n",
" \n",
"
To know more: \n", " For the two different flavours of calibration files (raw and processed), please refer to the documentation page of the calSelector service.\n", " |
\n",
" For the description of all possible semantics values, please refer to:\n",
"
|
Beware:\n", "When executing a request for processed calibrations, you might get back the raw calibrations instead! \n", "This is happening when no processed calibrations exists for the given raw frame, in which case the service, not to leave you empty-handed, gives back the raw calibrations instead.\n", "It is possible to check this, by reading the calibration cascade description, as shown here below.\n", " |
downloadURL
method of the eso_programmatic.py
module.\n",
"\n",
"All ESO calibration files are open to the public, hence there is no need to pass your token/session."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "strong-helen",
"metadata": {},
"outputs": [],
"source": [
"if answer == 'y':\n",
" print(\"Downloading the %d calibration reference files...\" % (len(calib_urls)) )\n",
"\n",
" i_calib=0\n",
" for url,category in calib_urls:\n",
" i_calib+=1\n",
" status, filename = eso.downloadURL(url)\n",
" if status==200:\n",
" print(\" CALIB: %4d/%d dp_id: %s (%s) downloaded\" % (i_calib, len(calib_urls), filename, category))\n",
" else:\n",
" print(\" CALIB: %4d/%d dp_id: %s (%s) NOT DOWNLOADED (http status:%d)\" % (i_calib, len(calib_urls), filename, category, status))\n"
]
},
{
"cell_type": "markdown",
"id": "living-sherman",
"metadata": {},
"source": [
"### 4.4 Getting the Association Tree describing the relations among the science frame and calibration files\n",
"\n",
"You might have spotted above, that the associated_calib_files
, generated invoking the raw2master_url, provides not only the #calibrator
entries, but also an entry for the association tree.\n",
"\n",
"Association Tree :== file describing the relations among the input raw frame(s)\n",
" and the calibration files (in custom XML format)
\n",
"\n",
"You can use its semantics to find its access_url, as shown here below."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "british-integer",
"metadata": {},
"outputs": [],
"source": [
"association_tree_semantics = 'http://archive.eso.org/rdf/datalink/eso#calSelector_raw2master'\n",
"\n",
"# Notice that the datalink service and the calselector service use the same semantics\n",
"# to indicate two different things:\n",
"# - in datalink: it points to the distinct list of calibration reference files (responseformat=votable);\n",
"# its eso_category is not defined\n",
"# - in calselector: it points to the calibration cascade description (format still XML but not votable);\n",
"# its eso_category is set to \"ASSOCIATION_TREE\"\n",
"\n",
"association_tree_mask = associated_calib_files['semantics'] == association_tree_semantics\n",
"association_tree = associated_calib_files.to_table()[association_tree_mask]['access_url','eso_category']\n",
"\n",
"for url, category in association_tree:\n",
" # the url points to the calselector service, which, for metadata protected files, needs a tokenised-session\n",
" status, filename = eso.downloadURL(url, session=session)\n",
" print(url)\n",
" if status == 200:\n",
" print(\" Association tree: %s (%s) downloaded\" % (filename, category))\n",
" else:\n",
" print(\" Association tree: %s (%s) NOT DOWNLOADED (http status:%d)\" % (filename, category, status))\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}