{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"# How to download data \n",
"
\n",
"\n",
"This section of the [\"ESO Science Archive Programmatic: HOWTOs\"](http://archive.eso.org/programmatic/HOWTO/) shows how to programmatically download ESO data, either anonymously (for public data) or with authentication (for proprietary data), using Python.\n",
"\n",
"_**Usage**: You can access this file either as a static HTML page [(download it here)](http://archive.eso.org/programmatic/HOWTO/jupyter/ESO_How_to_download_data.html), or as an interactive jupyter notebook [(download it here)](http://archive.eso.org/programmatic/HOWTO/jupyter/ESO_How_to_download_data.ipynb) which you can download and run on your machine [(instructions)](https://jupyter.org/install). To interact with the jupyter notebook: move up and down the various cells using the arrow keys, execute the code by pressing CTRL+ENTER; you can also modify the code and execute it at will._\n",
"\n",
"
\n",
"\n",
"Let's start by setting up the python modules:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os \n",
"import sys\n",
"\n",
"import requests\n",
"import cgi\n",
"import json\n",
"\n",
"import getpass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's define a couple of utility functions, useful to write the files on disk using the ESO file name (provided in the response http header, via the Content-Disposition field."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def getDispositionFilename( response ):\n",
" \"\"\"Get the filename from the Content-Disposition in the response's http header\"\"\"\n",
" contentdisposition = response.headers.get('Content-Disposition')\n",
" if contentdisposition == None:\n",
" return None\n",
" value, params = cgi.parse_header(contentdisposition)\n",
" filename = params[\"filename\"]\n",
" return filename\n",
"\n",
"def writeFile( response ):\n",
" \"\"\"Write on disk the retrieved file\"\"\"\n",
" if response.status_code == 200:\n",
" # The ESO filename can be found in the response header\n",
" filename = getDispositionFilename( response )\n",
" # Let's write on disk the downloaded FITS spectrum using the ESO filename:\n",
" with open(filename, 'wb') as f:\n",
" f.write(response.content)\n",
" return filename "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to retrieve a file anonymously\n",
"\n",
"Without the need to authenticate, any user can anonymously download public files, that is, files that are out of the proprietary period (of usually one year from the moment the observation takes place)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"file_url = 'https://dataportal.eso.org/dataportal_new/file/ADP.2016-11-17T12:51:01.877'\n",
"\n",
"response = requests.get(file_url)\n",
"filename = writeFile( response )\n",
"if filename:\n",
" print(\"Saved file: %s\" % (filename))\n",
"else:\n",
" print(\"Could not get file (status: %d)\" % (response.status_code))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to retrieve a file with authentication\n",
"\n",
"If the files you need to retrieve are under proprietary period, you can access them only if you have the rights to do so, that is, if you are the principal investigator [PI] of the observing program the files belong to, or one of his/her delegates. In this case, you certainly got already a (free) user account at the [ESO User Portal](https://www.eso.org/UserPortal).\n",
"\n",
"Before downloading the file you have to authenticate and get a token. Here is the method that, given your ESO credentials (username and password), returns the token."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def getToken(username, password):\n",
" \"\"\"Token based authentication to ESO: provide username and password to receive back a JSON Web Token.\"\"\"\n",
" if username==None or password==None:\n",
" return None\n",
" token_url = \"https://www.eso.org/sso/oidc/token\"\n",
" token = None\n",
" try:\n",
" response = requests.get(token_url,\n",
" params={\"response_type\": \"id_token token\", \"grant_type\": \"password\",\n",
" \"client_id\": \"clientid\",\n",
" \"username\": username, \"password\": password})\n",
" token_response = json.loads(response.content)\n",
" token = token_response['id_token']+'=='\n",
" except NameError as e:\n",
" print(e)\n",
" except:\n",
" print(\"*** AUTHENTICATION ERROR: Invalid credentials provided for username %s\" %(username))\n",
" \n",
" return token\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you get the token, you need to add it to the HTTP header before HTTP-getting the file. Let's see how:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Suppose the file you want to download is accessible via the following link\n",
"# (please change the identifier (ADP.2020-03-24T1:45:21.866) to one of your proprietary files):\n",
"\n",
"file_url = 'https://dataportal.eso.org/dataportal_new/file/ADP.2020-03-24T10:45:21.866'\n",
"\n",
"# If you have not modified the identifier of that file,\n",
"# likely you won't be authorised to download the file.\n",
"# Expect a failure \"Could not get file (status: 401)\" in that case.\n",
"\n",
"# Let's get the token, by inputting your credentials:\n",
"username = input(\"Type your ESO username: \")\n",
"password=getpass.getpass(prompt=\"%s user's password: \"%(username), stream=None)\n",
"token = getToken(username, password)\n",
"\n",
"# With successful authentication you get a valid token,\n",
"# which needs to be added to the HTTP header of your GET request,\n",
"# as a Bearer:\n",
"\n",
"headers = None\n",
"if token!=None:\n",
" headers = {\"Authorization\": \"Bearer \" + token}\n",
" response = requests.get(file_url, headers=headers)\n",
" filename = writeFile( response )\n",
" if filename:\n",
" print(\"Saved file: %s\" % (filename))\n",
" else:\n",
" print(\"Could not get file (status: %d)\" % (response.status_code))\n",
"else:\n",
" print(\"Could not authenticate\")\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}