This section of the \"ESO Science Archive Programmatic: HOWTOs\" shows how to programmatically use the phase3v2.product_files table via TAP (tap_obs) to programmatically search and eventually download ancillary files. After illustrating the columns of that table, a few query examples are provided.
Usage: You can access this file either as a static HTML page (download it here), or as an interactive jupyter notebook (download it here) which you can download and run on your machine (instructions). To interact with the jupyter notebook (if you have download it): move up and down the various cells using the arrow keys, execute the code by pressing CTRL+ENTER; you can also modify the code and execute it at will.
\n", "\n", "\n", "### Introduction\n", "\n", "Users can browse and download so-called science-ready data products (products from now on) from the ESO Science Archive.\n", "\n", "Products have instrument and atmospheric signatures removed, possibly are calibrated in physical units, and have noise properties (like limiting magnitude or signal-to-noise ratio) quantified and documented.\n", "\n", "A product is composed of a science file (mandatorily), and optionally of a number (0-N) of ancillary files, typically created at the same time as the product itself.\n", "\n", "Example of ancillary files are: weight-maps accompanying images, white-light images (2D image obtained by averaging a data cube along the wavelength axis), tar balls, previews, telluric spectra, masks, etc. For the full list, please see the ESO Science Data Product standard.\n", "\n", "It is useful at times to be able to search through the existing ancillary files. Examples of this are provided in what follows. \n", "\n", "\n", "Table of contents:\n", " * Initialisation step (using pyvo)\n", " * The phase3v2.product_files columns\n", " * Example 1: How to find all the files belonging to a given product, and their metadata\n", " * Example 2: The ESPRESSO ancillary files\n", " * Example 3: Getting all the ANCILLARY.PREVIEW of the MUSE collection\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pyvo version 1.4\n" ] } ], "source": [ "import os \n", "import sys\n", "\n", "import pyvo as vo\n", "from pyvo.dal import tap\n", "\n", "# Verify the version of pyvo \n", "from pkg_resources import parse_version\n", "if parse_version(vo.__version__) < parse_version('1.4'):\n", " raise ImportError('pyvo version must be 1.4 or higher')\n", " \n", "print('pyvo version {version}'.format(version=vo.__version__))\n", "\n", "# Defining the ESO tap service to query for phase 3 products:\n", "tap = vo.dal.TAPService(\"https://archive.eso.org/tap_obs\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***\n", "## The `phase3v2.product_files` columns\n", "\n", "A data product contains exactly one science file, and contains 0 or more ancillary files. The `phase3v2.product_files` table lists the file components of all Phase 3 data products. For each file, its category, estimated size, file extension, original file name, and its the access point are provided.\n", "\n", "\n", "| Column | Description |\n", "|-------------------|-----------------------------------------------------------------------------------------|\n", "| product_id | ESO identifier of a published phase 3 product. A product is composed of a science file, and optionally of a number of ancillary files. Same as the dp_id field in ivoa.ObCore. |\n", "| archive_id | ESO identifier of a file belonging to the product: either a science file (eso_category: \"SCIENCE.*\") or an ancillary file (eso_category: \"ANCILLARY.*\"). If a science file, its id is the same as the id of the product (product_id=archive_id). Ancillary files are not listed in the ivoa.ObsCore table. |\n", "| eso_category | ESO file category; It starts with \"SCIENCE.\" or \"ANCILLARY.\" followed by one or more dot-separated and uppercased tokens describing the file at hand. For the full list of categories, please refer to the ESO Science Data Product standard available at: https://www.eso.org/sci/observing/phase3.html |\n", "| extension | File name extension (upper case) of the product file, e.g.: FITS, PNG, TAR, FZ, etc. |\n", "| original_filename | Original name of the product file before ingestion in the ESO archive. It may provide useful hints on the file content (usually described in the release description of the product, available at: https://archive.eso.org/wdb/wdb/adp/phase3_main/query?dp_id=here_the_product_id). It is not an identifier, as in general, there could be multiple files with the same original_filename but different archive_ids. |\n", "| access_url | The download link of the individual archive_id file. |\n", "| size_kb | Estimated size of the downloaded file in KBytes. It is only \"estimated\" as in general, FITS headers can be patched at download time, making the file size varying with time. |\n", "| internal_file_id | Internal file identification number, useful in combination with the provenance table. |\n", "\n", "This table can be joined with the ivoa.ObsCore table by matching the product_id with the obscore.dp_id. It can be used, for example, to find all the ancillary files of a certain category for a given instrument.\n", "\n", "_Note: This table does not cover products that are obsolete or deprecated; the ancillary files of those can only be found using DataLink, as in this example of the obsolete VMC science image whose product_id is ADP.2013-06-20T17:15:42.690, and whose DataLink is: \n", "http://archive.eso.org/datalink/links?ID=ivo://eso.org/ID?ADP.2013-06-20T17:15:42.690_\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example 1: How to find all the files belonging to a given product" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "