Change Log of the programmatic and tool interfaces


The table below, in reverse chronological order, lists all the changes to the programmatic and tool interfaces that could affect your scripts or anyway your way of accessing the ESO science archive via TAP, SSAP, or via one of the URLs specified in the Learn dataset actions. Keep an eye on it, to make sure to be up-to-date with this newly born and still evolving functionality of the ESO Science Archive.
DateContextDescription of change
2024-11-27 tap_obs tables:
  • dbo.raw
  • phase3v2.files
  • phase3v2.product_files
 The column access_estsize has been added to all the tables that provide links to archived files.
Keeping in mind that FITS headers can be patched at download time with the latest updates, such column allows to estimate the amount of disk space necessary when downloading files. The above reflects in the following tables:
dbo.raw
  • Added the access_estsize field expressing the integer number of kilobytes.
phase3v2.files
  • Added the access_estsize field expressing the integer number of kilobytes.
  • Removed the size field (unit: bytes).
phase3v2.product_files
  • Column size_kb renamed access_estsize
PS: Nothing changes for the ivoa.ObsCore table which already provided the access_estsize.
2024-11-27 Update: jupyter notebook on ancillary files An update of the tutorial on how to query for ancillary files via TAP has been published: Query ancillary files.
2024-04-17 tap_obs
dbo.raw table
Three new columns have been added to the dbo.raw table: s_region, lambda_min, lambda_max.

column namepurposeexample and notes
s_region it allows fast spatial queries 
-- query for raw files within 0.1 deg radius from NGC 3627

select * from dbo.raw
where intersects(s_region, circle('', 170.063, 12.99, 0.1))=1

-- Note: Many raw files (flats, biases, etc.) have no valid
-- sky coordinatess assigned: their s_region remains null.
-- Hence, a s_region condition like the one above automatically
-- filters out any such files.

-- Please keep in mind these facts:
-- 1.- the accuracy of sky coordinates of the raw data varies
-- depending on the telescope/instrument used.
-- 2.- The s_region of the raw data represents a point in the sky based
-- on those same sky coordinates, with the same limitations.
-- 3.- The s_region of the raw data, contrarily to the s_region of the
-- science products (ivoa.ObsCore), does not provide the actual
-- footprint of the observation.
--
-- Allow enough tolerance by widening the search radius to cope with
-- the above inaccuracies.
lambda_min
lambda_max		 It allows queries by wavelength coverage (in nm) of imaging observations.
No need to remember complicated filter names, whose names differ from instrument to instrument.		 
-- find imaging raw files
-- whose filter band covers the Paschen β wavelength.

select * from dbo.raw
where lambda_min < 1282 and lambda_max > 1282

-- You could add a constraint on the band width
-- e.g., to search for narrow band filters only:
-- lambda_max - lambda_min < 20 

-- Important: Spectra, data cubes and other kinds of raw data are not supported.
-- That is, a query constraint on lambda removes from the results any non-imaging
-- data types.
-- In fact, you could use the constraint:
-- lambda_min is null
-- to exclude imaging files from your results.
Added examples in the tap_obs query pull-down menu: (raw02, raw03, raw04)

2024-04-04 jupyter notebook on ancillary files A fully-detailed tutorial on how to query for ancillary files via TAP is available: Query ancillary files.
2023-05-03 tap_obs
centroid		 The computed centroid of a footprint as large an hemisphere (not more, not less) always returns (0,90).
(from SQLServer documentation: In SQL Server 2012 (11.x) and higher, EnvelopeCentre returns the center of the circle representing the envelope of this instance as a point. For all large objects as defined by EnvelopeAngle() = 180, EnvelopeCenter() will return (90,0). This method isn't precise.)
2022-03-24 tap_obs
meteorological tables		 The measurements of the ESO meteorological stations are now avaiable for programmatic access under the tap_obs service:
table namelocationcoverage
asm.meteo_apexLlano de Chanjantor2006 onwards
asm.meteo_lasillaCerro La Silla1994 onwards
asm.meteo_paranalCerro Paranal1998 onwards
2021-08-24 jupyter notebook
auth & auth		 You can now execute and interact with the ESO Programmatic Authentication & Authorisation python jupyter notebook. To do that:
2021-08-02 programmatic access
auth & auth


Authentication & Authorisation are now supported by the programmatic access layer.

Read full description with examples ...

Shortly recapped here the main new supported features:

  • Token authentication
  • Authorised data searches to the dbo.raw table (using tap_obs) with the purpose of exercising specific user's permissions that allow to access data whose metadata are not accessible anonymously.
  • Authorised datalink access to data with protected metadata to find related data files
  • Authorised calSelector access to find calibration reference files of data with protected metadata


2021-08-02 jupyter notebook on auth & auth


A fully-detailed tutorial on how to authenticate and perform authenticated queries via TAP is available: ESO Programmatic Authentication & Authorisation python jupyter notebook.
2021-07-21 tap_obs
ObsCore table


The ivoa.ObsCore table was revised and curated:
last_mod_dateNew column: the last_mod_date provides the timestamp of the last modification to the record metadata, and can be used to find, in combination with the publication_date, all records that have been modified since a certain date.
release_descriptionNew column: the release_description provides the URL to the PDF descrbing the data release the specific dataset belongs to.
bib_referenceCurated: the bib_reference column now correctly report the bibcode or doi.
filter, o_calib_statusCurated: the columns filter and o_calib_status were curated for Paranal and La Silla observations (filter might have been NULL for some records, o_calib_status reported the incorrect value "any" for about 2M records)
ALMACurated: Various columns were curated for ALMA observations (e.g., s_resolution, pol_states, em_xel, and others). Though, please note that the last ALMA observation published here is from March 2020. we will update the ALMA records to the latest observations within the next 4 weeks.
access_urlMinor update: The access_url value is now defined using the obs_publisher_did (for the user nothing really changes, as the previous value definition, based on the internal ESO id instead, and the new one are both supported by datalink)


2021-05-04 Datalink


Datalink now supports the ESO Night Log files associated to raw frames, containing any annotation by the telescope operator while carrying out the observation (e.g. instrument problems, ambient conditions, etc.).


2021-05-04 tap_obs


New TAP version: The TAP serving observational data and measurements (tap_obs) has been upgraded to the IVOA TAP v1.1 standard (was v1.0). Its endpoint remains the same. The list of changes are described in the TAP1.1 standard, in the Appendix A "Changes from TAP-1.0 to TAP-1.1".
The most important differences are:

  • Supported output formats:
    mimealiascomment
    application/x-votable+xmlvotableas before
     votable/tddropped
    application/x-votable+xml;serialization=fitsvotable_fitswas aliased votable/fits
    application/x-votable+xml;serialization=binaryvotable_binarywas aliased votable/b
    application/x-votable+xml;serialization=binary2votable_binary2was aliased votable/b2
    text/csvcsvnew
    text/tab-separated-valuestsvnew
    application/fitsfitsas before
    text/plaintxtnew alias: the previously aliased "text" was formatting the text differently
    text/htmlhtmlnew
    application/jsonjsonas before
  • The addition of arraysize and xtype (also shown in the votable output) to the TAP_SCHEMA.columns table which permits:
    • the definition of arrays in column cells (e.g. to convey probability distribution functions)
    • a full characterisation of the TAP datatypes via the VOTable triplet (xtype, datatype, arraysize)


2021-05-04 ADQL


New user-defined-function:

ESO_SUBSTRING(string, from, length)

Example: eso_substring("string",3,4) = "ring".

Please note that lower() and upper() functions are not needed as the SQLServer is case insensitive.


2021-04-22 Jupyter Notebooks
ESO Archive Community Forum


  [2021-04-22] Exploring the Follow up of GW events


2021-04-07the TAP tab in this html pageAdded ability to query the GAIA TAP service
2021-04-07the TAP tab in this html pageFixed some issues when using the "Create link to this page" URL; extra comments that were shown in the textarea when clicking on the "Show the URL" have been removed to ease copy&paste
2020-09-17the TAP tab in this html pageAdded query example for LHATPRO data taken around a certain position and time (see here)
2020-09-17tap_obs
lhatpro_paranal		The original LHATPRO table, that contained the measurements of just only one of the 3 LHATPRO radiometers, has been decommissioned in favour of the new LHATPRO tables (see next news immediately below).
2020-09-17tap_obs
LHATPRO tables		Added 3 tables providing data measured, since 2014, by the 3 LHATPRO radiometers installed in Paranal, including precipitable water vapour, infrared temperature, liquid water path, plus the atmospheric profiles of abs. and rel. humidity, and of infrared temperatures at 39 different altitudes over the Paranal platform:
Description of the new LHATPROs tables:
  • asm.lhatpros_paranal
  • asm.lhatpros_paranal_irt
  • asm.lhatpros_paranal_profiles
2020-09-04scripted accessAdded demo script to illustrate how to download science raw frames with authentication, allowing to download proprietary files if authorised.
eso_authenticated_download_raw_and_calibs.py
2020-09-03Jupyter NotebooksMore python jupyter notebook examples are available in the ESO Archive Community Forum.
Currently available:
2020-04-23Jupyter NotebooksThe new page: ESO Science Archive Programmatic: HOWTOs offers a series of Jupyter Notebooks to learn how to script your access in python.
Currently available:
  • How to download data (both anonymously and via authentication)
  • How to query for reduced data (including queries to find areas of the sky covered in different bands for multiband photometry)
  • More to come soon...
When new notebooks will be available, they will be announced in this page.
2020-04-23AuthenticationProgrammatic authenticated access to proprietary data is now fully supported via both basic authentication and JWT tokens (OAuth2.0). (Read more...)
2020-04-23Cutout servicePositional and spectral cutouts of science data products is now available. The service is fully compliant to the IVOA Server-side Operations for Data Access (SODA) protocol. (Read more...)
2020-04-23tap_obs
ObsCore table		The ivoa.ObsCore table now supports the ALMA data, along with the already served reduced data from La Silla Paranal Observatory (including APEX) [LPO].
Pros:
  • Integrated data discovery experience for both ALMA and LPO processed data
  • Full support for powerful spatial queries
  • Ability to cross-match LPO and ALMA data
Limitations:
  • Currently, new ALMA data are published here manually about twice a month. We are working to make this operation fully automatic, and to remove this publishing delay. Time scale: Q4 2020.
  • The datalink service is not available for ALMA data.
  • The cutout service is not available for ALMA data.
2020-04-21tap_obs
dbo.raw table		Boosted performance (up to a factor 10) of the table that serves raw data. Also added 1 new column: datalink_url which, for the raw file at hand, provides access to its calibration reference files (see news 2019-11-25 calibration files service below) and/or to its pipeline-processed products (if it exists)
2019-12-04tap_obs
ObsCore table		added 4 new columns: multi_ob, obstech, p3orig, and publication_date (see page description above)
2019-11-25scripted accessadded demo script to illustrate how to download both the science raw frames, and the calibration files needed to calibrate them (via the DataLink and calibration files services):
eso_download_raw_and_calibs.py
2019-11-25new archive servicethe calibration files service is now available via http; it allows the users to discover which calibration files are needed to calibrate a given science raw frame, and then to download them.
2019-11-15ESASky v3.2The lastes version of the ESA portal that lets you visualise and download public astronomical data, offers now ESO data via the ESO TAP and DataLink services. Great use of VO standards for the interoperability of two leading astronomical institutes in Europe!
2019-02-20async Query Manageradded ability to set the execution duration of an async job (default: 60 s, maximum: 3600 s).
2019-02-13scripted accesseso_raw.py was updated to reflect a change to the table structure (the access_url field points to the datalink and no longer to the direct file download link).
2019-01-22the TAP tab in this html pageAdded mechanism to customise query form via URL (useful e.g. for archive news). Specs available. Correspondingly, added the Create link to this page button in the TAP tab, so that the tab (with its query and form settings) can be bookmarked and sent to colleagues.
2018-10-23tap_obs
tap_cat		re-organised and improved query examples (in the "Choose a query..." pulldown menu)
2018-10-22tap_catfixed erroneous datatypes of various catalog columns
2018-10-16dataset actionsfixed issue that did not allow to use the dataset ID of a raw file
2018-09-17scripted accessadded example to query and download raw data in python (eso_raw.py)
2018-08-02tap_obstable of raw data renamed to dbo.raw
2018-07-02this html pagecreation