Change Log of the programmatic and tool interfaces

The table below, in reverse chronological order, lists all the changes to the programmatic and tool interfaces that could affect your scripts or anyway your way of accessing the ESO science archive via TAP, SSAP, or via one of the URLs specified in the Learn dataset actions. Keep an eye on it, to make sure to be up-to-date with this newly born and still evolving functionality of the ESO Science Archive.

Date	Context	Description of change
2024-04-17	tap_obs dbo.raw table	Three new columns have been added to the dbo.raw table: s_region, lambda_min, lambda_max. column name purpose example and notes s_region it allows fast spatial queries -- query for raw files within 0.1 deg radius from NGC 3627 select * from dbo.raw where intersects(s_region, circle('', 170.063, 12.99, 0.1))=1 -- Note: Many raw files (flats, biases, etc.) have no valid -- sky coordinatess assigned: their s_region remains null. -- Hence, a s_region condition like the one above automatically -- filters out any such files. -- Please keep in mind these facts: -- 1.- the accuracy of sky coordinates of the raw data varies -- depending on the telescope/instrument used. -- 2.- The s_region of the raw data represents a point in the sky based -- on those same sky coordinates, with the same limitations. -- 3.- The s_region of the raw data, contrarily to the s_region of the -- science products (ivoa.ObsCore), does not provide the actual -- footprint of the observation. -- -- Allow enough tolerance by widening the search radius to cope with -- the above inaccuracies. lambda_min lambda_max It allows queries by wavelength coverage (in nm) of imaging observations. No need to remember complicated filter names, whose names differ from instrument to instrument. -- find imaging raw files -- whose filter band covers the Paschen β wavelength. select * from dbo.raw where lambda_min < 1282 and lambda_max > 1282 -- You could add a constraint on the band width -- e.g., to search for narrow band filters only: -- lambda_max - lambda_min < 20 -- Important: Spectra, data cubes and other kinds of raw data are not supported. -- That is, a query constraint on lambda removes from the results any non-imaging -- data types. -- In fact, you could use the constraint: -- lambda_min is null -- to exclude imaging files from your results. Added examples in the tap_obs query pull-down menu: (raw02, raw03, raw04)
2024-04-04	jupyter notebook on ancillary files	A fully-detailed tutorial on how to query for ancillary files via TAP is available: Query ancillary files.
2023-05-03	tap_obs centroid	The computed centroid of a footprint as large an hemisphere (not more, not less) always returns (0,90). (from SQLServer documentation: In SQL Server 2012 (11.x) and higher, EnvelopeCentre returns the center of the circle representing the envelope of this instance as a point. For all large objects as defined by EnvelopeAngle() = 180, EnvelopeCenter() will return (90,0). This method isn't precise.)
2022-03-24	tap_obs meteorological tables	The measurements of the ESO meteorological stations are now avaiable for programmatic access under the tap_obs service: table name location coverage asm.meteo_apex Llano de Chanjantor 2006 onwards asm.meteo_lasilla Cerro La Silla 1994 onwards asm.meteo_paranal Cerro Paranal 1998 onwards
2021-08-24	jupyter notebook auth & auth	You can now execute and interact with the ESO Programmatic Authentication & Authorisation python jupyter notebook. To do that: follow this link at MyBinder wait up to a couple of minutes for MyBinder to start the repository click on Programmatic_Authentication_and_Authorisation.ipynb to lunch it to interact with it see e.g. this python jupyter notebook primer
2021-08-02	programmatic access auth & auth	Authentication & Authorisation are now supported by the programmatic access layer. Read full description with examples ... Shortly recapped here the main new supported features: Token authentication Authorised data searches to the dbo.raw table (using tap_obs) with the purpose of exercising specific user's permissions that allow to access data whose metadata are not accessible anonymously. Authorised datalink access to data with protected metadata to find related data files Authorised calSelector access to find calibration reference files of data with protected metadata
2021-08-02	jupyter notebook on auth & auth	A fully-detailed tutorial on how to authenticate and perform authenticated queries via TAP is available: ESO Programmatic Authentication & Authorisation python jupyter notebook.
2021-07-21	tap_obs ObsCore table	The ivoa.ObsCore table was revised and curated: last_mod_date New column: the last_mod_date provides the timestamp of the last modification to the record metadata, and can be used to find, in combination with the publication_date, all records that have been modified since a certain date. release_description New column: the release_description provides the URL to the PDF descrbing the data release the specific dataset belongs to. bib_reference Curated: the bib_reference column now correctly report the bibcode or doi. filter, o_calib_status Curated: the columns filter and o_calib_status were curated for Paranal and La Silla observations (filter might have been NULL for some records, o_calib_status reported the incorrect value "any" for about 2M records) ALMA Curated: Various columns were curated for ALMA observations (e.g., s_resolution, pol_states, em_xel, and others). Though, please note that the last ALMA observation published here is from March 2020. we will update the ALMA records to the latest observations within the next 4 weeks. access_url Minor update: The access_url value is now defined using the obs_publisher_did (for the user nothing really changes, as the previous value definition, based on the internal ESO id instead, and the new one are both supported by datalink)
2021-05-04	Datalink	Datalink now supports the ESO Night Log files associated to raw frames, containing any annotation by the telescope operator while carrying out the observation (e.g. instrument problems, ambient conditions, etc.).
2021-05-04	tap_obs	New TAP version: The TAP serving observational data and measurements (tap_obs) has been upgraded to the IVOA TAP v1.1 standard (was v1.0). Its endpoint remains the same. The list of changes are described in the TAP1.1 standard, in the Appendix A "Changes from TAP-1.0 to TAP-1.1". The most important differences are: Supported output formats: mime alias comment application/x-votable+xml votable as before   votable/td dropped application/x-votable+xml;serialization=fits votable_fits was aliased votable/fits application/x-votable+xml;serialization=binary votable_binary was aliased votable/b application/x-votable+xml;serialization=binary2 votable_binary2 was aliased votable/b2 text/csv csv new text/tab-separated-values tsv new application/fits fits as before text/plain txt new alias: the previously aliased "text" was formatting the text differently text/html html new application/json json as before The addition of arraysize and xtype (also shown in the votable output) to the TAP_SCHEMA.columns table which permits: the definition of arrays in column cells (e.g. to convey probability distribution functions) a full characterisation of the TAP datatypes via the VOTable triplet (xtype, datatype, arraysize)
2021-05-04	ADQL	New user-defined-function: ESO_SUBSTRING(string, from, length) Example: eso_substring("string",3,4) = "ring". Please note that lower() and upper() functions are not needed as the SQLServer is case insensitive.
2021-04-22	Jupyter Notebooks ESO Archive Community Forum	[2021-04-22] Exploring the Follow up of GW events
2021-04-07	the TAP tab in this html page	Added ability to query the GAIA TAP service
2021-04-07	the TAP tab in this html page	Fixed some issues when using the "Create link to this page" URL; extra comments that were shown in the textarea when clicking on the "Show the URL" have been removed to ease copy&paste
2020-09-17	the TAP tab in this html page	Added query example for LHATPRO data taken around a certain position and time (see here)
2020-09-17	tap_obs lhatpro_paranal	The original LHATPRO table, that contained the measurements of just only one of the 3 LHATPRO radiometers, has been decommissioned in favour of the new LHATPRO tables (see next news immediately below).
2020-09-17	tap_obs LHATPRO tables	Added 3 tables providing data measured, since 2014, by the 3 LHATPRO radiometers installed in Paranal, including precipitable water vapour, infrared temperature, liquid water path, plus the atmospheric profiles of abs. and rel. humidity, and of infrared temperatures at 39 different altitudes over the Paranal platform: Description of the new LHATPROs tables: asm.lhatpros_paranal asm.lhatpros_paranal_irt asm.lhatpros_paranal_profiles
2020-09-04	scripted access	Added demo script to illustrate how to download science raw frames with authentication, allowing to download proprietary files if authorised. eso_authenticated_download_raw_and_calibs.py
2020-09-03	Jupyter Notebooks	More python jupyter notebook examples are available in the ESO Archive Community Forum. Currently available: Access to Spectroscopic Data: Example #1 Access to Catalogues: Example #1 and example #2
2020-04-23	Jupyter Notebooks	The new page: ESO Science Archive Programmatic: HOWTOs offers a series of Jupyter Notebooks to learn how to script your access in python. Currently available: How to download data (both anonymously and via authentication) How to query for reduced data (including queries to find areas of the sky covered in different bands for multiband photometry) More to come soon... When new notebooks will be available, they will be announced in this page.
2020-04-23	Authentication	Programmatic authenticated access to proprietary data is now fully supported via both basic authentication and JWT tokens (OAuth2.0). (Read more...)
2020-04-23	Cutout service	Positional and spectral cutouts of science data products is now available. The service is fully compliant to the IVOA Server-side Operations for Data Access (SODA) protocol. (Read more...)
2020-04-23	DataLink	The ESO DataLink service now supports: proprietary data the SODA protocol, to allow an easy programmatic access to the VO-compliant ESO cutout services (Read more...)
2020-04-23	tap_obs ObsCore table	The ivoa.ObsCore table now supports the ALMA data, along with the already served reduced data from La Silla Paranal Observatory (including APEX) [LPO]. Pros: Integrated data discovery experience for both ALMA and LPO processed data Full support for powerful spatial queries Ability to cross-match LPO and ALMA data Limitations: Currently, new ALMA data are published here manually about twice a month. We are working to make this operation fully automatic, and to remove this publishing delay. Time scale: Q4 2020. The datalink service is not available for ALMA data. The cutout service is not available for ALMA data.
2020-04-21	tap_obs dbo.raw table	Boosted performance (up to a factor 10) of the table that serves raw data. Also added 1 new column: datalink_url which, for the raw file at hand, provides access to its calibration reference files (see news 2019-11-25 calibration files service below) and/or to its pipeline-processed products (if it exists)
2019-12-04	tap_obs ObsCore table	added 4 new columns: multi_ob, obstech, p3orig, and publication_date (see page description above)
2019-11-25	scripted access	added demo script to illustrate how to download both the science raw frames, and the calibration files needed to calibrate them (via the DataLink and calibration files services): eso_download_raw_and_calibs.py
2019-11-25	DataLink	the ESO DataLink service has been modified to add, when exercised on science raw frames, the access urls (one for raw2raw and one for raw2master) that can be used to download the list of calibration files needed to calibrate them. (See documentation)
2019-11-25	new archive service	the calibration files service is now available via http; it allows the users to discover which calibration files are needed to calibrate a given science raw frame, and then to download them.
2019-11-15	ESASky v3.2	The lastes version of the ESA portal that lets you visualise and download public astronomical data, offers now ESO data via the ESO TAP and DataLink services. Great use of VO standards for the interoperability of two leading astronomical institutes in Europe!
2019-02-20	async Query Manager	added ability to set the execution duration of an async job (default: 60 s, maximum: 3600 s).
2019-02-13	scripted access	eso_raw.py was updated to reflect a change to the table structure (the access_url field points to the datalink and no longer to the direct file download link).
2019-01-22	the TAP tab in this html page	Added mechanism to customise query form via URL (useful e.g. for archive news). Specs available. Correspondingly, added the Create link to this page button in the TAP tab, so that the tab (with its query and form settings) can be bookmarked and sent to colleagues.
2018-10-23	tap_obs tap_cat	re-organised and improved query examples (in the "Choose a query..." pulldown menu)
2018-10-22	tap_cat	fixed erroneous datatypes of various catalog columns
2018-10-16	dataset actions	fixed issue that did not allow to use the dataset ID of a raw file
2018-09-17	scripted access	added example to query and download raw data in python (eso_raw.py)
2018-08-02	tap_obs	table of raw data renamed to dbo.raw
2018-07-02	this html page	creation