ESO Archive Frequently Asked Questions
How do I query the Archive programmatically via WDB?
The ESO Science Archive Facility allows browsing of its content like raw data, data products (e.g. Phase 3), ambient conditions, observation scheduling information, etc. via so-called query interfaces (for example the ones listed on the Archive home page) which are based on the same WDB (Web to DataBase) search engine.
All WDB-based query interfaces can be queried using a URL-based syntax; such syntax is explained here below.
In a simple sentence, to programmatically query a WDB-based interface you need to know:
- The URL representing the query result end-point of that interface
- How to specify query constraints using that URL
- How to specify which attributes should be shown in the query result
- How to specify the type of output (html, ascii [fixed length], csv, votable)
- How to set the maximum number of returned records to allow quick exploration and refinement of your query
- Examples
- The URL representing the query result end-point of that interface:
To find a query result end-point you have to point your browser to the query form you want to query programmatically (e.g. the raw data query form and, as first thing, just press the search button.
The obtained result page lists the result of a default query, and displays an html table containing a default set of columns (see Columns to display/hide below to learn how to change the default behaviour), and listing a default number of returned records (see Number of returned records below).
The location bar of your browser now contains the correct query result end-point for the eso archive main query form; for this example the query result end-point is:
http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?
Please remember to add the question mark as last character after the token “/query”.Now that you know the query result end-point you can use it to programmatically query that interface, that is, you can modify the query, the set of columns and the number of returned records, according to your needs. This can be obtained by adding a number of ampersand (&) separated tokens to the query result end-point. The following describes how to construct such &-separated tokens.
- Query constraints:
Each query interface is programmed to send a default SQL command to the database server. The user can modify the default SQL command by applying further constraints on each of the fields exposed by the interface, but always in addition to the default ones (for security reasons). Each further constraint is expressed with a token like this:
field_name=expression
where the expression acts on the indicated field name (which must not be repeated in the expression).
The field names exposed by a form can be found in various ways:
- hovering the mouse onto the displayed field label (either in the query form, or in the tabular column header of the query result page) opens up a little popup showing a description of the field prepended by the field name followed by a ':'. That is the field_name you should use.
- pointing your browser to the query result enp-point, but replacing the last "/query" token with the token "/programmatic": that page shows you many various details that allow you to properly formulate your URL query, among them the field names are provided in the displayed table (e.g. http://archive.eso.org/wdb/wdb/eso/amber/programmatic).
Example: Suppose the field is called mag (magnitude); a valid WDB expression is:
>18 or <15
The token to be sent to WDB would then be:
mag=>18 or <15
which is translated by WDB into the following SQL constraint:
(mag > 18 or mag < 15)
As said, the expression must not contain the field name; mag>18 or mag<15 is an invalid expression.
Expressions can be just a single value or a more complex construct using the operators described at: http://archive.eso.org/wdb/html/wdb_query_help.html
For example, if the query interface calls “instrument” and “exptime” the fields containing the instrument name and the exposure time, then the tokens:
instrument=HARPS (field_name: instrument, expression: HARPS)
exptime=>100 (field name: exptime, expression: >100)can be used to constrain the SQL to return just only HARPS entries, with an exposure time greater than 100 seconds.
Please note that within string values are NOT to be enclosed in quotes.
Multiple tokens are combined according to the following two rules.
- All tokens on different field names are logically AND-ed together. This means that:
instrument=HARPS&exptime=>100
are translated to SQL as in:
instrument=’HARPS’ AND exptime > 100 - All tokens on the same field name are logically OR-ed together. This means that:
instrument=HARPS&instrument=UVES
is translated to SQL as in:
(instrument=”HARPS” or instrument=”UVES”)
Please note that the token
instrument=HARPS or UVES
will also result in the same SQL
(instrument=”HARPS” or instrument=”UVES”)
- Columns displayed in the output result:
The output of a default query consists of a table with a pre-defined set of default columns. Within the browser, when on the query form, it is possible to change the output columns by (un)ticking the checkboxes that are normally displayed on the left of each field name. Programmatically, this can be done by adding a token for each of the fields in the form of:
tab_instrument=1 in order to display the instrument field
tab_instrument=0 in order not to display the instrument field - Output format:
Allowed output formats (field wdbo) are:
- html
- ascii (generates a table of fixed length columns)
- csv (generates records of comma separated values)
- votable (generates a virtual observatory VOTable, not available for all query interfaces)
The URL token to be used is, for example:
wdbo=csv
- Number of returned records:
The maximum number of returned records (by default typically set to 200) can be changed by using the top field, as in:
top=1000
- Examples:
Please note that on the top right of the query result page there is a "Programmatic" button which enables you to "save the http link". That URL can be used along with the wget or curl command to perform the same query within your terminal, on the command line.
For instance:
wget -O - "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?wdbo=html%2fdisplay&max_rows_returned=200&instrument=&tab_object=on&target=&resolver=simbad&ra=&dec=&box=00%2010%2000%C2%B0_or_hour=hours&tab_target_coord=on&format=SexaHour&wdb_input_file=&night=&stime=&starttime=12&etime=&endtime=12&tab_prog_id=on&prog_id=>o=&pi_coi=&obs_mode=&title=&tab_dp_cat=on&tab_dp_type=on&dp_type=&dp_type_user=&tab_dp_tech=on&dp_tech=&dp_tech_user=&tab_dp_id=on&dp_id=&origfile=&tab_rel_date=on&rel_date=&obs_name=&ob_id=&tab_exptime=on&exptime=&tab_filter_path=on&filter_path=&gris_path=&grat_path=&slit_path=&tab_instrument=on&add=&tab_tel_airm_start=on&tab_stat_instrument=on&tab_ambient=on&tab_stat_exptime=on&tab_HDR=on&tab_mjd_obs=on&aladin_colour=aladin_instrument&tab_stat_plot=on&order=&"
will display an html page containing the results of your query.
In order to get the query results in an ASCII file, one can add &wdbo=ascii" to use the above link:
wget -O output.ascii.txt "http//archive.eso.org/wdb/wdb/eso/eso_archive_main/query?max_rows_returned=2&..........&wdbo=ascii"
Below are a few basic examples how to write queries to the Archive.
- Example to search for an astronomical object, e.g. Vega (target=vega) using the Simbad name resolver (resolver=simbad) and getting an ascii output (wdbo=ascii), target coordinates written in decimal degrees (tab_target_coord=true&format=DecimDeg), limited to the first 20 entries (top=20) and saving the output in the output.ascii.txt file, the wget command would be:
wget -O output.ascii.txt "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&top=20&tab_target_coord=true&format=DecimDeg&wdbo=ascii"
The output file looks like:
> more output.ascii.txt
# SIMBAD coordinates for vega : 18 36 56.3, +38 47 01.2.
#
OBJECT RA DEC Program_ID Instrument Category Type Mode
Dataset ID Release_Date Exptime Filter MJD-OBS Airmass
---------------------- ----------- ----------- ------------- ----------- ---------- ------------------------------ --------------------
---------- ------------------------------ ---------------------------------------- ---------- ---------- -------------- -------
BS7001 279.232333 38.786278 60.A-9250(A) WFCAM SCIENCE OBJECT IMAGE
WFCAM.2011-11-08T04:15:23.904 Nov 8 2011 5.000 55873.177360 1.242
OBJECT 279.233500 38.783500 60.A-9051(A) CRIRES ACQUISITIO OBJECT IMAGE
CRIRE.2008-09-24T23:34:20.764 Sep 24 2009
.....
The parameters in the output ascii file are the ones ticked by default on the ESO main archive query form.
- Example to add a new parameter to the quey results, e.g. the original filename (Orig Name):
- if one adds origfile=, all files with any possible original name of the files for vega will be searched for
- if tab_origfile=on or tab_origfile=true or tab_origfile=1, this field will appear in the result of the query
- if tab_origfile does not appear in the command line or if tab_origfile=0 then the field will not appear in the query result
wget -O output.ascii.txt "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&top=20&tab_target_coord=true&format=DecimDeg&tab_origfile=on&ascii_out_mode=true"
Now the output file looks like:
> more output.ascii.txt
# SIMBAD coordinates for vega : 18 36 56.3, +38 47 01.2.
#
OBJECT RA DEC Program_ID Instrument Category Type Mode Dat
aset ID Orig Name Release_Date Exptime Filter MJD-OBS Airmass
---------------------- ----------- ----------- ------------- ----------- ---------- ------------------------------ ------------------------------ ---
--------------------------- ------------------------------ ---------------------------------------- ---------- ---------- -------------- -------
BS7001 279.232333 38.786278 60.A-9250(A) WFCAM SCIENCE OBJECT IMAGE WFC
AM.2011-11-08T04:15:23.904 w20111108_00115.fits Nov 8 2011 5.000 55873.177360 1.242
OBJECT 279.233500 38.783500 60.A-9051(A) CRIRES ACQUISITIO OBJECT IMAGE CRI
RE.2008-09-24T23:34:20.764 CRIRES_ACQ_SLIT268_0001.fits Sep 24 2009 0.303 HX5E-2 54733.982185 2.260
.....
- For a very simple list of results, one would have to set all the default ticked fields on the ESO main archive query form to 0 as shown in the following example:
wget -O output.ascii.dp_id.txt "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_prog_id=0&tab_insmode=0&tab_dp_cat=0&tab_dp_type=0&tab_dp_tech=0&tab_rel_date=0&tab_exptime=0&tab_filter_path=0&tab_tel_airm_start=0&target=vega&resolver=simbad&max_rows_returned=2&tab_target_coord=0&tab_instrument=0&tab_mjd_obs=0&ascii_out_mode=true"
> more output.ascii.dp_id.txt
# SIMBAD coordinates for vega : 18 36 56.3, +38 47 01.2.
#
OBJECT RA DEC Dataset ID
---------------------- ----------- ----------- ------------------------------
BS7001 18:36:55.76 +38:47:10.6 WFCAM.2011-11-08T04:15:23.904
OBJECT 18:36:56.04 +38:47:00.6 CRIRE.2008-09-24T23:34:20.764
More info on parameters of the eso_archive_main query form, for which a selection of them follows:
- top (or max_rows_returned): controls the maximum numbers of rows to be returned; used by the "Return a maximum of" option.
- Parameters governing extra fields to be returned:
- tab_object: target information
- tab_dp_cat: category information
- tab_target_coord: actual target coordinates
- tab_night: extra night information
- tab_obs_name: actual name of the observing block
- tab_origfile: adds the original file name as provided to the archive system
- Common query parameters:
- target: target name; it must be used jointly with resolver, so that the target is converted to coordinates.
- resolver: one of simbad or ned; indicates which name resolver service will be used to convert the target name into coordinates.
- ra: right ascension; only if no target name is provided; can be a blank separated sexagesimal value, or a decimal one. See deg_or_hour.
- dec: declination; only if no target name is provided; can be a blank separated sexagesimal value, or a decimal one.
- deg_or_hour: can be one of degrees, or hours, and affects only the interpretation of the ra field when given in sexagesimal format.
- format: affects how coordinates are going to be shown in the results. SexaHour means sexagesimal format, with RA in HMS format; DecimHour means decimal format, with RA in hours, and DecimDeg means decimal format, with both RA and Dec in degrees. Requires tab_target_coord set to true.
- Format parameters:
- ascii_out_mode: when set to true will generate a ascii file with a table containing all the parameters ticked by default on the ESO Main archive query form
- votable_out_mode: when set to true, returns an on-the-fly generated VOTable. The VOTable data format is a metadata-rich output format, that can be manipulated by GUI tools such as TOPCAT or command-line tools
Please remember that different query forms might use different field names, always check the /programmatic end point.
- Example to search for an astronomical object, e.g. Vega (target=vega) using the Simbad name resolver (resolver=simbad) and getting an ascii output (wdbo=ascii), target coordinates written in decimal degrees (tab_target_coord=true&format=DecimDeg), limited to the first 20 entries (top=20) and saving the output in the output.ascii.txt file, the wget command would be: