ESO Archive Frequently Asked Questions

How do I query the Archive programmatically via WDB?

 

 

The ESO Science Archive Facility allows browsing of its content like raw data, data products (e.g. Phase 3), ambient conditions, observation scheduling information, etc. via so-called query interfaces (for example the ones listed on the Archive home page) which are based on the same WDB (Web to DataBase) search engine.

All WDB-based query interfaces can be queried using a URL-based syntax; such syntax is explained here below.

In a simple sentence, to programmatically query a WDB-based interface you need to know:

  1. The URL representing the query result end-point of that interface
  2. How to specify query constraints using that URL
  3. How to specify which attributes should be shown in the query result
  4. How to specify the type of output (html, ascii [fixed length], csv, votable)
  5. How to set the maximum number of returned records to allow quick exploration and refinement of your query
  6. Examples

 

  1. The URL representing the query result end-point of that interface:

    To find a query result end-point you have to point your browser to the query form you want to query programmatically (e.g. the raw data query form and, as first thing, just press the search button.

    The obtained result page lists the result of a default query, and displays an html table containing a default set of columns (see Columns to display/hide below to learn how to change the default behaviour), and listing a default number of returned records (see Number of returned records below).

    The location bar of your browser now contains the correct query result end-point for the eso archive main query form; for this example the query result end-point is:
    http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?
    Please remember to add the question mark as last character after the token “/query”.

    Now that you know the query result end-point you can use it to programmatically query that interface, that is, you can modify the query, the set of columns and the number of returned records, according to your needs. This can be obtained by adding a number of ampersand (&) separated tokens to the query result end-point. The following describes how to construct such &-separated tokens.

  2. Query constraints:

    Each query interface is programmed to send a default SQL command to the database server. The user can modify the default SQL command by applying further constraints on each of the fields exposed by the interface, but always in addition to the default ones (for security reasons). Each further constraint is expressed with a token like this:

    field_name=expression

    where the expression acts on the indicated field name (which must not be repeated in the expression).

    The field names exposed by a form can be found in various ways:

    • hovering the mouse onto the displayed field label (either in the query form, or in the tabular column header of the query result page) opens up a little popup showing a description of the field prepended by the field name followed by a ':'. That is the field_name you should use.
    • pointing your browser to the query result enp-point, but replacing the last "/query" token with the token "/programmatic": that page shows you many various details that allow you to properly formulate your URL query, among them the field names are provided in the displayed table (e.g. http://archive.eso.org/wdb/wdb/eso/amber/programmatic).

     

    Example: Suppose the field is called mag (magnitude); a valid WDB expression is:

    >18 or <15

    The token to be sent to WDB would then be:

     

    mag=>18 or <15

    which is translated by WDB into the following SQL constraint:

     

    (mag > 18 or mag < 15)

    As said, the expression must not contain the field name; mag>18 or mag<15 is an invalid expression.

    Expressions can be just a single value or a more complex construct using the operators described at: http://archive.eso.org/wdb/html/wdb_query_help.html

    For example, if the query interface calls “instrument” and “exptime” the fields containing the instrument name and the exposure time, then the tokens:

    instrument=HARPS (field_name: instrument, expression: HARPS)
    exptime=>100 (field name: exptime, expression: >100)

    can be used to constrain the SQL to return just only HARPS entries, with an exposure time greater than 100 seconds.

    Please note that within string values are NOT to be enclosed in quotes.

    Multiple tokens are combined according to the following two rules.

    1. All tokens on different field names are logically AND-ed together. This means that:
      instrument=HARPS&exptime=>100
      are translated to SQL as in:
      instrument=’HARPS’ AND exptime > 100
    2. All tokens on the same field name are logically OR-ed together. This means that:
      instrument=HARPS&instrument=UVES
      is translated to SQL as in:
      (instrument=”HARPS” or instrument=”UVES”)
      Please note that the token
      instrument=HARPS or UVES
      will also result in the same SQL
      (instrument=”HARPS” or instrument=”UVES”)

      http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?instrument=HARPS&instrument=UVES&exptime=>100

     

  3. Columns displayed in the output result:

    The output of a default query consists of a table with a pre-defined set of default columns. Within the browser, when on the query form, it is possible to change the output columns by (un)ticking the checkboxes that are normally displayed on the left of each field name. Programmatically, this can be done by adding a token for each of the fields in the form of:

    tab_instrument=1 in order to display the instrument field
    tab_instrument=0 in order not to display the instrument field

  4. Output format:

    Allowed output formats (field wdbo) are:

    • html
    • ascii (generates a table of fixed length columns)
    • csv (generates records of comma separated values)
    • votable (generates a virtual observatory VOTable, not available for all query interfaces)

     

    The URL token to be used is, for example:
    wdbo=csv

  5. Number of returned records:

    The maximum number of returned records (by default typically set to 200) can be changed by using the top field, as in:

    top=1000

  6. Examples:

    Please note that on the top right of the query result page there is a "Programmatic" button which enables you to "save the http link". That URL can be used along with the wget or curl command to perform the same query within your terminal, on the command line.
    For instance:

    wget -O - "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?wdbo=html%2fdisplay&max_rows_returned=200&instrument=&tab_object=on&target=&resolver=simbad&ra=&dec=&box=00%2010%2000%C2%B0_or_hour=hours&tab_target_coord=on&format=SexaHour&wdb_input_file=&night=&stime=&starttime=12&etime=&endtime=12&tab_prog_id=on&prog_id=>o=&pi_coi=&obs_mode=&title=&tab_dp_cat=on&tab_dp_type=on&dp_type=&dp_type_user=&tab_dp_tech=on&dp_tech=&dp_tech_user=&tab_dp_id=on&dp_id=&origfile=&tab_rel_date=on&rel_date=&obs_name=&ob_id=&tab_exptime=on&exptime=&tab_filter_path=on&filter_path=&gris_path=&grat_path=&slit_path=&tab_instrument=on&add=&tab_tel_airm_start=on&tab_stat_instrument=on&tab_ambient=on&tab_stat_exptime=on&tab_HDR=on&tab_mjd_obs=on&aladin_colour=aladin_instrument&tab_stat_plot=on&order=&"

    will display an html page containing the results of your query.

    In order to get the query results in an ASCII file, one can add &wdbo=ascii" to use the above link:

    wget -O output.ascii.txt "http//archive.eso.org/wdb/wdb/eso/eso_archive_main/query?max_rows_returned=2&..........&wdbo=ascii"

    Below are a few basic examples how to write queries to the Archive.

    • Example to search for an astronomical object, e.g. Vega (target=vega) using the Simbad name resolver (resolver=simbad) and getting an ascii output (wdbo=ascii), target coordinates written in decimal degrees (tab_target_coord=true&format=DecimDeg), limited to the first 20 entries (top=20) and saving the output in the output.ascii.txt file, the wget command would be:

       

      wget -O output.ascii.txt "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&top=20&tab_target_coord=true&format=DecimDeg&wdbo=ascii"

      The output file looks like:

      > more output.ascii.txt
      # SIMBAD coordinates for vega :  18 36 56.3, +38 47 01.2.
      #
      OBJECT                 RA          DEC         Program_ID    Instrument  Category   Type                           Mode               
                Dataset ID                     Release_Date                             Exptime    Filter     MJD-OBS        Airmass
      ---------------------- ----------- ----------- ------------- ----------- ---------- ------------------------------ --------------------
      ---------- ------------------------------ ---------------------------------------- ---------- ---------- -------------- -------
      BS7001                 279.232333  38.786278   60.A-9250(A)  WFCAM       SCIENCE    OBJECT                         IMAGE              
                WFCAM.2011-11-08T04:15:23.904  Nov  8 2011                                   5.000            55873.177360   1.242
      OBJECT                 279.233500  38.783500   60.A-9051(A)  CRIRES      ACQUISITIO OBJECT                         IMAGE              
                CRIRE.2008-09-24T23:34:20.764  Sep 24 2009
      .....

      The parameters in the output ascii file are the ones ticked by default on the ESO main archive query form.

    •  

    •  Example to add a new parameter to the quey results, e.g. the original filename (Orig Name):
      • if one adds origfile=, all files with any possible original name of the files for vega will be searched for
      • if tab_origfile=on or tab_origfile=true or tab_origfile=1, this field will appear in the result of the query
      • if tab_origfile does not appear in the command line or if tab_origfile=0 then the field will not appear in the query result

       

      wget -O output.ascii.txt "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&top=20&tab_target_coord=true&format=DecimDeg&tab_origfile=on&ascii_out_mode=true"

      Now the output file looks like:

      > more output.ascii.txt
      # SIMBAD coordinates for vega :  18 36 56.3, +38 47 01.2.
      #
      OBJECT                 RA          DEC         Program_ID    Instrument  Category   Type                           Mode                           Dat
      aset ID                     Orig Name                      Release_Date                             Exptime    Filter     MJD-OBS        Airmass
      ---------------------- ----------- ----------- ------------- ----------- ---------- ------------------------------ ------------------------------ ---
      --------------------------- ------------------------------ ---------------------------------------- ---------- ---------- -------------- -------
      BS7001                 279.232333  38.786278   60.A-9250(A)  WFCAM       SCIENCE    OBJECT                         IMAGE                          WFC
      AM.2011-11-08T04:15:23.904  w20111108_00115.fits           Nov  8 2011                                   5.000            55873.177360   1.242
      OBJECT                 279.233500  38.783500   60.A-9051(A)  CRIRES      ACQUISITIO OBJECT                         IMAGE                          CRI
      RE.2008-09-24T23:34:20.764  CRIRES_ACQ_SLIT268_0001.fits   Sep 24 2009                                   0.303 HX5E-2     54733.982185   2.260
      .....

    • For a very simple list of results, one would have to set all the default ticked fields on the ESO main archive query form to 0 as shown in the following example:

      wget -O output.ascii.dp_id.txt "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_prog_id=0&tab_insmode=0&tab_dp_cat=0&tab_dp_type=0&tab_dp_tech=0&tab_rel_date=0&tab_exptime=0&tab_filter_path=0&tab_tel_airm_start=0&target=vega&resolver=simbad&max_rows_returned=2&tab_target_coord=0&tab_instrument=0&tab_mjd_obs=0&ascii_out_mode=true"

      > more output.ascii.dp_id.txt
      # SIMBAD coordinates for vega :  18 36 56.3, +38 47 01.2.
      #
      OBJECT                 RA          DEC         Dataset ID                    
      ---------------------- ----------- ----------- ------------------------------
      BS7001                 18:36:55.76 +38:47:10.6 WFCAM.2011-11-08T04:15:23.904
      OBJECT                 18:36:56.04 +38:47:00.6 CRIRE.2008-09-24T23:34:20.764

    •  

     

    More info on parameters of the eso_archive_main query form, for which a selection of them follows:

     

    • top (or max_rows_returned): controls the maximum numbers of rows to be returned; used by the "Return a maximum of" option.
    • Parameters governing extra fields to be returned:
      • tab_object: target information
      • tab_dp_cat: category information
      • tab_target_coord: actual target coordinates
      • tab_night: extra night information
      • tab_obs_name: actual name of the observing block
      • tab_origfile: adds the original file name as provided to the archive system
    • Common query parameters:
      • target: target name; it must be used jointly with resolver, so that the target is converted to coordinates.
      • resolver: one of simbad or ned; indicates which name resolver service will be used to convert the target name into coordinates.
      • ra: right ascension; only if no target name is provided; can be a blank separated sexagesimal value, or a decimal one. See deg_or_hour.
      • dec: declination; only if no target name is provided; can be a blank separated sexagesimal value, or a decimal one.
      • deg_or_hour: can be one of degrees, or hours, and affects only the interpretation of the ra field when given in sexagesimal format.
      • format: affects how coordinates are going to be shown in the results. SexaHour means sexagesimal format, with RA in HMS format; DecimHour means decimal format, with RA in hours, and DecimDeg means decimal format, with both RA and Dec in degrees. Requires tab_target_coord set to true.
    • Format parameters:
      • ascii_out_mode: when set to true will generate a ascii file with a table containing all the parameters ticked by default on the ESO Main archive query form
      • votable_out_mode: when set to true, returns an on-the-fly generated VOTable. The VOTable data format is a metadata-rich output format, that can be manipulated by GUI tools such as TOPCAT or command-line tools

    Please remember that different query forms might use different field names, always check the /programmatic end point.