ESO Archive Frequently Asked Questions

How do I export search results?

In order to export search results from large datasets, users can follow one of the two options below:

Export links

There is an export option into three different ASCII data formats, with values separated by tabs (\t), commas, or pipe simbols (|). This can be combined with the "Return a maximum of n rows" setting, in order to retrieve a really large number of rows.
This can be accomplished by performing the following actions on the ESO Archive main query form:

  • Check the Use tabular output radio button
  • Check the Exports results to file checkbox, selecting additionally whether a Tab-separated-values, Comma-separated-values, or Pipe-separated-values is to be generated.
  • Optionally, set the "Return a maximum of" to the number of expected rows (ie, 20000).
  • When more than 10000 rows are expected, it is advised to uncheck the "Number of Frames / Instrument" and "Cumulative Exptime / Filter" options, as they might cause a timeout precluding the form from completion, and might end up not showing the export link.

After the query is completed (it might take as long as 15 minutes for a 20000 results query), the download link provided will be available for XXX time [please check] for reuse.

Command-line downloads, and VOTables

For users who prefer to download query results using command-line tools, they will have to build a query URL that can be used with downloading tools such as curl or wget.

Building a query URL

Query parameters have to be identified on the ESO Main archive query form (where they are identified as INPUT HTML tags). A selection of them follows:

  • max_rows_returned: controls the maximum numbers of rows to be returned; used by the "Return a maximum of" option.
  • Parameters governing extra fields to be returned:
    • tab_object: target information
    • tab_dp_cat: category information
    • tab_target_coord: actual target coordinates
    • tab_night: extra night information
    • tab_obs_name: actual name of the observing block
    • tab_origfile: adds the orinal file name as provided to the archive system (i.e., as saved in the PI pack)
  • Common query parameters:
    • target: target name; it must be used jointly with resolver, so that the target is converted to coordinates.
    • resolver: one of simbad or ned; indicates which name resolver service will be used to convert the target name into coordinates.
    • ra: right ascension; only if no target name is provided; can be a blank separated sexagesimal value, or a decimal one. See deg_or_hour.
    • dec: declination; only if no target name is provided; can be a blank separated sexagesimal value, or a decimal one.
    • deg_or_hour: can be one of degrees, or hours, and affects only the interpretation of the ra field when given in sexagesimal format.
    • format: affects how coordinates are going to be shown in the results. SexaHour means sexagesimal format, with RA in HMS format; DecimHour means decimal format, with RA in hours, and DecimDeg means decimal format, with both RA and Dec in degrees. Requires tab_target_coord set to true.
  • Format parameters:

Which if used would cause a browser, or command line tool, to download up to 2000 rows (max_rows_returned=2000) of calibration data (category=CALIB), corresponding to observations around an object known as Vega (target=vega), as resolved by Simbad (resolver=simbad) in VOTable format (votable_out_mod=true), with target coordinates written in decimal degrees (tab_target_coord=true&format=DecimDeg).
In order to download such VOTable, we would have to issue a suitable curl or wget command (see examples below), or used in a web browser, or users' favourite download manager.

Curl command

curl -o output.votable.xml  "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&tab_dp_cat=true&category=SCIENCE&category=CALIB&max_rows_returned=2000&deg_or_hour=degrees&tab_target_coord=true&format=DecimDeg&votable_out_mode=true"

Wget command

wget -O output.votable.xml "http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&tab_dp_cat=true&category=SCIENCE&category=CALIB&max_rows_returned=2000&deg_or_hour=degrees&tab_target_coord=true&format=DecimDeg&votable_out_mode=true"

In order to manipulate the VOTable from the command line, a package such as STILTS can be used. A few examples follow using the URL above, but more information can be found on the STILTS package documentation.

STILTS commands

The output.votable.xml file can later be transformed into ASCII, CSV, TSV, or even a FITS binary table using the STILTS package, or interactively with the TOPCAT table editor (based on the same libraries as STILTS).

As the STILTS toolset allows for the concept of "remote files", the URL can be used as input parameter:

stilts tcopy ofmt=ascii in="http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&tab_dp_cat=true&category=SCIENCE&category=CALIB&max_rows_returned=2000&deg_or_hour=degrees&tab_target_coord=true&format=DecimDeg&votable_out_mode=true" out=-
 

By using the tpipe option, some additional filtering can be performed:

stilts tpipe ofmt=text in="http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?tab_object=true&target=vega&resolver=simbad&tab_dp_cat=true&category=SCIENCE&category=CALIB&max_rows_returned=2000&deg_or_hour=degrees&tab_target_coord=true&format=DecimDeg&votable_out_mode=true" out=- cmd='fixcolnames' cmd="select 'equals(data_products_dp_cat,\"SCIENCE\")'" cmd="select 'equals(data_products_dp_type,\"OBJECT\")'"
 

The fixcolnames command substitutes characters problematic for STILTS in column names by underscores (in the case of the ESO archive, the dot between data_products and dp_cat, for instance.