TAP: Manage your Asynchronous Queries
Credit: This page is losely based on an equivalent query management page of the Space Telescope Science Institute, whose developers are gratefully thanked.
This page contains simple web forms that can manage the asynchronous ADQL(1) queries
sent to the ESO Tabular Access Protocol (TAP) services. Examples and information on how to control asynchronous jobs, also in a programmatic way, are provided.
The starting point: you launched an asynchronous query because you know or expect that your query is going to take long time (more than 60 seconds) to execute,
and therefore you want to avoid any time out.
If you reached this page, very likely it is beacuse you have submitted an asynchronous query to one of the ESO TAP services. What you are looking for is a way to get to the results of your query.
This page helps you acheiving that.
Please, visit the Science Archive Programmatic and Tool Access page in case you do not know how to submit an asynchronous query.
Queries can also be submitted on the command line, using wget
, or similar programs, as shown here:
wget 'http://archive.eso.org/tap_obs/async' --post-data 'REQUEST=doQuery&LANG=ADQL&MAXREC=2&FORMAT=text&QUERY=select%20top%202%20*%20from%20ivoa.ObsCore' -O -
curl -Lw -X POST 'http://archive.eso.org/tap_obs/async' -d 'REQUEST=doQuery&LANG=ADQL&MAXREC=2&FORMAT=text&QUERY=select%20top%202%20*%20from%20ivoa.ObsCore'
Said more technically, this page helps you managing an asynchronous job that was created behind the scenes when you submitted your query.
- It is here assumed that the user has already submitted (http POST) an asynchronous query (ADQL(1)) to a given TAP(2) end point (e.g.: http://archive.eso.org/tap_obs/async)
- The system immediately answered with an XML document containing, among other things, the job identifier (see an example in the floating box).
Example of an answer received when submitting an asynchronous query:
<job xsi:schemaLocation="http://www.ivoa.net/xml/UWS/v1.0 http://www.ivoa.net/xml/UWS/v1.0 http://www.w3.org/1999/xlink http://www.w3.org/1999/xlink.xsd http://www.w3.org/2001/XMLSchema http://www.w3.org/2001/XMLSchema.xsd">
<quote xsi:nil="true"/><startTime xsi:nil="true"/>
<parameter id="query">SELECT TOP 10 * FROM ivoa.ObsCore</parameter>
The minimal job management
Using the job identifier the user:
- can optionally negotiate the amount of time the job is allowed to run or the amount of time its results will be stored on the server. This negotiation can only take place before the next step (RUN).
- must ask to execute the job, by issuing a PHASE=RUN command
- can poll the server to request information about the status of the job (e.g.: PENDING, EXECUTING, COMPLETED, ERROR).
- can optionally request to abort the job.
- can, once in status COMPLETED, retrieve the results via a URL provided by the system in 3 different ways:
- found in the XML response of a COMPLETED status poll request
- found in the XML response of a "results" request
- computed as follows: URL = end point of the async service + '/' + job identifier +'/results/result'
- can optionally request to delete the job, to be gentle with other users that might need the disk space, or to avoid that any other user could get to these results (no protection is currently in place, all results are publicly visible by all users). Not deleting the job could be useful if the results are to be shared with other users, in which case the URL of the results (or just the jobID) could be passed to them.
Undeleted jobs remain available for a limited amount of time, set by default to 1 day,
unless the user has specifically set a destruction datetime, which can anyway not exceed an imposed server limit of 6 months .
Two TAP services are available, as described in the table here below.
|tap_obs||offers access to the tables related to science data (both raw and reduced) and ambient conditions of the La Silla, Paranal, and Chajnantor Observatory.|
|tap_cat||offers access to the scientific catalogs provided by the principal investigators of the ESO observing programmes (public survey catalogs, etc.). |
Asynchronous Query (using the Universal Worker Service (UWS))
Asynchronous queries are managed using the UWS(3) protocol, which allows queries with much larger result sets
to be efficiently managed by the service. In short, they are set up by the client, potentially negotiated with
regards to the amount of time they are allowed to run or be stored on the server, then run by the client. The client
then polls the server to discover when the query has run and where potential error or completed result sets are stored.
This is all managed via a 'job ID'. To run a UWS query, first submit your ADQL-based query. Given the RESTful
nature of UWS as implemented, you will be forwarded to a status page with a job number. Copy down this job
number - you will use it for the rest of the lifetime of your query. You can also query the UWS engine for a list of
currently existent jobs.
Asynchronous Query Management
Once you have a job number, you can enter it in the forms below to either renegotiate the deletion time of your results (if you expect it to take a long time to run, or will
be personally unable to check up on your results before they would be automatically deleted), to trigger the job to be run,
to poll your job for another status page giving the URL of its result set or error documentation,
and to delete your job when it is finished. Job deletion will bring you back to the job summary page.
First of all, choose the TAP service you submitted your job to:
The main URL for programmatic access to the TAP service is:
||This action can be taken only before running a job.
||This action can be taken only before running a job.
Controlling programmatically your asynchronous queries
Instead of using the web forms provided in the previous section,
you might prefer to control your jobs programmatically.
This section shows you how to achieve that via few examples. You do not need to necessarily use wget, the important thing is that you use http POST when modifying the job (i.e. modifying its status or its parameters), and that you use http GET when accessing read-only the job (e.g. to get to know its status, or to retrieve the results).
Full explanation of the programmatic interface is to be found in the Universal Worker Service Pattern Version 1.1 IVOA standard.
In the examples here below, it is assumed that you submitted an asynchronous query, and that the job identifier assigned to it was: 12345678.
Change it with the number provided in the initial answer you received when submitting your asynchronous query.
wget -O - 'http://archive.eso.org/tap_obs/async/12345678' --post-data 'PHASE=RUN'
wget -O - 'http://archive.eso.org/tap_obs/async/12345678'
wget -O - 'http://archive.eso.org/tap_obs/async/12345678' --post-data 'executionduration=120'
|SET DESTRUCTION TIME:||
wget -O - 'http://archive.eso.org/tap_obs/async/12345678' --post-data 'destruction=2018-09-01T00:00:00'
wget -O - 'http://archive.eso.org/tap_obs/async/12345678/results/result'
wget -O - 'http://archive.eso.org/tap_obs/async/12345678' --post-data 'PHASE=ABORT'
wget -O - 'http://archive.eso.org/tap_obs/async/12345678' --post-data 'ACTION=DELETE'
Equivalent curl command would look like this:
curl -Lw -X POST 'http://archive.eso.org/tap_obs/async/12345678' -d 'PHASE=RUN'
Last modification date of Manage your Asynchronous Queries: 2019-02-20
The following links may help to explain the basic query language and systems at work.
- ADQL - Astronomical Data Query Language.
- TAP - Table Access Protocol. This is the underlying standard
behind ObsTAP, or Observation Data Model / TAP, services. All queries built using the forms on this page are formatted for TAP.
The standard provides a basic query mechanism, a mechanism for schema-related configuration queries, and VOSI support.
- UWS - Universal Worker Service for asynchronous
web service communication. We have implemented this using REST protocols as explained in an IVOA note.
- VOSI - Virtual Observatory Standard Interface.
This defines a protocol for discovering basic uptime, capability, descriptive, and schema-related information about a service.
We have implemented the REST version as explained in the VOSI 1.0 documentation.
This can be used for SQL-like string queries to the service