view  

The 3taps Data Commons

The Polling API

The 3taps Polling API makes it possible for external systems to "poll" the Data Commons server to obtain a list of new and updated postings as they come in.

The Nature of Polling

All postings are ordered by the date and time at which they are inserted, updated or deleted, like this:

As time goes on, this forms a "stream" of newly-inserted, updated, and deleted postings.

When you issue a /poll request, you are interested in finding all the new postings which have been received (or updated, or deleted) since you last made a request -- that is, all the new entries in the posting stream:

Because a large number of postings may have come in since the last time you issued a polling request, you may not get all the postings at once. However, making multiple requests to the Polling API will allow you to eventually catch up.

Note that, while postings are ordered by date and time, a timestamp value is not accurate enough to use for looking for new postings. Timestamp values are only accurate to the nearest second, and it is highly likely that multiple postings come in within the same second. For this reason, the /poll API call makes use of the concept of an anchor. The anchor is a value that uniquely identifies a point in the posting stream. When you poll, you are looking for entries in the posting stream that are more recent than the position identified by the anchor:

By default, when you make a request to the Polling API without an anchor, the API will return no postings but an anchor value that represents the most recent entry in the posting stream. Thus, subsequent calls to the Polling API will return any new postings, updates and deletions which have come in since that time.

Note that an anchor is not a posting ID, or a timestamp, or any other value which can be interpreted by the system calling the API. The anchor should be an opaque data structure (usually represented as a string) that uniquely refers to a given position within the posting stream. Internally, the Polling API may choose to implement the anchor as a timestamp accurate to the nearest microsecond (though this may still not work if the postings are not inserted/updated in the database with microsecond accuracy), or it may choose to use an internal "update identifier" that points to a given position within the stream of inserted/updated/deleted postings. However it does it, the Polling API should obfuscate the anchor so that callers cannot interpret or use this value for anything other than future calls to the Polling API.

Finally, because you might want to populate an external database with a representative sample of postings, without having to wait for new postings to come in, the Polling API allows you to set the anchor to some point in the past. Using this anchor API call, you can obtain an anchor value to use when polling the database, retrieving all postings that have been inserted or updated since a given point in time.

Calling the 3taps Polling API

The Polling API is accessed via the following base URL:

http://polling.3taps.com

The following API calls are supported by the Polling API:

anchor

Calculate the anchor value to use for the most recent posting received since a given point in time. This allows you to "go back" and poll for postings that have been added, updated and deleted sometime in the past.

poll

Retrieve any postings which have been added, updated or deleted since the last time the poll call was made, or since a given anchor value retrieved via an anchor call.

Let's take a closer look at each of these API calls in turn.

The "Anchor" API Call

The anchor API call is intended to let users of the Data Commons poll for older postings. This is mainly useful for testing, or for situations where you want to download a dump of all the postings that have come in since a given point in time (for example, to pre-load a visualisation of the posting stream)

To start polling from a given point in time, you should make an HTTP GET request to the following URL:

http://polling.3taps.com/anchor

The following parameters should be supplied, usually in the form of a URL-encoded parameter:

auth_token (required)

The authentication token used to ensure that you are authorized to make this API call.

timestamp (required)

The desired date and time, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

Upon completion, the Polling API will send back a response of type application/json, and the body of the response will be a JSON-formatted object with some combination of the following fields:

success

This will be set to true if an anchor value could be successfully calculated.

anchor

The calculated anchor value, if one was calculated, in the form of a string.

error

A string describing what went wrong, if the anchor could not be calculated.

The "Poll" API Call

The poll API call retrieves a set of new postings from the database. This call should be made using an HTTP GET request to the following URL:

http://polling.3taps.com/poll

This API call accepts the following parameters:

auth_token (required)

The authentication token used to ensure that you are authorized to make this API call.

anchor (optional)

The anchor value to use for this polling request. Note that this parameter is optional -- if no anchor is supplied, the API call will return an empty list of postings, along with a new anchor value that can be used to retrieve new, updated and deleted postings that have come in since the last time this API call was made. This has the effect of polling "from now on".

source (optional)

The desired 3taps data source code. If this is supplied, only postings from the given data source will be included in the polling results. Note that this parameter can include logical operators, as described below.

category_group (optional)

The desired 3taps category group code. If this is supplied, only postings with the given category group will be included in the polling results. Note that this parameter can include logical operators, as described below.

category (optional)

The desired 3taps category code. If this is supplied, only postings with the given category will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.country (optional)

The desired 3taps country code. If this is supplied, only postings in the given country will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.state (optional)

The desired 3taps state code. If this is supplied, only postings in the given state will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.metro (optional)

The desired 3taps metro area code. If this is supplied, only postings in the given metro area will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.region (optional)

The desired 3taps region code. If this is supplied, only postings in the given region will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.county (optional)

The desired 3taps county code. If this is supplied, only postings in the given county will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.city (optional)

The desired 3taps city code. If this is supplied, only postings in the given city will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.locality (optional)

The desired 3taps locality code. If this is supplied, only postings in the given locality will be included in the polling results. Note that this parameter can include logical operators, as described below.

location.zipcode (optional)

The desired 3taps ZIP code. If this is supplied, only postings in the given ZIP code will be included in the polling results. Note that this parameter can include logical operators, as described below.

status (optional)

Only include postings with the given status value. The following status values are currently supported:

registered
for_sale
for_hire
for_rent
wanted
lost
stolen
found

Note that this parameter can include logical operators, as described below.

state (optional)

Only include postings in the given state. The following state values are currently supported:

available
unavailable
expired

Note that this parameter can include logical operators, as described below.

retvals (optional)

A string listing the fields which should be returned back to the caller. The various fields should be separated by commas. At present, the following fields can be included in this parameter:

id
account_id
source
category
category_group
location
external_id
external_url
heading
body
timestamp
timestamp_deleted
expires
language
price
currency
images
annotations
status
state
immortal
deleted
flagged_status

If no retvals parameter is provided, the following default will be used:

id source
category
location
external_id
external_url
heading
timestamp

Upon completion, the Polling API will send back a response of type application/json, where the body of the response will be a JSON-formatted object with some combination of the following fields:

success

A boolean indicating whether or not the polling attempt was successful.

error

If the polling attempt failed, this will be a string describing what went wrong.

anchor

This string will contain the anchor value to use the next time the poll API call is made. Using this anchor will ensure that only newly-inserted, updated or deleted postings will be included in the next poll request.

postings

An array containing the found postings, sorted by the date and time at which the postings were inserted, updated or deleted. Each entry in this array will be an object with all or some of the following fields, depending on the value of the retvals parameter, above:

id

The internal record ID used to uniquely identify this posting. Note that this will be a (possibly very large) integer.

account_id

A string identifying the user who submitted the posting.

source

The code for the source system where this posting originated from.

category

The 3taps category code for this posting.

category_group

The 3taps category group code for this posting.

location

An object with some or all of the following fields:

lat

The latitude of this posting, in decimal degrees.

long

The longitude of this posting, in decimal degrees.

accuracy

An integer indicating the accuracy of the supplied lat/long value.

country

The 3taps country code for this posting.

state

The 3taps state code for this posting.

metro

The 3taps metro area code for this posting.

region

The 3taps region code for this posting.

county

The 3taps county code for this posting.

city

The 3taps city code for this posting.

locality

The 3taps locality code for this posting.

zipcode

The 3taps ZIP code for this posting.

formatted_address

A string containing the address associated with this posting as a one-line string, if this information is available.

external_id

A string that uniquely identifies the posting in the source system.

external_url

A URL pointing to the original posting.

heading

A string containing the heading for this posting.

body

A string containing the body of this posting, in plain (unformatted) text.

timestamp

The date and time at which the posting was created, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

timestamp_deleted

The date and time at which the posting was deleted, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

expires

The date and time at which this posting should expire, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

language

The 2-character ISO 639-1 language code indicating which language the posting is in.

price

The price associated with this posting, if any.

currency

The 3-character ISO-4217 currency code indicating which currency the price is in.

images

An array of image objects representing the images associated with this posting. Each image object can have the following fields:

full

The URL pointing to the full-sized image for this posting.

full_width

The width, in pixels, for the full-sized image, if known.

full_height

The height, in pixels, for the full-sized image, if known.

thumbnail

The URL pointing to a thumbnail-sized image for this posting.

thumbnail_width

The width, in pixels, for the thumbnail-sized image, if known.

thumbnail_height

The height, in pixels, for the thumbnail-sized image, if known.

annotations

A object holding the various annotations to associate with this posting. Each field in this object maps the annotation name to its associated value, formatted as a string.

status

The posting's current status. This will be one of the following values:

registered
for_sale
for_hire
for_rent
wanted
lost
stolen
found

state

The state the posting is in. This will be one of the following values:

available
unavailable
expired

immortal

true if the posting is immortal (ie, never expires), false otherwise.

deleted

true if the posting has been deleted, false otherwise.

flagged_status

A number indicating the current "flagged" status of the posting. This will be one of the following values:

0 = The posting has never been flagged.
1 = The posting has been flagged by a user.
2 = The flagged status was overruled.

Notes:

  1. The maximum number of matching postings returned by a single call to the poll API endpoint depends on which fields you asked to be returned:

    • If retvals includes the body field, then a limit of 100 postings will be enforced.

    • If retvals consists only of the id field, then there is a limit of 10,000 postings per API call.

    • For any other combination of fields, a maximum of up to 1,000 postings will be returned.

    These limits keep the size of the data returned by the server down to a reasonable level, ensuring the server isn't overloaded with large requests.

    If more postings are required, you should make another poll request, using the returned anchor value, and keep going until you have either retrieved as many postings as you want, or there are no more postings to retrieve (in which case, the returned postings array will be empty).

    2. If there are no new postings, the Polling API will return the following results:

 {"success" : true,
  "postings": []
 }

It is not considered to be an error if there are no new postings.

Logical Operators

A number of parameters can make use of logical operators to allow for more sophisticated polling options. For these parameters, a single value can be supplied simply by providing the value itself, like this:

category=SBIK

However, special characters can be used to change the way in which the parameter is interpreted. Two special characters are currently supported: