view  

The 3taps Data Commons

The Posting API

The 3taps Posting API is responsible for receiving incoming postings from external data sources. These postings are then added to the 3taps posting database so they can be searched against, and are also sent to outgoing notification streams.

Postings can be received either as formatted posting objects, or as raw data. When postings are sent in as raw data, an appropriate format must be specified, and a matching parser will be used to extract the posting details from the raw data.

Accessing the 3taps Posting API

The Posting API is accessed via the following URL:

http://posting.3taps.com

The various API endpoints are all defined as relative URLs within this top-level URL.

Submitting Formatted Postings

Postings can be submitted in the form of posting objects, which are JSON-formatted objects representing a single posting. Each posting object should consist of some combination of the following fields:

source (required)

The 3taps code identifying the source system where the posting originated from.

category (required)

The 3taps category code to associate with this posting.

Note: Because each category code maps to one and only one category group, the data source does not need to supply or calculate the category group for each incoming posting. Instead, the category group will be calculated automatically by the Posting API when the posting is received.

location

An object representing this posting's location. The location object can have any of the following fields:

lat

The latitude of this posting, in decimal degrees.

long

The longitude of this posting, in decimal degrees.

accuracy

An integer indicating the accuracy of the supplied lat/long value.

bounds

An array defining the bounding box which completely encloses the feature identified by the supplied lat/long value. The array should have 4 entries:

(min_lat, max_lat, min_long, max_long)

country

The 3taps country code for this posting.

state

The 3taps state code for this posting.

metro

The 3taps metro area code for this posting.

region

The 3taps region code for this posting.

county

The 3taps county code for this posting.

city

The 3taps city code for this posting.

locality

The 3taps locality code for this posting.

zipcode

The 3taps ZIP code for this posting.

Note: If a posting comes in with only lat and long values, and optionally either an accuracy or a bounds value, the 3taps Posting API will ask the geolocator to calculate the exact location for the posting, automatically filling in the country, state, metro, region, city, locality and zipcode fields as appropriate.

external_id (required)

A string or number that uniquely identifies the posting in the source system.

external_url (required)

A URL pointing to the original posting.

heading (required)

A string containing the heading for this posting.

body

A string containing the body of this posting, in plain (unformatted) text.

html

The original HTML text for this posting, encoded using base64 character encoding to avoid problems with JSON representation of HTML text.

timestamp (required)

The date and time at which the posting was created, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

timestamp_deleted (optional)

The date and time at which the posting was deleted, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

expires

The date and time at which this posting should expire, as an integer number of seconds since the 1st of January 1970 ("unix time"), in UTC.

Note: If a posting comes in without an expires value, an appropriate value will be calculated to expire the posting in seven days from when it is added to the database.

language

The 2-character ISO 639-1 language code indicating which language the posting is in.

price

The price associated with this posting, if any. Note that this can be an integer or a floating-point value, as desired.

currency

The 3-character ISO-4217 currency code indicating which currency the price is in.

images

An array of image objects representing the images associated with this posting. Each image object can have the following fields:

full

The URL pointing to the full-sized image for this posting.

full_width

The width of the full-sized image, in pixels.

full_height

The height of the full-sized image, in pixels.

thumbnail

The URL pointing to a thumbnail-sized image for this posting.

thumbnail_width

The width of the thumbnail-sized image, in pixels.

thumbnail_height

The height of the thumbnail-sized image, in pixels.

annotations

A object holding the various annotations to associate with this posting. Each field in this object maps the annotation name to its associated value, which should always be a string. Note that each individual annotation value is limited to 1,000 characters.

status

A string describing the posting's status. The following values are currently supported:

registered
for_sale
for_hire
for_rent
wanted
lost
stolen
found

Note: If a posting comes in without a status value, the posting's status will be set to for_sale.

state

A string describing the current state the posting is in. The following values are currently supported:

available
unavailable
expired

Note: If a posting comes in without a state value, the posting's state will be set to available.

deleted

Set this to true if the posting is to be marked as deleted.

immortal

Set this to true if the posting is to be immortal (ie, never expire).

flagged_status

A number indicating the current "flagged" status of the posting. This should be one of the following values:

0 = The posting has never been flagged.
1 = The posting has been flagged by a user.
2 = The flagged status was overruled.

Note: If a posting comes in without a flagged_status value, the posting's flag status will be set to zero (ie, never flagged).

origin_ip_address

The IP address of the party that created the posting, if known.

transit_ip_address (required)

The IP address of the party which gathered and sent this posting to 3taps, system, if known.

Note that any field not marked as required is optional.

Submitting Raw Postings

Postings can also be submitted in the form of raw HTML, text, or binary data. When submitting raw postings, the caller must supply the following information:

Posting

A string containing the contents of the raw posting. The posting string can be in plain text, UTF-8 encoding, HTML, or even binary data as required.

Format

A string telling the Posting API how to go about parsing the supplied data. The following raw posting formats are currently supported by the 3taps Posting API:

CRAIG_HTML

A posting from Craigslist scraped as raw HTML

Note that the list of raw posting formats will be updated as more formats are supported.

External URL

A URL linking back to the original posting in the external system where it was created.

Caller Authentication

The Posting API requires the caller to supply a valid authentication token. Note that only suitably-authorized systems will be allowed to access the 3taps Posting API.

The Posting Process

When a suitably-authorized system submits one or more pre-formatted postings to the 3taps Posting API, the API must first vet the incoming data to make sure that it is acceptable. There are four steps involved in vetting a posting:

  1. Ensuring that each supplied field is of the correct type. For example, the API checks that each boolean field contains a true or false value, that the location field is an object, and that the timestamp field contains an integer timestamp value.

  2. Making sure that all the fields marked as required have been supplied.

  3. Ensuring that any supplied fields which accept a 3taps code (ie, the source, the category, and the various location fields) contain a known value, as defined by the 3taps Reference API.

  4. Checking that the language and currency codes are valid ISO values, if they have been supplied.

  5. Calculating the category group for the posting, based on the supplied category code.

If there is anything wrong with the posting, an appropriate error response will be calculated for the posting. Otherwise, if the posting is acceptable, the error response will be set to a null value.

Any postings which pass the vetting process will be inserted into the database. Note that the database can contain duplicate copies of a posting. If a posting is submitted multiple times, for example with updated values or a change in status, each submitted copy of the posting will be added to the database in turn. It is up to client software to filter out the duplicates (based on the source and external_id field values), and only use the most recent version.

For raw postings, the Posting API will first attempt to parse the raw data, based on the supplied format value. If the posting data could not be parsed for some reason, an appropriate error response will be calculated for the posting, just as if there was something wrong with a pre-formatted posting. Once the posting has been parsed, it will be vetted in the way described above, and then inserted into the database, exactly as if the posting had already been formatted.

Calling the Posting API

As well as submitting either formatted or raw data, the caller can send in either a single posting, or multiple postings at once. Thus, there are four different types of data which can be sent to the Posting API:

Let's take a closer look at how to submit each of these types of data.

Submitting a Single Pre-Formatted Posting

To send a single pre-formatted posting to the 3taps Posting API, make an HTTP POST call to the following URL:

 http://posting.3taps.com/post_single

The request should have a Content-Type of application/json, and the body of the request should be a JSON-encoded object with the following fields:

auth_token

The authentication token used to authenticate the caller.

posting

The posting object to be submitted.

If the API request is missing some required information, or there is something else wrong with the request (eg, the caller used the wrong Content-Type), the Posting API will send back a response with a status code of 400 (bad request), and the body of the response will be a string describing what was wrong with the request.

If the supplied auth_token is invalid or missing, the Posting API will send back a response with a status code of 401 (Unauthorized), and the body of the response will be a string describing why the caller's auth_token was rejected.

If the request was successful, the API will return an HTTP status code of 200 (OK), the response will have a Content-Type of application/json, and the body of the response will be a JSON-formatted object with the following fields:

error_response

The error response for the submitted posting, or null if the posting was processed successfully.

wait_for

An integer value telling the caller how long to wait (in seconds) before sending in another request. This is used to regulate the speed at which postings are sent in, so that the Posting API doesn't get overloaded.

Submitting Multiple Pre-Formatted Postings

To send multiple pre-formatted postings to the 3taps Posting API at once, make an HTTP POST call to the following URL:

 http://posting.3taps.com/post_multi

The request should have a Content-Type of application/json, and the body of the request should be a JSON-encoded object with the following fields:

auth_token

The authentication token used to authenticate the caller.

postings

An array of posting objects to be submitted.

If the API request is missing some required information, or there is something else wrong with the request (eg, the caller used the wrong Content-Type), the Posting API will send back a response with a status code of 400 (bad request), and the body of the response will be a string describing what was wrong with the request.

If the supplied auth_token is invalid or missing, the Posting API will send back a response with a status code of 401 (Unauthorized), and the body of the response will be a string describing why the caller's auth_token was rejected.

If the request was successful, the API will return an HTTP status code of 200 (OK), the response will have a Content-Type of application/json, and the body of the response will be a JSON-formatted object with the following fields:

error_responses

An array containing the error responses for each of the submitted postings. The error responses will be in the same order as the original postings, so the index into the postings array can be used as the index into the error_responses array to get the error response for a particular posting. Note that the error response for a posting will be null if the posting was processed successfully.

wait_for

An integer value telling the caller how long to wait (in seconds) before sending in another request. This is used to regulate the speed at which postings are sent in, so that the Posting API doesn't get overloaded.

Submitting a Single Raw Posting

To send a single unformatted (raw) posting to the 3taps Posting API, make an HTTP POST call to the following URL:

 http://posting.3taps.com/post_raw_single

The request should have a Content-Type of application/json, and the body of the request should be a JSON-encoded object with the following fields:

auth_token

The authentication token used to authenticate the caller.

format

A string indicating which format of the raw posting. This is used to select an appropriate parser for processing the raw posting. The list of supported formats is given in the section on raw postings, above.

posting

The raw posting to process, as a string. Note that if the string contains non-ASCII characters, it should encoded in UTF-8 format.

external_url

The URL of the original posting in the external system.

If the API request is missing some required information, or there is something else wrong with the request (eg, the caller used the wrong Content-Type), the Posting API will send back a response with a status code of 400 (bad request), and the body of the response will be a string describing what was wrong with the request.

If the supplied auth_token is invalid or missing, the Posting API will send back a response with a status code of 401 (Unauthorized), and the body of the response will be a string describing why the caller's auth_token was rejected.

If the supplied format is missing or unknown, the Posting API will send back a response with a status code of 400 (bad request), and the body of the response will be a string describing why the caller's format value was rejected.

If the request was successful, the API will return an HTTP status code of 200 (OK), the response will have a Content-Type of application/json, and the body of the response will be a JSON-formatted object with the following fields:

error_response

The error response for the submitted posting, or null if the posting was processed successfully.

wait_for

An integer value telling the caller how long to wait (in seconds) before sending in another request. This is used to regulate the speed at which postings are sent in, so that the Posting API doesn't get overloaded.

Submitting Multiple Raw Postings

To send multiple unformatted (raw) posting to the 3taps Posting API, make an HTTP POST call to the following URL:

 http://posting.3taps.com/post_raw_multi

The request should have a Content-Type of application/json, and the body of the request should be a JSON-encoded object with the following fields:

auth_token

The authentication token used to authenticate the caller.

format

A string indicating which format of the raw postings. This is used to select an appropriate parser for processing the raw postings. The list of supported formats is given in the section on raw postings, above.

postings

An array of raw postings to process, where each array entry is a JSON object with the following fields:

posting

A string containing the raw posting. Note that if the string contains non-ASCII characters, it should encoded in UTF-8 format.

external_url

The URL of the original posting in the external system.

If the API request is missing some required information, or there is something else wrong with the request (eg, the caller used the wrong Content-Type), the Posting API will send back a response with a status code of 400 (bad request), and the body of the response will be a string describing what was wrong with the request.

If the supplied auth_token is invalid or missing, the Posting API will send back a response with a status code of 401 (Unauthorized), and the body of the response will be a string describing why the caller's auth_token was rejected.

If the supplied format is missing or unknown, the Posting API will send back a response with a status code of 400 (bad request), and the body of the response will be a string describing why the caller's format value was rejected.

If the request was successful, the API will return an HTTP status code of 200 (OK), the response will have a Content-Type of application/json, and the body of the response will be a JSON-formatted object with the following fields:

error_responses

An array containing the error responses for each of the submitted postings. The error responses will be in the same order as the original postings, so the index into the postings array can be used as the index into the error_responses array to get the error response for a particular posting. Note that the error response for a posting will be null if the posting was processed successfully.

wait_for

An integer value telling the caller how long to wait (in seconds) before sending in another request. This is used to regulate the speed at which postings are sent in, so that the Posting API doesn't get overloaded.