view  

The 3taps Data Commons

The Stats API

The 3taps Stats API is responsible for collecting statistics from other parts of the system, and then using this information to produce reports, dashboards and alerts.

Accessing the 3taps Stats API

The 3taps Stats API can be accessed at the following URL:

http://stats.3taps.com/

Note that, while we call this the "Stats API", it is not only an API. In addition to API calls, the Stats API provides a web interface which allows suitably-authorized people to access the various stats dashboards and reports.

Statistic Groups

While a large number of statistics can be collected, these statistics are arranged into groups depending on the activity that we are collecting statistics for:

A single call to the 3taps Stats API records an entire group of statistics all at once.

The following statistic groups are currently supported by the 3taps Stats API:

reverse_geocode

A request was made to the Geolocator API's reverse-geocoder.

sources

A request was made to the Reference API for a list of data sources.

categories

A request was made to the Reference API for a list of categories.

locations

A request was made to the Reference API for a list of locations.

location_lookup

A request was made to the Reference API to look up a location.

signup

A user was signed up using the 3taps Identity API.

authenticate

A user authenticated themselves with the 3taps Identity API.

postings_processed

A number of postings were received and processed by the 3taps Posting API.

postings_geolocator_request

A number of postings were required to be geolocated by the 3taps Posting API.

More statistic groups will be added as they are required.

Individual Statistics

Each collected statistic has the following information associated with it:

timestamp

The date and time at which this statistic was collected.

group

A string indicating the group that this statistic belongs to.

name

The name of the statistic being collected.

code

Each statistic can optionally have a code associated with it. The meaning of this code depends upon the statistic name and type. For example, it may be a 3taps location code, or a status value. Note that only certain statistics have codes associated with them, as described below.

value

The value associated with this statistic. Note that the collected statistic values are always integers.

Each group of statistics can have a given set of individual statistics within it. The exact set of statistics, and how their values should be interpreted, depends upon the group, as described below:

reverse_geocode

The reverse_geocode group contains the following statistics:

num_coords

The number of lat/long coordinates which were processed, as a integer.

tot_time

The total amount of time it took to process those coordinates, as an integer number of milliseconds.

avg_time

The average amount of time it took to process a coordinate, as an integer number of milliseconds.

max_time

The maximum amount of time it took to process a coordinate, as an integer number of milliseconds.

sources

The sources group contains the following statistics:

num_sources

The number of data sources which were returned to the caller.

time_taken

The total amount of time it took to process the request, as an integer number of milliseconds.

categories

The categories group contains the following statistics:

num_categories

The number of categories which were returned to the caller.

time_taken

The total amount of time it took to process the request, as an integer number of milliseconds.

locations

The locations group contains the following statistics:

num_locations

The number of locations which were returned to the caller.

time_taken

The total amount of time it took to process the request, as an integer number of milliseconds.

location_lookup

The location_lookup group contains the following statistic:

time_taken

The total amount of time it took to process the request, as an integer number of milliseconds.

signup

The signup group contains the following statistic:

time_taken

The total amount of time it took to process the request, as an integer number of milliseconds.

authenticate

The authenticate group contains the following statistic:

time_taken

The total amount of time it took to process the request, as an integer number of milliseconds.

postings_processed

The postings_processed group contains the following statistics:

num_received

The number of postings which were received by the API in this time period.

num_rejected

The number of received postings which were rejected due to an error.

tot_time

The total amount of time it took to process these postings, as an integer number of milliseconds.

avg_time

The average amount of time it took to process an individual posting.

max_time

The maximum amount of time it took to process a single posting.

avg_validation_time

The average amount of time it took to validate (ie, check that the posting data was acceptable) each individual posting.

max_validation_time

The maximum amount of time it took to validate a posting in this time period.

avg_response_time

The average time taken to respond to an incoming API call to the Posting API.

max_response_time

The maximum amount of time it took to respond to an incoming API call to the Posting API.

num_received_by_source

The number of postings which were received by the API, summarized by 3taps source code.

num_received_by_category

The number of postings which were received by the API, summarized by 3taps category code.

num_received_by_location

The number of postings which were received by the API, summarized by 3taps location code.

num_received_by_remote_ip

The number of postings which were received by the API, summarized by remote IP address.

num_received_by_auth_token

The number of postings which were received by the API, summarized by authentication token.

num_rejected_by_source

The number of postings which were rejected due to an error, summarized by 3taps source code.

num_rejected_by_category

The number of postings which were rejected due to an error, summarized by 3taps category code.

num_rejected_by_location

The number of postings which were rejected due to an error, summarized by 3taps location code.

num_rejected_by_remote_ip

The number of postings which were rejected due to an error, summarized by remote IP address.

num_rejected_by_auth_token

The number of postings which were rejected due to an error, summarized by authentication token.

avg_time_by_source

The average time taken to process an incoming posting, summarized by 3taps source code.

max_time_by_source

The maximum time taken to process an incoming posting, summarized by 3taps source code.

min_time_by_source

The minimum time taken to process an incoming posting, summarized by 3taps source code.

avg_time_by_auth_token

The average amount of time it took to process an individual posting, summarized by authentication token.

max_time_by_auth_token

The maximum time taken to process an individual posting, summarized by authentication token.

avg_polling_api_latency

The average time between when a posting is received by the 3taps Posting API and that posting becoming available in the Polling API.

max_polling_api_latency

The maximum time between when a posting is received by the 3taps Posting API and that posting becoming available in the Polling API.

avg_search_api_latency

The average time between when a posting is received by the 3taps Posting API and that posting becoming available in the Search API.

max_search_api_latency

The maximum time between when a posting is received by the 3taps Posting API and that posting becoming available in the Search API.

avg_geolocator_delay

The average time time taken to geolocate a posting in this time period.

max_geolocator_delay

The maximum time taken to geolocate a posting in this time period.

postings_geolocator_request

The postings_geolocator_request group contains the following statistics:

num_postings

The total number of postings which were received.

num_already_geolocated

The number of postings which were received with location data and so didn't need to be geolocated.

num_sent_to_geolocator

The number of postings which had to be sent to the Geolocator API for reverse geocoding.

Recording Statistics

To record a group of statistics, make an HTTP GET or POST request to the following URL:

http://stats.3taps.com/add

Parameters can be supplied to this API call using either URL-encoding or form-encoding. The parameters must include a group value, which is the name of the group to use for these statistics.

For backward compatibility, the group can also be supplied in a parameter named type, though this will be removed in the near future.

The individual statistics should be supplied as additional parameters to the API call. For example:

http://stats.3taps.com/add?group=locations&num_locations=412&time_taken=63

While most of the statistics can be supplied as plain text, more complex statistics need to be encoded as a JSON string. This applies to statistics that contain objects mapping codes to values. For example, the postings_processed statistic might include the following parameter value:

num_by_source={"HMNGS":17, "EBAYC":43}

If necessary, the parameter can also be URL-encoded to avoid issues with submitting strings with punctuation via HTTP, eg:

num_by_source=%7B%22HMNGS%22%3A17%2C%20%22EBAYC%22%3A43%7D

If the group of statistics were successfully recorded, the Stats API will return an HTTP status code of 200 (OK), and the body of the response consist of the string "OK". If there is something wrong with the request, the Stats API will return an HTTP status code of 400 (bad request), and the body of the response will contain a textual description of what went wrong.

The Admin Interface

To access the admin interface for the stats system, go to:

http://stats.3taps.com

You will be asked to log in, and then presented with various options for viewing the collected statistics.