This API provides access to information about the motus data processing server, jobs, and errors. It also permits some server-side tasks to be executed (job retry, server restart, processing of new uploaded file, ...)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip
in which case the reply
will be gzip-compressed, rather then bzip2-compressed (see below)json
, which is a JSON-encoded object.json
are the parameters listed for each API entrypoint below.authToken
value, which can be obtained by a call
to authenticate_user
; alternatively, it can be the value (not URL-encoded) of a cookie
named auth_tkt
which is generated by the server's main login page
according to the Apache mod-auth-tkt module. Code for generating the cookie can
be seen in R
or in PHP"par":X
and "par":[X]
if X
is a double, integer,
boolean or stringContent-Type = application/json
Content-Encoding = bzip2
. To support browsers and other
contexts without native bzip2 decompression, if the request had a
header called Accept-Encoding
that includes the string "gzip", then the
reply is gzip-compressed, with header Content-Encoding: gzip
.error
in the reply; other
fields might be present, giving additional information. If no field error
is present, the request succeeded.paging
call to the same API
would return 0 rows. The maximum number of rows can be obtained by
calling api_info
Examples are given for each call using the command-line
client curl with quoting
appropriate for the Bash shell. These examples return the raw
bzip2-compressed data. To view the response, redirect the output of
curl into a file and use 7zip to decompress it
(for example), or add | bunzip2 -cd
to the end of the command in
Bash.
The server is at https://sgdata.motus.org and the URL prefix is "/status".
status_api_info (authToken)
e.g.
curl https://sgdata.motus.org/status2/status_api_info
return a list with these items:
maxRows: integer, maximum number of rows returned by a query
uploadPath: string, path to upload folder relative to top-level of NAS storage;
the path
of an uploaded file passed in a call to process_new_upload
API must be
relative to uploadPath
; e.g. if uploadPath
is sgdata/sgm/uploads
, and if the
process_new_upload
API is called with path
=partial/232_2017-08-15T11-12-13.123_newfile.zip
,
the uploaded file must be at NAS:/sgdata/sgm/uploads/partial/232_2017-08-15T11-12-13.123_newfile.zip
authenticate_user (user, password)
- user: username
- password: password (in cleartext)
e.g.
curl --data-urlencode json='{"user":"someone","password":"bigsecret"}' https://sgdata.motus.org/status2/authenticate_user
or
The authToken
returned by this API must be included in most other
API calls (for an alternative, see 2.)
Instead of obtaining the token from this API entry, you can use
the non-URL-encoded value of the cookie called auth_tkt
, obtained by
logging in via
the main login page. The
non-URL-encoded value of the auth_tkt
cookie can be used as
authToken
, which avoids having to call authenticate_user
if the
user has already logged in. To use auth_tkt
, the IP address from
which the login occurred must be passed as a dotted-quad string in the
header field X-Forwarded-For
to any of the requests in this API,
along with passing the cookie value in the JSON field authToken
.
The correct X-Forwarded-For
header is generated automatically if the status
API is accessed from
client-side javascript. However, if the URL is accessed from a PHP
script on a server, the automatically-generated IP address for the
header will be that of the server running the PHP script, rather than
of the ultimate client, which is wrong. So to call this API from
a server-side script, you need to specify the X-Forwarded-For
header explicitly, in which case no value will be generated automatically.
Here's an example in php:
/// @param $url: full URL of API entry
/// @param $par: associative array of API parameters
function post ($url, $par) {
$ch = curl_init($url);
$par['authToken'] = $_COOKIE['auth_tkt'];
$json = 'json=' . urlencode(json_encode($par));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/x-www-form-urlencoded',
'X-Forwarded-For: ' . $_SERVER['REMOTE_ADDR']));
curl_setopt($ch, CURLOPT_POSTFIELDS, $json);
$res = curl_exec($ch);
curl_close($ch);
return(json_decode(bzdecompress($res), true));
}
Authorization is by project: if a user has permission for a project, then that user can see:
status of all jobs submitted for that project
If an API call does not find any information for which the user is authorized, it will return a json object of the usual structure, except that column arrays will have length zero. This represents an R data.frame with the correct column names but zero rows.
The API doesn't currently provide a way to tell whether there are additional data which would be returned for a given call if the user had authorization for more projects.
list_jobs (projectID, select, order, options, authToken)
- projectID: integer; motus projectID (s); jobs must belong to the given project(s)
- select: object with fields for selecting which jobs to list. A job must match all
fields provided in order to be included in the list. Fields are:
- userID: integer; motus user ID
- stump: ID of top level job; select only from that job and its descendents; for increased
flexibility, if the provided stump is not the ID of a top-level job, then that job's
stump is used instead.
- jobID: integer array; to select only those jobs specified
- type: string array; the job type(s)
- done: integer; 0: job not yet complete; 1: job completed successfully; -1: job had error
- log: string; job whose log matches the string `log`, which can include globbing
characters ('*' and '.')
- serno: string; serial number; jobs that processed data from this receiver; any top-level
job at least one of whose subjobs was for that receiver are included. (An upload job
might have data from multiple receivers)
- order: object with fields for ordering and paging the selected jobs:
- sortBy: string scalar; null, or the sort key, i.e. one of these string constants:
- "ctime": job creation time
- "mtime": last job activity time
- "id": job ID number
- "type": job type
- "motusProjectID": motus project ID
- "motusUserID": motus user ID
If `sortBy` is not specified, it is set to `"mtime"`. To allow paging, an implici "id" is
added to sortBy if it is not already there
- sortDesc: optional; logicl scalar; if `true`, sorting is in descending order by key (and id within
key); otherwise, in ascending order by key (and id within key)
- lastKey: optional; vector giving "last" obtained value of the field specified in `sortBy`; an optional
second element gives the "last" obtained value of the "id", if `sortBy` is not "id". This is
used for paging. If not specified, returns the first page according to `sortBy` criteria.
- forwardFromKey: boolean; default: true. If `lastKey` is specified and this field is true,
return a page of items past the key in the forward (sort) direction. Otherwise, if this field is false,
return a page of items past the key in the backward (reverse sort) direction.
- options: object with fields giving options:
- includeUnknownProjects: include jobs with no associated motus project; typically for
jobs initiated by staff or otherwise not having a useful concept of project
- includeSubjobs: boolean: include jobs which are not top-level jobs?; default `false`
- errorOnly: boolean: if true, only show top jobs for which a subjob had an error.
- excludeSync: boolean: if true, omit `syncReceiver` jobs (there will be many of these)
- full: if `true`, then full details for the job (typically its parameters, log, summary, and list of
product files) are returned in a JSON-formatted column called `data`
- countOnly: boolean; if `true`, return only a count of jobs for the given projectID and/or userID,
as an object with field `count`, and value an integer array of length one. e.g. `{count:[123]}`
- limit: integer; if present, maximum number of records to return. This is ignored if
- it is larger than the maximum result size specified when the server was launched (default: 20)
** or **
- `select.stump` is specified, in which case **all** subjobs are returned. This is because the API
does not currently provide any way to paginate subjobs.
e.g.
curl --data-urlencode json='{"select":{"userID":232},"order":{"sortBy":"id","lastKey":[5000]},"authToken":"XXX"}' https://sgdata.motus.org/status2/list_jobs
return a list of jobs; unless includeSubjobs is true, return only top-level jobs. A top level job has one of these types:
uploadFiles
will typically generate one or more
tag finder jobs of types LtFindtags
or SGfindtags
, a data summary job of type plotData
,
and a data merging job of type exportData
.Only jobs to which the user has permission are returned.
if countOnly
is false
, fields in the returned object are these arrays:
id
: integer array; job idspid
: integer array; parent job idsstump
: integer array; top-level job idsctime
: double array; creation times (unix timestamp; seconds since 1 Jan 1970 GMT)mtime
: double array; modification time (unix timestamp; seconds since 1 Jan 1970 GMT)type
: string array; type of job; e.g. "uploadFile"done
: integer array; 0 if not yet run; +1 if successful; < 0 if errorqueue
: integer array; number of queue job is in; 0 means waitingpath
: string array; file system path to job folder, if anymotusUserID
: integer array; motus user ID of person who submitted jobmotusProjectID
: integer array; project ID for results of jobdata
: (if full
was true
in the request) string array; json-encoded job parameters, log, summary, productssjDone
: (if includeSubjobs
is not true
in the request) integer array; minimum done
value for all subjobs
of the top-level job, including that of the top-level job itself. So: 0, if all subjobs are either not yet done
or completed without error; < 0 if at least one subjob had an error; +1 if all subjobs completed successfullyprocess_new_upload (userID, projectID, path, ts, email, authToken)
- userID: integer scalar; motus user ID (who uploaded the file)
- projectID; integer scalar; motus projectID (what project should own the products; as chosen by upload user)
- path; string scalar; path to the new file on the NAS, relative to the value of `uploadPath` returned by the `status_api_info` API.
e.g. if `path` is given as 'partial/123_2017-10-20T11-12-33_upload.zip`, then
on linux, we'll expect to find the file at `nfs://174.140.177.35:/volume1/sgdata/sgm/uploads/partial/123_2017-10-20T11-12-33_upload.zip`
which will have actual path /sgm/uploads/partial/123_2017-10-20T11-12-33_upload.zip, given the current NAS mountpoints.
path` must not include any '..' component; i.e. ascent up the file tree is not permitted, to prevent malicious use
from leaking system information.
Both forward (`/`) and reverse (`\`) slashes are interpreted as folder delimiters.
- ts; either double or string scalar; time at which file upload completed. If double, interpreted as seconds since 1 Jan 1970, GMT;
If string, interpreted as a datetime compatible with `R:lubridate::ymd_hms()`. If not supplied, the current
time is used.
- email; optional string scalar; if present, an email is sent to that address when processing of the uploaded file is complete.
e.g.
curl --data-urlencode json='{"userID":232,"projectID":57,"path":"232/232_2017-10-20T11-12-33_myupload.zip","ts":1508497953,"authToken":"XXX"}' https://sgdata.motus.org/status2/process_new_upload
return: an object with these items:
jobID
: the integer motus ID for the new jobuploadID
: the integer motus ID for the uploadnewPath
: character; the new path to the file, relative to the NAS root.side effect: the file will have been moved to newPath
, and its ownership
will be changed to sg:sg
with permissions rw-rw-r--
If the file has already been uploaded, then error
is set to a
message, and the field details
contains the details of the
existing copy of the file, with subfields jobID
, uploadID
,
motusUserID
, motusProjectID
, filename
, sha1
, ts
corresponding to the file's treatment when it was first uploaded.
Note: The check for whether a file has been uploaded is by contents, not name, so renaming a file and attempting to upload it again will result in the same error.
Recommended format for file paths:
We want file paths:
- encoded in UTF-8
- no :
(colon) or "
(double quote) characters in path
- path separator is /
(forward slash)
- to provide intrinsic collision avoidance
- identify the user who uploaded them
So, a file path should look like this:
XXX/123_2017-10-20T15-21-35.123_user_name_for_file.zip
where:
XXX/
component is an optional prefix path, in case the upload tree isn't rooted at uploadPath
(see above).
It must not contain any /../
components.123
is the userID of the upload user; it is followed by an underscore (_
)2017-10-20T15-21-35.123
is the timetamp of the upload; in the filename, it is followed by an underscore (_
)user_name_for_file.zip
is the full user-supplied filename, in UTF-8; it must not include any forward slashes
(/
), double-quotes ("
), or colons (:
).Debugging feature: if path
begins with testing/
, then no processing will be done, and if otherwise
successful, this call will return with an error whose message will confirm the file was received but that no processing
will take place.
list_receiver_files (serno, day, authToken)
- serno: string scalar; receiver serial number
- day: string scalar; day, formatted as "YYYY-MM-DD" (treated as GMT); if missing or null,
this API returns a list of days, rather than a list of files
e.g.
curl --data-urlencode json='{"serno":"SG-1513BBBK0291","day":"2017-04-04","authToken":"XXX"}' https://sgdata.motus.org/status2/list_receiver_files
serno
: receiver serial numberday
, along with a field called fileDetails
with these array items:fileID
: integer; ID of file (relative to receiver), if it has been processed; null otherwisename
: character; name of filebootnum
: integer; boot count, uncorrected, if it has been processed; null otherwisemonoBN
: integer; corrected boot count, if it has been processed; null otherwisecontentsSize
: integer; uncompressed file size in bytes, if it has been processed; null otherwisefileSize
: integer; size of file on disk, if present; null otherwisecomplete
: boolean; true if we have the complete .gz version of the filejobID
: the integer motus ID for the job in which this file was most recently updated
Sort order for this item is ascending by fileID
.fileCounts
with these array items:day
: character; day, formatted as 'YYYY-MM-DD'countDB
: integer; number of files for this receiver from given day known to receiver database
For Lotek receivers, this is always 1, because for a given day, a file either exists with that
day in it, or not.countFS
: integer; number of files for this receiver from given day stored in file system
Sort order for this item is descending by day
. For a Lotek receiver, countFS
is not meaningful
because files span multiple days, so it is set to the same value as countDB
.get_receiver_file (serno, fileID, authToken)
return: a text file; for this API call, the reply has headers that a browser will interpret as if a file were being downloaded.
serno
: receiver serial numberfileID
: integer file ID for that receiverNote: to minimize server resources used for this API call, files are transmitted as
they appear in the file repository (i.e. either uncompressed as .txt
or
compressed as .txt.gz
).
Additionally, the Content-Encoding
header for a .gz file is given as gzip
, which
means a web browser will automatically decompress the file before saving it.
Either way, the user ends up with an uncompressed text file whose name ends in .txt
,
not in .txt.gz
get_receiver_info (serno, authToken)
- serno: string scalar; receiver serial number
e.g.
curl --data-urlencode json='{"serno":"SG-1513BBBK0291","authToken":"XXX"}' https://sgdata.motus.org/status2/get_receiver_info
serno
: receiver serial numberdeviceID
: motus device IDreceiverType
: SENSORGNOME
or LOTEKXXX
, where XXX is a model namedeployments
: an object with these array items:deployID
: integer; deployment ID (internal to motus, but links to antDeps)projectID
: integer; ID of project that deployed the receiverreceiverType
: string; "SENSORGNOME" or "LOTEK"status
: string; deployment statusname
: string; typically a site namefixtureType
: string; what is the receiver mounted on?latitude
: double; (initial) location, degrees Northlongitude
: double; (initial) location, degrees Eastelevation
: double; (initial) location, metres ASLisMobile
: integer; non-zero means a mobile deploymenttsStart
: double; timestamp of deployment starttsEnd
: double; timestamp of deployment end, or NA if ongoing
Sort order for this item is descending by tsStart
.products
: string array; list of URLs to summary products for this receiver.
Only those products assigned to projects the user has permissions for are included.get_job_stackdump (jobID, authToken) - administrative users only
- jobID: integer; ID of job with an error (i.e. `done` < 0)
The .rds file contains an object of R class dump.frames
. It is a list whose names are the calls
and whose elements are the environments of the calls (i.e. contain the variables 'defined within' each level of
function call.
Because stack dump files might leak passwords or other credentials, this API call only works for administrators.
retry_job (jobID, message, authToken) - administrative users only
- jobID: integer; ID of job with an error, or one of whose siblings has an error
- message: string; optional message to add to job log, indicating e.g. reason for retry
jobID
\Otherwise, an error message is returned in item error
.
get_upload_info (uploadID, sha1, listContents, authToken)
- uploadID: integer (optional); ID of uploaded file
- sha1: string, hex digits (optional); sha1 hash of file contents
- listContents: boolean (optional); return summary of file contents?
default: true.
Exactly one of uploadID
or sha1
must be given, and is used to look-up the file.
Specifying sha1
is meant to let a client check whether a file has already been uploaded (by
sha1 hash of its contents) before wasting a user's time and bandwidth.
USERID_TIMESTAMP_
prefixlistContents
is true. Otherwise, "".Otherwise, an error message is returned in item error
.
serno_collision_rules (action, id, serno, cond, suffix, authToken) - administrative users only
- action: string (required); one of "get", "put", or "delete". Presence and
use of other parameters depends on this value.
- id: integer scalar or array; id of rule in database
- serno: string scalar or array; serial number(s) of receiver(s) to which the
rule applies. This must be the bare serial number, **without** the
disambiguation suffix.
- cond: string scalar; an R expression using certain terms extracted from
a file or filename. When this condition evaluates to TRUE for a file, the
file is deemed to have come from the receiver given by concatenating
`serno` and `suffix`. Only the first matching rule is used when trying
to decide which receiver in a collision is the source of the file.
- suffix: string scalar; '_N', where N is a small integer. When two (
or more?!) receivers have the same serial number, the suffix `_1` is
added to the second one, `_2` to the third one, and so on. The first
receiver with a given serial number receives no suffix. (The ordering
implied by `first`, `second` etc. is given implicitly by the rules).
Semantics:
action == "get"
: fetch all rules whose id
was specified or whose serno
was
specified, or all rules if neither was specified. The return value is an
object with these named fields:
- id; integer; rule ID
- serno; string; bare receiver serial number
- cond; string; R expression
- suffix; string; '' or '_1', or '_2', ...
action == "put"
: insert new rules. The user must specify serno
, cond
, and
suffix
, and each must be a scalar, or all must be arrays of the same length.
The id
field will be automatically assigned. The return value is as
for action == "get"
, but includes only those rules just
created. NO ATTEMPT IS MADE TO ENSURE THE SANITY OF THE RULES
action == "delete"
: delete any rules for which the user specified an id
,
or for which the user specified a serno
. The return value is as
for action == "get"
, but includes only those rules just deleted.
get_param_overrides(id, projectID, serno, progName, authToken) - administrative users only
- id: optional; integer array of param override IDs
- projectID: optional; integer array of motus project IDs
- serno: optional; string array of receiver serial numbers
- progName: optional; string array of program names to which parameter overrides apply
return the set of all parameter overrides matching all specified criteria. If no criteria are supplied, then all overrides are returned. The returned object has these array items:
- id: integer; IDs of parameter overrides
- projectID: integer; motus project IDs (each can be null)
- serno: character; device serial numbers (each can be null)
- tsStart: double; starting timestamps (each can be null; seconds since 1 Jan 1970 GMT)
- tsEnd: double; ending timestamps (each can be null; seconds since 1 Jan 1970 GMT)
- monoBNlow: integer; starting boot session numbers (each can be null; used for SGs only)
- monoBNhigh: integer; ending boot session numbers (each can be null; used for SGs only)
- progName: character; names of programs to which override applies; typically 'find_tags_motus'
- paramName: character; names of parameters (e.g. 'default_freq')
- paramVal: double; values for parameters (each can be null if parameter is just a flag)
- why: character; human-readable reason for each override
delete_param_overrides(id, authToken) - administrative users only
- id: integer array of param override IDs
delete the parameter overrides whose IDs are in id
, returning a boolean array of the
same length indicating which IDs are now not in the database (i.e. were deleted
or were already not present).
add_param_override(projectID, serno, tsStart, tsEnd, monoBNlow, monoBNhigh, progName, paramName, paramVal, why, authToken)
- projectID: integer; motus project ID (can be null)
- serno: character; device serial number (can be null)
Exactly one of `serno` or `projectID` must not be null.
- tsStart: double; starting timestamp (can be null; seconds since 1 Jan 1970 GMT)
- tsEnd: double; ending timestamp (can be null; seconds since 1 Jan 1970 GMT)
- monoBNlow: integer; starting boot session number (can be null; used for SGs only)
- monoBNhigh: integer; ending boot session number (can be null; used for SGs only)
- progName: character; name of program to which override applies; typically 'find_tags_motus'
- paramName: character; name of parameter (e.g. 'default_freq')
- paramVal: double; value for parameter (can be null if parameter is just a flag)
- why: character; human-readable reason for override
returns an object with this item:
- id: integer ID of new parameter override
or an item called error
if the override already exists or there were problems
with the specified parameters.
describe_program(progName, authToken) - administrative users only
- progName: (optional) string scalar giving name of program for which
to return information
return information about a program.
If progName
is not specified,
then return an object with this array item:
- progName: string array of possible values for `progName`.
If progName
is specified, then return an object with these array items:
- paramName: parameter name
- paramIsFlag: boolean; if `true`, parameter does not take a value; otherwise,
parameter takes a floating point value
- paramInfo: human-readable description of parameter
and these non-array items:
- version: string scalar; current version of program
rerun_receiver(projectID, serno, minBN, maxBN, authToken) - administrative users only
- projectID: integer scalar; Motus project ID, only detections during deployments belonging to this project will be plotted
- serno: string scalar; receiver serial number
- minBN: integer scalar; the least boot number to reprocess
- maxBN: integer scalar; the greatest boot number to reprocess
e.g. curl --data-urlencode json='{"projectID":1,"serno":"SG-2616BBBK1111_1","minBN":1,maxBN:"2","authToken":"XXX"}' https://sgdata.motus.org/status2/rerun_receiver
error
If successful, all boot numbers from minBN to maxBN will be reprocessed.
2019-10-10
- add new entry rerun_receiver
which lets administrators submit a job to reprocess receiver data
2019-01-21 - clarify return value of delete_param_overrides
2018-03-27
- add new entry get_receiver_file
to allow download of individual raw receiver
files (by serno, fileID)
2018-02-22
- add new entry serno_collision_rules
which can get, set, or delete rules for
resolving serial number collisions between receivers
- document authToken
parameter explicitly for a few entries where it
was missing
2018-02-06
list_jobs
: clarify semantics of option.limit
parameter
2018-01-30
get_upload_info
:
- no longer for admin users only
- return error if specified file does not belong to a project user has permissions to
- allow query by sha1 hash instead of uploadID
- allow specifying listContents=false
for lightweight test of file existence
2018-01-29
- list_jobs
now supports "serno":"XXX"
in its selector
field, to extract top-level jobs
any of whose subjobs has a serno='XXX'
parameter, as this is quick.
2018-01-26
- list_jobs
now supports having both options countOnly
and errorOnly
be true
,
which returns a count of jobs having at least one subjob with errors.
2018-01-18
- new get_upload_info
returns details about an uploaded file
2018-01-17
- new retry_job
submits a job with errors for retrying.
2018-01-11
- (doc only) detail error returned by process_new_upload
2018-01-10
- process_new_upload
now optionally accepts an email
address to notify user when
processing completes.
2018-01-09
- process_new_upload
now optionally accepts ts as a string timestamp, not just a double.
2018-01-06
- process_new_upload
now moves file, changes ownership, and returns path to new location
2017-12-18:
- new get_job_stackdump
returns a URL and path for the stackdump of a job with an error.
(admin users only)
2017-12-15:
- list_receiver_files
unify handling of file counts in SG, Lotek receivers
2017-12-12:
- list_receiver_files
moves table results to a field named either fileCounts
or fileDetails
, adds field serno
- new get_receiver_info
returns device info including deployments table
2017-12-11:
- list_jobs
now uses a bare sortBy
with just a field name; the optional desc
is moved to its own boolean field: sortDesc
.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.