inst/doc/status_api.md

The motusServer status API

This API provides access to information about the motus data processing server, jobs, and errors. It also permits some server-side tasks to be executed (job retry, server restart, processing of new uploaded file, ...)

API summary

Request

Reply

Examples are given for each call using the command-line client curl with quoting appropriate for the Bash shell. These examples return the raw bzip2-compressed data. To view the response, redirect the output of curl into a file and use 7zip to decompress it (for example), or add | bunzip2 -cd to the end of the command in Bash.

The server is at https://sgdata.motus.org and the URL prefix is "/status".

API calls

status_api info

status_api_info (authToken)

  e.g.
  curl https://sgdata.motus.org/status2/status_api_info

authenticate user

authenticate_user (user, password)

  - user: username
  - password: password (in cleartext)

  e.g.
  curl --data-urlencode json='{"user":"someone","password":"bigsecret"}' https://sgdata.motus.org/status2/authenticate_user

or

Notes

  1. The authToken returned by this API must be included in most other API calls (for an alternative, see 2.)

  2. Instead of obtaining the token from this API entry, you can use the non-URL-encoded value of the cookie called auth_tkt, obtained by logging in via the main login page. The non-URL-encoded value of the auth_tkt cookie can be used as authToken, which avoids having to call authenticate_user if the user has already logged in. To use auth_tkt, the IP address from which the login occurred must be passed as a dotted-quad string in the header field X-Forwarded-For to any of the requests in this API, along with passing the cookie value in the JSON field authToken.

The correct X-Forwarded-For header is generated automatically if the status API is accessed from client-side javascript. However, if the URL is accessed from a PHP script on a server, the automatically-generated IP address for the header will be that of the server running the PHP script, rather than of the ultimate client, which is wrong. So to call this API from a server-side script, you need to specify the X-Forwarded-For header explicitly, in which case no value will be generated automatically. Here's an example in php:

/// @param $url: full URL of API entry
/// @param $par: associative array of API parameters

function post ($url, $par) {
    $ch = curl_init($url);
    $par['authToken'] = $_COOKIE['auth_tkt'];
    $json = 'json=' . urlencode(json_encode($par));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_HEADER, false);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/x-www-form-urlencoded',
                                               'X-Forwarded-For: ' . $_SERVER['REMOTE_ADDR']));
    curl_setopt($ch, CURLOPT_POSTFIELDS,  $json);
    $res = curl_exec($ch);
    curl_close($ch);
    return(json_decode(bzdecompress($res), true));
}
  1. Authorization is by project: if a user has permission for a project, then that user can see:

  2. status of all jobs submitted for that project

If an API call does not find any information for which the user is authorized, it will return a json object of the usual structure, except that column arrays will have length zero. This represents an R data.frame with the correct column names but zero rows.

The API doesn't currently provide a way to tell whether there are additional data which would be returned for a given call if the user had authorization for more projects.

list_jobs

list_jobs (projectID, select, order, options, authToken)

   - projectID: integer; motus projectID (s); jobs must belong to the given project(s)
   - select: object with fields for selecting which jobs to list.  A job must match all
     fields provided in order to be included in the list.  Fields are:
      - userID: integer; motus user ID
      - stump: ID of top level job; select only from that job and its descendents; for increased
        flexibility, if the provided stump is not the ID of a top-level job, then that job's
        stump is used instead.
      - jobID: integer array; to select only those jobs specified
      - type: string array; the job type(s)
      - done: integer; 0: job not yet complete; 1: job completed successfully; -1: job had error
      - log: string; job whose log matches the string `log`, which can include globbing
        characters ('*' and '.')
      - serno: string; serial number; jobs that processed data from this receiver; any top-level
        job at least one of whose subjobs was for that receiver are included.  (An upload job
        might have data from multiple receivers)
   - order: object with fields for ordering and paging the selected jobs:
      - sortBy: string scalar; null, or the sort key, i.e. one of these string constants:
         - "ctime": job creation time
         - "mtime": last job activity time
         - "id": job ID number
         - "type": job type
         - "motusProjectID": motus project ID
         - "motusUserID": motus user ID
        If `sortBy` is not specified, it is set to `"mtime"`. To allow paging, an implici "id" is
        added to sortBy if it is not already there
      - sortDesc: optional; logicl scalar; if `true`, sorting is in descending order by key (and id within
        key); otherwise, in ascending order by key (and id within key)
      - lastKey: optional; vector giving "last" obtained value of the field specified in `sortBy`; an optional
        second element gives the "last" obtained value of the "id", if `sortBy` is not "id".  This is
        used for paging.  If not specified, returns the first page according to `sortBy` criteria.
      - forwardFromKey: boolean; default: true.  If `lastKey` is specified and this field is true,
        return a page of items past the key in the forward (sort) direction.  Otherwise, if this field is false,
        return a page of items past the key in the backward (reverse sort) direction.
   - options: object with fields giving options:
      - includeUnknownProjects: include jobs with no associated motus project; typically for
        jobs initiated by staff or otherwise not having a useful concept of project
      - includeSubjobs: boolean: include jobs which are not top-level jobs?; default `false`
      - errorOnly: boolean: if true, only show top jobs for which a subjob had an error.
      - excludeSync: boolean: if true, omit `syncReceiver` jobs (there will be many of these)
      - full: if `true`, then full details for the job (typically its parameters, log, summary, and list of
        product files) are returned in a JSON-formatted column called `data`
      - countOnly: boolean; if `true`, return only a count of jobs for the given projectID and/or userID,
        as an object with field `count`, and value an integer array of length one. e.g. `{count:[123]}`
      - limit: integer; if present, maximum number of records to return.  This is ignored if
        - it is larger than the maximum result size specified when the server was launched (default: 20)
        ** or **
        - `select.stump` is specified, in which case **all** subjobs are returned.  This is because the API
        does not currently provide any way to paginate subjobs.

  e.g.
  curl --data-urlencode json='{"select":{"userID":232},"order":{"sortBy":"id","lastKey":[5000]},"authToken":"XXX"}' https://sgdata.motus.org/status2/list_jobs

process new upload

process_new_upload (userID, projectID, path, ts, email, authToken)

  - userID: integer scalar; motus user ID (who uploaded the file)
  - projectID; integer scalar; motus projectID (what project should own the products; as chosen by upload user)
  - path; string scalar; path to the new file on the NAS, relative to the value of `uploadPath` returned by the `status_api_info` API.
    e.g. if `path` is given as 'partial/123_2017-10-20T11-12-33_upload.zip`, then
    on linux, we'll expect to find the file at `nfs://174.140.177.35:/volume1/sgdata/sgm/uploads/partial/123_2017-10-20T11-12-33_upload.zip`
    which will have actual path /sgm/uploads/partial/123_2017-10-20T11-12-33_upload.zip, given the current NAS mountpoints.
    path` must not include any '..' component; i.e. ascent up the file tree is not permitted, to prevent malicious use
    from leaking system information.
    Both forward (`/`) and reverse (`\`) slashes are interpreted as folder delimiters.
  - ts; either double or string scalar; time at which file upload completed.  If double, interpreted as seconds since 1 Jan 1970, GMT;
    If string, interpreted as a datetime compatible with `R:lubridate::ymd_hms()`.  If not supplied, the current
    time is used.
  - email; optional string scalar;  if present, an email is sent to that address when processing of the uploaded file is complete.

  e.g.
  curl --data-urlencode json='{"userID":232,"projectID":57,"path":"232/232_2017-10-20T11-12-33_myupload.zip","ts":1508497953,"authToken":"XXX"}' https://sgdata.motus.org/status2/process_new_upload

If the file has already been uploaded, then error is set to a message, and the field details contains the details of the existing copy of the file, with subfields jobID, uploadID, motusUserID, motusProjectID, filename, sha1, ts corresponding to the file's treatment when it was first uploaded.

Note: The check for whether a file has been uploaded is by contents, not name, so renaming a file and attempting to upload it again will result in the same error.

Recommended format for file paths: We want file paths: - encoded in UTF-8 - no : (colon) or " (double quote) characters in path - path separator is / (forward slash) - to provide intrinsic collision avoidance - identify the user who uploaded them

So, a file path should look like this:

      XXX/123_2017-10-20T15-21-35.123_user_name_for_file.zip

where:

Debugging feature: if path begins with testing/, then no processing will be done, and if otherwise successful, this call will return with an error whose message will confirm the file was received but that no processing will take place.

list_receiver_files

list_receiver_files (serno, day, authToken)

  - serno: string scalar; receiver serial number
  - day: string scalar; day, formatted as "YYYY-MM-DD" (treated as GMT); if missing or null,
    this API returns a list of days, rather than a list of files

  e.g.
  curl --data-urlencode json='{"serno":"SG-1513BBBK0291","day":"2017-04-04","authToken":"XXX"}' https://sgdata.motus.org/status2/list_receiver_files

get_receiver_file

get_receiver_file (serno, fileID, authToken)

get_receiver_info

get_receiver_info (serno, authToken)

  - serno: string scalar; receiver serial number
  e.g.
  curl --data-urlencode json='{"serno":"SG-1513BBBK0291","authToken":"XXX"}' https://sgdata.motus.org/status2/get_receiver_info

get_job_stackdump

get_job_stackdump (jobID, authToken) - administrative users only

  - jobID: integer; ID of job with an error (i.e. `done` < 0)

The .rds file contains an object of R class dump.frames. It is a list whose names are the calls and whose elements are the environments of the calls (i.e. contain the variables 'defined within' each level of function call.

Because stack dump files might leak passwords or other credentials, this API call only works for administrators.

retry_job

retry_job (jobID, message, authToken) - administrative users only

  - jobID: integer; ID of job with an error, or one of whose siblings has an error
  - message: string; optional message to add to job log, indicating e.g. reason for retry

Otherwise, an error message is returned in item error.

get_upload_info

get_upload_info (uploadID, sha1, listContents, authToken)

  - uploadID: integer (optional); ID of uploaded file
  - sha1: string, hex digits (optional); sha1 hash of file contents
  - listContents: boolean (optional); return summary of file contents?
    default: true.

Exactly one of uploadID or sha1 must be given, and is used to look-up the file. Specifying sha1 is meant to let a client check whether a file has already been uploaded (by sha1 hash of its contents) before wasting a user's time and bandwidth.

Otherwise, an error message is returned in item error.

serno_collision_rules

serno_collision_rules (action, id, serno, cond, suffix, authToken) - administrative users only

  - action: string (required); one of "get", "put", or "delete".  Presence and
    use of other parameters depends on this value.
  - id: integer scalar or array; id of rule in database
  - serno: string scalar or array; serial number(s) of receiver(s) to which the
    rule applies.  This must be the bare serial number, **without** the
    disambiguation suffix.
  - cond: string scalar; an R expression using certain terms extracted from
    a file or filename.  When this condition evaluates to TRUE for a file, the
    file is deemed to have come from the receiver given by concatenating
    `serno` and `suffix`.  Only the first matching rule is used when trying
    to decide which receiver in a collision is the source of the file.
  - suffix: string scalar; '_N', where N is a small integer.  When two (
    or more?!) receivers have the same serial number, the suffix `_1` is
    added to the second one, `_2` to the third one, and so on.  The first
    receiver with a given serial number receives no suffix.  (The ordering
    implied by `first`, `second` etc. is given implicitly by the rules).

Semantics:

action == "get": fetch all rules whose id was specified or whose serno was specified, or all rules if neither was specified. The return value is an object with these named fields: - id; integer; rule ID - serno; string; bare receiver serial number - cond; string; R expression - suffix; string; '' or '_1', or '_2', ...

action == "put": insert new rules. The user must specify serno, cond, and suffix, and each must be a scalar, or all must be arrays of the same length. The id field will be automatically assigned. The return value is as for action == "get", but includes only those rules just created. NO ATTEMPT IS MADE TO ENSURE THE SANITY OF THE RULES

action == "delete": delete any rules for which the user specified an id, or for which the user specified a serno. The return value is as for action == "get", but includes only those rules just deleted.

get param_overrides

get_param_overrides(id, projectID, serno, progName, authToken) - administrative users only

  - id: optional; integer array of param override IDs
  - projectID: optional; integer array of motus project IDs
  - serno: optional; string array of receiver serial numbers
  - progName: optional; string array of program names to which parameter overrides apply

return the set of all parameter overrides matching all specified criteria. If no criteria are supplied, then all overrides are returned. The returned object has these array items:

  - id: integer; IDs of parameter overrides
  - projectID: integer; motus project IDs (each can be null)
  - serno: character; device serial numbers (each can be null)
  - tsStart: double; starting timestamps (each can be null; seconds since 1 Jan 1970 GMT)
  - tsEnd: double; ending timestamps (each can be null; seconds since 1 Jan 1970 GMT)
  - monoBNlow: integer; starting boot session numbers (each can be null; used for SGs only)
  - monoBNhigh: integer; ending boot session numbers (each can be null; used for SGs only)
  - progName: character; names of programs to which override applies; typically 'find_tags_motus'
  - paramName: character; names of parameters (e.g. 'default_freq')
  - paramVal: double; values for parameters (each can be null if parameter is just a flag)
  - why: character; human-readable reason for each override

delete_param_overrides

delete_param_overrides(id, authToken) - administrative users only

  - id: integer array of param override IDs

delete the parameter overrides whose IDs are in id, returning a boolean array of the same length indicating which IDs are now not in the database (i.e. were deleted or were already not present).

add_param_override

add_param_override(projectID, serno, tsStart, tsEnd, monoBNlow, monoBNhigh, progName, paramName, paramVal, why, authToken)

  - projectID: integer; motus project ID (can be null)
  - serno: character; device serial number (can be null)
  Exactly one of `serno` or `projectID` must not be null.

  - tsStart: double; starting timestamp (can be null; seconds since 1 Jan 1970 GMT)
  - tsEnd: double; ending timestamp (can be null; seconds since 1 Jan 1970 GMT)
  - monoBNlow: integer; starting boot session number (can be null; used for SGs only)
  - monoBNhigh: integer; ending boot session number (can be null; used for SGs only)
  - progName: character; name of program to which override applies; typically 'find_tags_motus'
  - paramName: character; name of parameter (e.g. 'default_freq')
  - paramVal: double; value for parameter (can be null if parameter is just a flag)
  - why: character; human-readable reason for override

returns an object with this item:

  - id: integer ID of new parameter override

or an item called error if the override already exists or there were problems with the specified parameters.

describe_program

describe_program(progName, authToken) - administrative users only

  - progName: (optional) string scalar giving name of program for which
  to return information

return information about a program.

If progName is not specified, then return an object with this array item:

  - progName: string array of possible values for `progName`.

If progName is specified, then return an object with these array items:

  - paramName: parameter name
  - paramIsFlag: boolean; if `true`, parameter does not take a value; otherwise,
    parameter takes a floating point value
  - paramInfo: human-readable description of parameter

and these non-array items:

  - version: string scalar; current version of program

Changelog

2019-01-21 - clarify return value of delete_param_overrides

2018-03-27 - add new entry get_receiver_file to allow download of individual raw receiver files (by serno, fileID)

2018-02-22 - add new entry serno_collision_rules which can get, set, or delete rules for resolving serial number collisions between receivers - document authToken parameter explicitly for a few entries where it was missing

2018-02-06 list_jobs: clarify semantics of option.limit parameter

2018-01-30 get_upload_info: - no longer for admin users only - return error if specified file does not belong to a project user has permissions to - allow query by sha1 hash instead of uploadID - allow specifying listContents=false for lightweight test of file existence

2018-01-29 - list_jobs now supports "serno":"XXX" in its selector field, to extract top-level jobs any of whose subjobs has a serno='XXX' parameter, as this is quick.

2018-01-26 - list_jobs now supports having both options countOnly and errorOnly be true, which returns a count of jobs having at least one subjob with errors.

2018-01-18 - new get_upload_info returns details about an uploaded file

2018-01-17 - new retry_job submits a job with errors for retrying.

2018-01-11 - (doc only) detail error returned by process_new_upload

2018-01-10 - process_new_upload now optionally accepts an email address to notify user when processing completes.

2018-01-09 - process_new_upload now optionally accepts ts as a string timestamp, not just a double.

2018-01-06 - process_new_upload now moves file, changes ownership, and returns path to new location

2017-12-18: - new get_job_stackdump returns a URL and path for the stackdump of a job with an error. (admin users only)

2017-12-15: - list_receiver_files unify handling of file counts in SG, Lotek receivers

2017-12-12: - list_receiver_files moves table results to a field named either fileCounts or fileDetails, adds field serno - new get_receiver_info returns device info including deployments table

2017-12-11: - list_jobs now uses a bare sortBy with just a field name; the optional desc is moved to its own boolean field: sortDesc.



jbrzusto/motusServer documentation built on May 19, 2019, 8:19 a.m.