atomic completion of jobs and subjobs
on interrupt / restart, must be able to resume uncompleted jobs and jobSteps (we're not there yet)
full details on what has been done and its status
easy to read for web server
NO: pure filesystem (e.g. filenames that encode steps, parameters; existing version)
NO: .sqlite database in each job directory
YES: master database in /sgm/server.sqlite; transient files and email messages in file tree
incoming emails processed by /sgm/bin/incomingEmail.sh
message -> drop carriage returns -> /sgm/inbox/
emailServer: job is to watch for new messages (via close_write) in /sgm/inbox, then create and queue a new job; does sanity checks and recursively unpacks archives until there are no recognized archive types left
uploadServer: watches for new uploads (via the ProjectSend code hooked on sensorgnome.org/upload; unpacks newly-uploaded files recursively, then queues job folder for processing
processServer: process jobs; there can be multiple instances of processServer running.; each watches the master incoming directory /sgm/queue/0 and competes to claim new jobs which are moved there; this happens via atomic operations on the 'queue' field of records in the job table.
statusServer: generates web-page summaries of job processing; currently this is called only via the server-side includes on sensorgnome.org pages:
and
https://sensorgnome.org/My_Job_Status
which don't permit full navigation, as I haven't found a convenient way to do that on the sensorgnome.org dekiwiki
queue: queue in which this job resides; 'E': email; 'U': upload '1'...'8' process server; '0': awaiting a process server; only valid in top-level jobs
ctime: numeric; -- creation timestamp
type: type of job e.g. 'uploadFile' or 'email'
data: text; -- json string with additional job details:
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.