archimedes-jobs
v1.0.4
Published
Archimedes is an abstract distributed job engine on AWS.
Downloads
23
Readme
Archimedes
Archimedes is an abstract distributed job engine on AWS.
SDK use
Import SDK
const Archimedes = require("archimedes-jobs")("<NAME_OF_LAMBDA_FUNCTION>");
import archimedes from "archimedes-jobs";
const Archimedes = archimedes("<NAME_OF_LAMBDA_FUNCTION>");
Create job
Archimedes.create({
jobType: "<JOB_TYPE>",
inputData: {
//<all json-like input data the job needs to know>
},
inputFiles: {
//<all file-like input data like urls to s3 the job needs to know>
},
estimations: {
//<all job meta metrics the system needs to know in order to triage job execution>
}
}).then(job => {
console.log(job);
}).catch(err => {
console.log(err);
});
Run job
Archimedes.signal({
jobId: job.id,
state: "DISPATCH"
}).then().catch();
Handle job
Payload for job execution will be:
{
"id": "<JOB_ID>"
}
Then use Archimedes to get all job data including inputData
and inputFiles
:
let job = await Archimedes.get({
jobId: "<JOB_ID>"
});
Then signal that we are executing this:
job = Archimedes.signal({
jobId: job.id,
state: "EXECUTE"
})
This will in particular now add a currentExecution
object to the job object:
let jobExecution = job.currentExecution; /* {
executionId: 'XXX',
createdAt: 'YYY',
updatedAt: 'ZZZ',
metrics: {},
progress: 0,
state: 'EXECUTE',
failureReason: '',
finishedAt: null
} */
While you're executing you might want to periodically upgrade metrics (e.g. max cpu load, max memory usage etc.) as well as progress. This can be of use two-fold:
- your own application might query this data to show job progress to the user
- Archimedes is scanning jobs in the background for non-progression to make sure to kill and reschedule stuck jobs
To update progress and/or metrics:
Archimedes.updateExecution({
jobId: "XXX",
executionId: jobExecution.executionId,
progress: 0.5,
metrics: {
cpu: 1234,
memory: 5678
}
})
Once you're done with the job, you can store your outputData
and outputFiles
like so:
Archimedes.update({
jobId: "XXX",
outputData: {
...
},
outputFiles: {
...
}
})
And you transition the job to success or failure like so:
Archimedes.signalExecution({
jobId: "XXX",
executionId: jobExecution.executionId,
state: "SUCCESS" // "FAILURE"
// failureReason: "XXXX"
})