circuitscan-pipeline-runner

Pipeline packages must have this package as a production dependency:

yarn add circuitscan-pipeline-runner

Pipeline default export

The default export async function from the main file specified in package.json on your pipeline package will be invoked.

export default async function(event, { status })

Arguments

event

Includes a payload object key which is the input from the user's CLI.

status

Includes a synchronous log(msg, data) method for sending status updates to the user's CLI and website frontend build output.

status.log('Reticulating Splines...', { splineLen: 123 });

Return value

It should return the pkgName string on success or throw an error on failure. This package name should have a suffix appended by the uniqueName() function.

During its operation, the pipeline package should upload a file at build/<pkgName>/info.json (using uploadLargeFileToS3()) containing the necessary data to display the information about the circuit. Other build artifacts should also go in this directory.

Standard library

Also includes a standard library for interacting with build artifacts and relaying compiler status.

Function Name | Arguments | Note --------------|-----------|---------- uniqueName | prefix (string) | Add a suffix that contains an adjective, color, and animal to make the prefix unique async execPromise | cmd (string)options (object, optional) | Node.js fs.exec promisified monitorProcessMemory | processName (string)timeout (number)callback (function) | Callback invoked at interval, returns function that can be invoked to stop monitoring async downloadBinaryFile | url (string)outputPath (string) | Download file from HTTPS using streams async uploadLargeFileToS3 | keyName (string)filePath (string)logger (object, optional) | Upload filePath to the artifact S3 bucket using streams. If specified, logger must have a log(msg) method. async zipDirectory | sourceDir (string)outPath (string)logger (object, optional) | Create a zip file of a directory. If specified, logger must have a log(msg) method. mkdirpSync | targetDir (string) | Recreation of shell command mkdir -p async deleteS3Keys | keys (Array[string]) | Delete some files in the BLOB_BUCKET from S3 class MockStatusReporter | none | For testing, logs are available as array on logs property

Barebones Example

import {writeFileSync} from 'node:fs';
import {join} from 'node:path';
import {tmpdir} from 'node:os';
import {
    execPromise,
    monitorProcessMemory,
    uniqueName,
    uploadLargeFileToS3,
} from 'circuitscan-pipeline-runner';

export default async function(event, { status }) {
    // TODO ... input validation ...

    const pkgName = uniqueName(circuitName);
    const dirPkg = join(tmpdir(), pkgName);

    const compilePromise = execPromise(`gcc foo.c`);
    const cancelMemoryMonitor = monitorProcessMemory(
      'gcc',
      10000,
      memoryUsage => {
        status.log(`Compiler memory usage`, { memoryUsage });
      }
    );
    await compilePromise;
    cancelMemoryMonitor();

    writeFileSync(join(dirPkg, 'info.json'), JSON.stringify({
      requestId: event.payload.requestId,
      type: 'mypipeline',
      importantProperty: 123,
      foo: 'bar',
    }, null, 2));
    await uploadLargeFileToS3(`build/${pkgName}/info.json`, join(dirPkg, 'info.json'));

    return pkgName;
}

See Also

• Circuitscan Circom Pipeline repository

License

MIT