bazhuayu-dataapp
v0.0.2
Published
Developer SDK for building Bazhuayu DataHub Data Apps: read input, push records, handle aborts
Maintainers
Readme
bazhuayu-dataapp
The developer SDK for building Bazhuayu DataHub Data Apps in JavaScript / TypeScript
(npm package bazhuayu-dataapp). Your app.js is the program; the SDK is a library you
call inside DataApp.main() to read the run input, push records and react to aborts.
To discover, run and consume Data Apps as a caller, use bazhuayu-client instead.
Installation
npm install bazhuayu-dataappRequires Node.js 20.17 or newer. The package is ESM (import) and ships TypeScript types.
Usage
import { DataApp, Event } from "bazhuayu-dataapp";
await DataApp.main(async () => { // enter: fetch the run input
const inp = await DataApp.getInput();
DataApp.on(Event.ABORTING, async () => { // cancel / timeout / record cap
await DataApp.exit();
});
const apiKey = DataApp.getEnv("example_api_key"); // alias from .dataapp/runtime.json
let i = 0;
for (const keyword of inp.keywords) {
DataApp.log.info("processing %s", keyword);
await DataApp.pushData({ id: keyword, title: `Result for ${keyword}` });
await DataApp.setProgress(++i, inp.keywords.length);
}
}); // leave: report the final stateRun it with node app.js, both locally and on the platform.
API
| API | Description |
|---|---|
| await DataApp.main(async () => { ... }) | Runs the Data App. Returning normally reports succeeded; a thrown error reports failed (or partially_succeeded if the platform has accepted records) and is rethrown. |
| await DataApp.getInput() | The run input, with top-level defaults from input_schema.json applied. |
| await DataApp.pushData(recordOrArray) | Pushes records. Each record must be a plain object that is JSON-serializable (no BigInt, NaN or Infinity) and match output_schema.json. The SDK batches writes and retries transient failures in the background; if the platform keeps refusing writes, pushData waits instead of buffering without limit. A batch the platform rejects outright fails the run with push-rejected. |
| DataApp.getEnv(alias, defaultValue?) | Reads an env entry declared in runtime.json env. Treat values as secrets. |
| await DataApp.setProgress(done, total, statusText?, { upstreamRef }) | Progress and a human-readable status. Throttled; only the latest value is sent. upstreamRef is the upstream system's own task id, shown on the run for tracing. It is kept until a progress report succeeds, even if later progress values replace the rest. statusText longer than 500 characters is truncated. |
| DataApp.on(Event.ABORTING, handler) | Called once on cancel, timeout, record cap or SIGTERM/SIGINT, with { reason } ("cancel", "timeout", "max-records" or "signal"). Without a handler the SDK calls exit() for you. On timeout, if the run is still going 15 seconds after your handler returns, the SDK ends it (final state run-timeout); cancel stays cooperative. |
| await DataApp.exit() | Flushes, reports the final state and ends the run. |
| await DataApp.fail(message, { code, retryable }) | Ends the run with a structured error. |
| DataApp.signal | An AbortSignal that is aborted when the run is ending. Pass it to fetch and other cancellable I/O. |
| DataApp.log | A logger writing to stderr (debug / info / warn / error, util.format placeholders); env values are masked. |
| DataApp.isAtHome | true on the platform, false locally. |
Custom billing quantities are derived by the platform from pushed records
(pricing.json events[].qty_from); there is no manual charge call.
Ending the run from inside main
JavaScript cannot cancel a running async function. After await DataApp.exit() or
await DataApp.fail(...) the final state is already reported: return from your function
right away. Records pushed afterwards are dropped with a warning. If the function is still
running one second later (for example an ABORTING handler called exit() while a long loop
is in flight), the SDK ends the process so the platform can reap the run; use
DataApp.signal to stop in-flight I/O promptly.
Local mode
Without DATAHUB_SERVER_URL and DATAHUB_WORKER_TOKEN the SDK runs locally:
- input is read from
storage/input.jsonand validated against.dataapp/input_schema.json; - records are written to
storage/dataset/records.jsonl(cleared on each run) and checked against.dataapp/output_schema.json(warnings only); getEnvreads the process environment, then.envin the project root;- progress and the final state are logged; Ctrl-C triggers
ABORTING, a second Ctrl-C ends the run as cancelled;fail()exits with status 1.
Platform mode
The platform runs your image as is (its own ENTRYPOINT / CMD, e.g. node app.js) and
injects DATAHUB_SERVER_URL and DATAHUB_WORKER_TOKEN (a credential scoped to this one run).
On entering DataApp.main() the SDK claims the run, then fetches the env entries declared in
runtime.json if there are any. The run time limit comes from execution.timeout_seconds.
Every request carries the runtime identity header
X-DataHub-Runtime: lang=javascript; sdk=<version>; protocol=1.
If the claim returns nothing to run (the run was cancelled before it started, or this SDK
version is no longer supported and the platform has already failed the run), the process
exits with status 0 and the main body never runs.
Exit codes only matter if the process exits before it claims its run. The platform reads them as follows:
| Code | Meaning |
|---|---|
| 0 | Nothing to run, or the claimed run already reached a final state |
| 2 | Only one of DATAHUB_SERVER_URL / DATAHUB_WORKER_TOKEN is set (a wrapper dropped the environment) |
| 75 | The platform could not be reached or rejected the credential; the platform retries the run |
Do not exit with 75 from your own code: a pre-claim exit with that code is treated as a
platform failure and retried.
Logs
On the platform, everything your app writes to stdout and stderr (DataApp.log, console.log
and uncaught errors) becomes the run's log. The person who started the run can read it, for
test runs and production runs alike; the app owner cannot read the logs of runs started by
others. The output of your image build is the build log, visible to you as the owner.
Before a log is stored, the platform masks the env values handed to the run, inputs marked
sensitive, platform credentials and common credential patterns. Anything else you print is
shown as is, so do not print personal data or secrets you obtained in other ways. Run logs
are kept for 30 days. A run keeps at most 2 MiB of log (the beginning and the end of each
attempt), and lines longer than 16 KiB are cut.
License
MIT
