octoparse-client
v0.2.10
Published
Official JavaScript SDK for Octoparse DataHub: discover, run and consume Data Apps
Maintainers
Readme
octoparse-client-js — the official DataHub JavaScript SDK
The official JavaScript client for Octoparse DataHub (npm package octoparse-client): discover, run
and consume Data Apps. Written in TypeScript and shipped as compiled JS plus type declarations, so
plain JS projects can use it directly.
A small self-contained project with zero runtime dependencies (Node's built-in fetch only) that
speaks only the platform's public /v1 REST API.
Method-for-method aligned with the Python SDK (the octoparse-client pip package).
Requires Node >= 20.17 (ESM distribution; from this version CJS can require it directly).
Installation
npm install octoparse-clientUsage
import { Client } from "octoparse-client";
// baseUrl defaults to the production endpoint https://api-datahub.octoparse.com;
// pass it explicitly for local / staging environments
const client = new Client({ apiKey: "demo-key" });
// Discover
const result = await client.search("reviews", { limit: 10 });
const detail = await client.getApp("demo/reviews-store-query"); // or the card's app_id
// Run (wait for the terminal state) and consume. Time parameters are always in
// milliseconds (timeout / pollInterval; the Python SDK uses seconds). A timeout
// throws TimeoutError but does not cancel the server-side run.
const run = await client.call("demo/partner-reviews-api", { product: "p-9001" }, {
timeout: 120_000, raiseOnFailure: true,
});
for await (const record of client.iterateRecords(run.run_id)) {
console.log(record.content, record.rating);
}
// Inventory and reconciliation
for await (const r of client.iterateRuns({
runKind: "production", createdFrom: "2026-08-01T00:00:00Z",
})) {
console.log(r.run_id, r.state, r.billing);
}
console.log(await client.billing({ groupBy: "data_app", tzOffset: 480 }));
// Result data is kept for 90 days by default; mark datasets you want to keep
await client.setDatasetRetention(run.dataset_id, true);call() starts a run and polls until it reaches a terminal state; for non-blocking semantics use
run() to get a run_id, then getRun() / cancel() yourself. Apps shared with you
point-to-point do not appear in marketplace results; query them with
search("", { sharedWith: "me" }). All methods and the error mapping live in src/client.ts.
iterateRecords() and iterateDatasetRecords() automatically follow the opaque
pagination.next_cursor returned by cursor-capable DataHub deployments. This keeps all pages on
the same server snapshot, including when filtering produces an empty page. They page by offset
until a response first carries a cursor (older servers and local storage never send one, so they
stay on offset paging); from then on they follow cursors only and never go back to offset. For manual paging, pass that opaque value back as cursor to getRecords() or
getDatasetRecords(); cursor requests omit offset, and combining a cursor with a nonzero
offset is rejected.
getApp(appId, { version }) reads a historical version's contract, the same version string
run(appId, input, { version }) pins. listAppVersions(appId) lists the versions open to you
(newest first; latest marks the default), and getAppReadme(appId, { version }) returns the
App's README as Markdown text.
Run logs
Only code-kind runs have logs; for api-kind runs the endpoint returns an empty list with
complete=true. A code-kind run's log is the stdout and stderr of the App's code, and it is
visible only to whoever started the run (anyone else, the App's owner included, gets 404).
// Tail a run that is still in progress; stops once the log is complete
for await (const line of client.iterateRunLogs(run.run_id)) {
console.log(line.time, line.source, line.text);
}
// Or read one page: {items, pagination, complete, truncated}
const page = await client.getRunLogs(run.run_id, { offset: 0, limit: 1000 });iterateRunLogs() reads on from the last line number + 1 and long-polls with wait (seconds, at
most 20 per request) until complete is true; follow: false reads only the lines that already
exist. source is app for the App's own output and platform for lines the platform writes
(attempt separators, the size-limit marker). truncated: true means the output exceeded the size
limit: the middle was dropped, the beginning and the end of each attempt are kept. Values the
platform handed to the run (environment values, inputs marked sensitive) and common credential
patterns are masked before the log is stored. The endpoint also serves plain text with
format=text, which the SDK does not wrap.
App references come in two forms, accepted interchangeably by getApp() / run() / call() and
by the dataApp filter of the run listings: the two-part <namespace>/<app_name> reference
(taken from the card's namespace and app_name, human-readable, breaks once the publisher renames
the App); and the stable identifier app_id (app_<hex>, taken from the card's / detail's app_id
field, immune to renames). For long-term integrations such as config files or scheduled jobs, pin
the app_id.
Configuration
When new Client() is constructed without baseUrl / apiKey, they fall back to the
OCTOPARSE_BASE_URL / OCTOPARSE_API_KEY environment variables (template in .env.example; the SDK
does not load .env itself, the caller does), then to the production endpoint
https://api-datahub.octoparse.com / anonymous access.
Tests
npm test # offline unit tests (injected fetch mock, no server required)
npm run typecheck # strict type checking (including tests)