rainbo

Rainbo command line tools for data migration workflows.

The published rainbo npm package is a lightweight meta package. It installs the matching platform binary package for the current machine at install time. Those platform packages are published under the @singdata scope.

Requirements

• Node.js 22.12.0+
• Rust toolchain for source development. End users installing from npm get a prebuilt binary.
• Supported CI release platforms: macOS x64/arm64 and Windows x64/arm64.

Install

First-time install:

npm install -g rainbo
rainbo version

Rainbo supports user-level global installation only: npm install rainbo in a project is rejected. Project-scoped skills installation is also unsupported. Sync the AI-agent skills globally with:

rainbo update --skills-only --force

If GitLab is unavailable, use the offline package from 数据校验宝典:

rainbo update --skills-only --force --skills-package /path/to/rainbo-skills.zip

Maintainers can regenerate the offline package from the repository root with:

cd packages/rainbo
npm run pack:skills

This writes ../../skills.zip, a standard npx skills source archive that contains skills/index.json plus all official Rainbo skill directories. The archive is a local build artifact and is not committed. Rainbo production images generate /home/skills.zip from the image's skills/ directory at container startup, so the source files and offline package cannot drift.

If rainbo is already installed, prefer the built-in upgrade flow instead of running bare npm install -g rainbo:

rainbo update

rainbo update upgrades the npm package and syncs official Rainbo skills in the user global scope, which avoids _notice.skills drift.

Development

node packaging/npm/run.js version
node packaging/npm/run.js help
npm run check:npm
npm run check:skills
npm run generate:openapi
npm run build:npm:local

generate:openapi exports the current FastAPI schema to the ignored rust/src/generated/openapi.json build input. npm test, build:release, and build:npm:local regenerate or verify it automatically before compiling Rust.

For iterative source development, prefer the source launcher or npm link. Do not run rainbo update while validating local source changes, because it installs the latest published npm package and can replace the local build.

cd packages/rainbo
npm link
rainbo version
rainbo help

To test the same package layout users install from npm, build local tarballs and install them globally:

cd packages/rainbo
npm run install:local

install:local builds the current platform binary, creates temporary platform and meta npm packages, installs both globally, and also installs the local platform package inside the global rainbo meta package. This avoids npm reusing an older nested optional dependency during local validation.

Use --sync-skills when the local skill package should be installed globally together with the local binary without triggering npm self-update:

npm run install:local -- --sync-skills

The local skills are installed for the current user, not in the repository.

Output

Command output is structured for humans, scripts, and AI agents:

rainbo version                 # JSON envelope on stdout
rainbo version --format pretty # Human-readable output

Successful JSON output uses:

{"ok":true,"data":{...}}

Errors are written to stderr:

{"ok":false,"error":{"type":"validation","subtype":"invalid_argument","message":"..."}}

Update

rainbo update --check
rainbo update
rainbo update --force
rainbo update --skills-only --force
rainbo update --skills-only --force --skills-package /path/to/rainbo-skills.zip

update --check only reports status. Normal JSON command output may include _notice.update when a newer npm version is known from the local update cache and _notice.skills when locally synced skills are out of sync with the running binary. The update cache refreshes at most every 2 hours. Both notices recommend rainbo update.

update only runs npm when the registry reports a newer version. When the installed version is already current, it skips both the npm install and skills sync. Use update --force to reinstall the current CLI version deliberately.

update --skills-only --force force-syncs Rainbo skills globally without checking npm or attempting a self-update. Project-scoped installation is not supported.

When a newer version is available, update detects npm installations, runs npm install -g rainbo@<latest>, verifies the new binary, and syncs repository skills with:

npx skills add [email protected]:ee/rainbo.git -g -y --agent '*'

Skill sync records detailed state in the user configuration directory and updates installed official skills incrementally when possible. If incremental planning is not available, it falls back to a full global install.

When users cannot access the GitLab skills repository, they can use the offline skills package from 数据校验宝典 and run rainbo update --skills-only --force --skills-package /path/to/rainbo-skills.zip. The CLI skips npm and syncs skills from the local package path instead of the GitLab source. The offline package is installed through the same global npx skills protocol after Rainbo extracts the zip. The user-level skills state records the package path for future _notice.skills version drift checks.

If automatic update is unavailable, the JSON action is manual_required and the response includes release and changelog URLs.

Auth

rainbo auth connect --connection-string '<rainbo://connect?...>'
rainbo env list
rainbo env use <profile>
rainbo auth status
rainbo auth status --online
rainbo auth logout
rainbo auth datasource list
rainbo auth datasource set-default --id 2 --type dataworks --workspace ee_test_bj_01 --yes
rainbo migration-data-source +test --id 2

The Rainbo Web account security page generates a one-time rainbo://connect?... connection string. rainbo auth connect --connection-string '<connection-string>' exchanges it once and stores the resulting rotating refresh credential in the profile-local credentials.json file with owner-only permissions; auth.json contains only non-secret session metadata. Device Flow, --base-url, static CLI tokens, cookies, and RAINBO_CLI_TOKEN are not accepted. Access tokens refresh automatically, and auth logout revokes the server session before removing the local credential file.

The Web connection string establishes authentication only; it does not carry datasource or workspace defaults. After the user selects a datasource, persist that non-secret choice locally with rainbo auth datasource set-default. Use the explicit --workspace value only when the user selected one; otherwise pass workspace explicitly to the business command. This updates session metadata and never changes the rotating refresh credential or requires a new connection string.

rainbo auth status reads local non-secret metadata only. Use rainbo auth status --online only when an explicit server-side session validation is needed. Business commands load and refresh their own session, so they do not require a separate online status preflight.

CLI proxy precedence is --proxy / --no-proxy, scheme-specific proxy environment variables, ALL_PROXY, then direct access. NO_PROXY and no_proxy bypass matching hosts. Browser proxy settings remain independent.

Sensitive CLI requests receive HTTP 428 with an operation authorization URL. For agent-driven work, add --no-wait: the first command returns immediately with authorization_id and approval_url without executing the operation. After the same user completes a fresh TOTP verification in Rainbo Web, resume the unchanged request with:

rainbo auth operation wait <authorization_id> -- <original command...>

If waiting times out, rerun the same wait/resume command after the user confirms authorization. It reuses the existing authorization instead of creating another TOTP challenge. A caller that manages waiting separately can also rerun the original command with --operation-authorization-id <id>.

API Commands

Authenticated API commands reuse the session created by rainbo auth connect. Complex request bodies can be passed with --data-json, --data-file, or --data @payload.json. Add --dry-run to print the request method, path, query, and body without reading auth state or calling Rainbo APIs.

rainbo data-sync +cmt-jobs --data-file cmt-jobs.json
rainbo data-sync +cmt-jobs --data-file cmt-jobs.json --dry-run
rainbo data-sync +cmt-refresh --source-name mc-bj --db-name mc_project --table orders --wait-seconds 30
rainbo data-sync +cmt-refresh --source-name mc-bj --db-name mc_project --wait-seconds 0
rainbo data-sync +cmt-refresh-status --source-name mc-bj --db-name mc_project --table orders
rainbo data-verify +lineage-verify-table --data-file verify-table.json
rainbo data-verify +diff-diagnosis +create --data-file diff-diagnosis.json
rainbo data-verify +diff-diagnosis +list --status 6
rainbo data-verify +diff-diagnosis +status 123
rainbo data-verify +diff-diagnosis +result 123 --side source_only
rainbo data-verify +diff-diagnosis +columns 123
rainbo data-verify +diff-diagnosis +match 123 --key-column order_id
rainbo data-verify +diff-diagnosis +delete 123
rainbo convert notebook +parse --file notebook.sql
rainbo convert notebook +parse --file notebook.sql --file notebook.py
rainbo convert notebook +create --file notebook.sql --target zettapark
rainbo studio +tasks --datasource-id 13 --workspace demo_ws --max-files 5
rainbo studio task +list --datasource-id 13 --workspace demo_ws --file-type 400
rainbo studio task +detail --workspace demo_ws --file-id 10151727
rainbo studio task +flow-dag --project-id 49003 --flow-id 10157136
rainbo studio task +flow-node-detail --workspace demo_ws --flow-id 10157136 --node-id 10157232
rainbo studio +flow-use-config --workspace demo_ws --flow-id 10157152 --preview-only
rainbo migration-data-source +test --id 15
rainbo toolbox +spark2zetta +list --filename fact_hr_ot_hour
rainbo toolbox +spark2zetta +detail 42
rainbo toolbox +spark2zetta +download-source 42 --output ./source.py
rainbo toolbox +spark2zetta +download-result 42 --output ./result_zp.py
rainbo toolbox +spark2zetta +create --file ./spark_job.py
rainbo toolbox +sql-debug --datasource-id 1 --sql "select 1"
rainbo toolbox +volume-content --datasource-id 1 --volume public.v --file-path dir/result.json
rainbo toolbox +volume-download --datasource-id 1 --volume public.v --file-path dir/result.json --output ./result.json
rainbo toolbox +lineage-spark --data-file lineage-spark.json
rainbo toolbox dataworks-openapi search --params '{"keyword":"Task"}'
rainbo toolbox dataworks-openapi stats --params '{"date":"2026-09-15","datasource_id":2}'
rainbo toolbox dataworks-openapi show ListProjects
rainbo toolbox dataworks-openapi call --datasource-id 1 --operation ListProjects --params '{"PageSize":10}'

Every `dataworks-openapi call` requires a fresh one-time Web/TOTP user authorization before Rainbo invokes DataWorks. `dataworks-openapi stats` reports daily attempts and upstream calls by datasource and operation, so per-datasource quotas can be tracked.
Only read-only operations (`Get`/`List`/`Download`/`Find`/`Preview`/`Test`) can be called. Create, update, delete, execute, publish, and other write operations remain visible in metadata but are rejected before authorization or DataWorks invocation.

rainbo lineage +graph +create --data-file lineage-graph.json
rainbo lineage +graph +get "lg0624d"
rainbo lineage +graph --view-mode TABLE_GRAPH --graph-key "lg0624d"
rainbo lineage +upstream "lakehouse.dwd.user" --graph-key "lg0624d" --side TARGET --depth 1
rainbo lineage +downstream "lakehouse.ods.user" --graph-key "lg0624d" --side TARGET --depth 1
rainbo lineage +graph +delete "lg0624d" --password pengyu
rainbo lineage +graph +clear-others --password pengyu

Release

Rainbo releases are intended to be published by GitLab CI from release tags. Local publishing scripts remain available for emergency/manual validation, but the normal path is:

1. Merge code to the release branch after local and merge-request validation.
2. Create a prerelease tag for preview builds, for example rainbo-v0.2.3-beta.0. GitLab CI publishes it to the npm next dist-tag.
3. Ask testers to install the preview with npm install -g rainbo@next.
4. After acceptance, create the stable tag, for example rainbo-v0.2.3. GitLab CI publishes it to the npm latest dist-tag.

The tag version is the source of truth in CI. The pipeline runs scripts/set-release-version.js "$RELEASE_VERSION" so package.json, rust/Cargo.toml, Cargo.lock, and platform optional dependency versions are made consistent before building and publishing.

Use prerelease tags for shared preview testing. Avoid publishing preview builds directly from personal branches unless the pipeline explicitly uses a non-latest dist-tag such as next or a user-specific tag. For purely local acceptance, use npm run install:local and do not push or publish.

If publishing manually, first update package.json, rust/Cargo.toml, and the rainbo package entry in Cargo.lock to the same version. Do not publish a metadata-only version bump; the bundled Rust binary must report the same version as npm. Then export the npm publish token from the release machine:

source ~/.zshrc
export NPM_TOKEN="${NPM_TOKEN}"

NODE_AUTH_TOKEN is also supported. Use a granular access token with read/write access and "Bypass two-factor authentication" enabled. The publish script requires token-based publishing, creates a temporary npm config, verifies npm whoami --registry returns singdata, and refuses to publish any package whose package.json name is not rainbo.

Build, verify, dry-run pack, and publish from this directory:

npm run publish:npm:local

Preview and stable manual publish helpers are also available:

npm run publish:preview      # publishes with --tag next
npm run publish:release      # publishes with --tag latest

The publish script refuses to continue if package.json and rust/Cargo.toml versions differ. It also runs npm run check:npm to verify npm metadata, generated meta-package contents, and platform optional dependencies before publishing. After building the current-platform binary, it runs the bundled binary and verifies rainbo version matches package.json. The publish script is cross-platform and can be run from macOS shells, Windows PowerShell, or Windows CMD.

If npm returns EOTP, use an automation-capable publish token or pass the current OTP:

npm run publish:npm:local -- --otp=<code>

By default the local script builds the current platform binary into packaging/npm/bin/<platform>/. CI then uses those binaries to generate temporary platform packages such as @singdata/rainbo-cli-darwin-arm64 and @singdata/rainbo-cli-win32-x64, followed by the top-level rainbo meta package.

To build multiple platforms from a machine with the required Rust targets/toolchains installed, set:

RAINBO_NPM_BUILD_TARGETS=linux-x64,linux-arm64,darwin-arm64,darwin-x64,win32-x64,win32-arm64 npm run publish:npm:local

The package source lives under the Rainbo GitLab repository as packages/rainbo. Releases can be published locally or from the GitLab tag pipeline after the required platform artifacts are built. The GitLab pipeline generates platform npm packages dynamically in CI and then publishes the top-level rainbo meta package that depends on those platform packages. Prerelease tags such as rainbo-v0.2.3-beta.0 publish with npm dist-tag next; stable tags such as rainbo-v0.2.3 publish with npm dist-tag latest.

Skills

Repository skills live in skills/ and install globally for the current user:

npx skills add [email protected]:ee/rainbo.git -g -y --agent '*'

[email protected]:ee/rainbo.git is the skills repository source used by rainbo update. The old aggregate rainbo skill has been removed; current Rainbo skills use focused names such as rainbo-shared, rainbo-auth, rainbo-data-sync, rainbo-data-verify, rainbo-studio, rainbo-lineage, and rainbo-toolbox.

The repository root skills/index.json is the official skill list consumed by the standard npx skills installer. Keep it in sync with skills/*/SKILL.md and run npm run check:skills before publishing. Do not add agent-specific install paths such as Claude or Codex directories to the Rainbo CLI; the installer owns host compatibility.

New Rainbo skills should follow the style in the repository-level skill-template/: the top-level SKILL.md acts as a routing and safety rulebook, while detailed command families live in references/*.md. Business workflow skills should point to rainbo-shared first and then rainbo-auth when they call authenticated APIs instead of duplicating shared output, risk, dry-run, session, or secret-handling rules.

Project layout

packages/rainbo/:

rust/             Rust CLI implementation
Cargo.toml        Cargo workspace for the Rust CLI
packaging/npm/    npm binary bundler and bin launcher
scripts/          local npm publishing and skill format checks

Repository root:

skills/           Rainbo skills installable with `npx skills add`
skill-template/   Template for new Rainbo skills

Inspect the connected API contract

rainbo schema <METHOD> <PATH> reads the embedded build-time OpenAPI contract and returns schema_source: embedded. It does not prove the connected server supports the operation. To diagnose a 405 or a deployment mismatch, use:

rainbo schema POST /api/data-verify/diff-diagnosis/prepare --live

--live reads /api/docs/openapi.json with the selected environment's Bearer session, expands that document's references, and returns schema_source: server. It never falls back to the embedded contract when the server operation is missing. Check the environment and API deployment if it returns server_operation_unavailable; if the operation is advertised but requests still return 405, check proxy routing and API replica versions. No diagnostic task is created by this check.