@xdxer/dws-lark-bench

v0.1.13

Published

18 days ago

Benchmark harness for comparing DWS and Lark CLI in E2B sandboxes.

0High
0Medium
0Low

xiadongxiang

dingtalk dws lark feishu benchmark e2b cli agent-skill

dws-lark-bench

Compare DingTalk and Feishu/Lark head-to-head from the command line — same task, both platforms — and get one report telling you which actually got it done, and at what cost.

You hand dws-lark-bench a plain-language task (e.g. "read my latest chat with Xumo"). It runs that same task two ways — once through the DingTalk Workspace CLI (dws) and once through the Feishu/Lark CLI (lark-cli) — each in its own isolated E2B sandbox, driven by an OpenCode agent. Then it prints one Markdown report comparing completion status, token cost, and the evidence each side produced.

Reach for it when you want to answer "can DWS do X the way Lark can — and how much does each cost?" without hand-running both CLIs and eyeballing the difference.

The whole flow

install ─▶ doctor ─▶ auth login (dws + lark, once) ─▶ run <task> ─▶ report
           verify     authorize in a sandbox,           both CLIs,     done? cost?
           env+CLIs    save the credentials              concurrently   evidence?

The two targets always run in separate sandboxes, concurrently — neither slows the other down, and a failure on one side still leaves you a report for the other.

Quick Start

Install once, then run from anywhere:

npm i -g @xdxer/dws-lark-bench

dws-lark-bench init        # interactive: saves your keys to ~/.dws-lark-bench/.env
dws-lark-bench doctor      # verify env + local CLIs are ready
dws-lark-bench auth login --target dws    # one-time DingTalk auth (device flow in a sandbox)
dws-lark-bench auth login --target lark   # one-time Feishu/Lark auth (full authorization)
dws-lark-bench run \
  --id latest-chat-xumo \
  --text '阅读我和须莫最新的聊天内容，DWS 和 Lark 两边都输出结果。'

Don't want to install? Every command also runs straight from GitHub with npx:

npx --yes --prefer-online github:xdxer/dws-lark-bench doctor

init saves your config to a global ~/.dws-lark-bench/.env, so a new shell or a different directory just works. Precedence is exported shell vars → project ./.env → global ~/.dws-lark-bench/.env, so an export always wins.

Required Environment

Only two secrets are required by default:

export E2B_API_KEY=e2b_...
export DWS_LARK_BENCH_OPENAI_API_KEY=sk_...

OpenAI-compatible endpoint and model are configurable:

export DWS_LARK_BENCH_OPENAI_BASE_URL=https://api.openai.com/v1
export DWS_LARK_BENCH_OPENAI_MODEL=gpt-5

For DeepSeek-compatible testing:

export DWS_LARK_BENCH_OPENAI_BASE_URL=https://api.deepseek.com
export DWS_LARK_BENCH_OPENAI_MODEL=deepseek-v4-pro

Supported environment variables:

| name | required | purpose | | --- | --- | --- | | E2B_API_KEY | yes | Creates and connects to E2B sandboxes. | | DWS_LARK_BENCH_OPENAI_API_KEY | yes | Default model API key env consumed by OpenCode. | | DWS_LARK_BENCH_OPENAI_BASE_URL | no | OpenAI-compatible endpoint. Defaults to https://api.openai.com/v1. | | DWS_LARK_BENCH_OPENAI_MODEL | no | Model id. Defaults to gpt-5. | | DWS_LARK_BENCH_OPENAI_API_KEY_ENV | no | Alternative env var name for the model API key. | | DWS_LARK_BENCH_OPENAI_PROVIDER_ID | no | OpenCode provider id. Defaults to dws-lark-bench-openai. | | DWS_LARK_BENCH_E2B_TEMPLATE | no | E2B template name. Defaults to dws-lark-bench. | | DWS_LARK_BENCH_CONTROLLER_TOKEN | no | Optional bearer token for remote controller mode. |

Never commit real API keys or auth artifacts.

Doctor

doctor tells you whether you're ready to run:

dws-lark-bench doctor

It reports:

Runtime environment — E2B_API_KEY and your model config (DWS_LARK_BENCH_OPENAI_*), each shown only as set / missing, never the value;
Local CLIs — opencode, dws, lark-cli versions; these are needed only for run --local, since an E2B sandbox run carries them inside the sandbox image;
copy-ready next commands for whatever still needs setup.

A sandbox run needs just E2B_API_KEY plus your model key — nothing else installed locally. (Earlier versions also probed railway / supabase / gh; those checks were removed.)

Interactive Auth

Auth is done inside an E2B sandbox and then saved locally as an artifact:

.dws-lark-bench/auth/dws-auth.tar.gz
.dws-lark-bench/auth/lark-auth.tar.gz

These files are restored into later benchmark sandboxes before OpenCode runs.

DWS

npx github:xdxer/dws-lark-bench auth login --target dws

The command runs dws auth login --device --force inside E2B and prints the real DingTalk device URL and user code locally:

DingTalk authorization required

Open this URL to authorize DingTalk:
https://login.dingtalk.com/oauth2/device/verify.htm?user_code=XXXX-XXXX

Authorization code:
XXXX-XXXX

Open the URL, finish DingTalk authorization, and keep the command running. On success, the CLI exports and downloads .dws-lark-bench/auth/dws-auth.tar.gz.

Lark / Feishu

Simplest — just run it. Lark auth defaults to full authorization (lark-cli --domain all), so you don't pass any scope:

dws-lark-bench auth login --target lark

To restrict to specific read-only scopes (IM-only example):

dws-lark-bench auth login \
  --target lark \
  --scope 'im:message:readonly im:chat:read im:message.p2p_msg:get_as_user'

If the sandbox has no Lark app client_secret, the CLI starts lark-cli config init --new --brand feishu and prints the app configuration URL. Complete the app setup, then follow the user authorization URL. On success, the CLI downloads .dws-lark-bench/auth/lark-auth.tar.gz.

Reset & re-test

Wipe saved config/auth and start clean (handy for repeated testing):

dws-lark-bench reset          # remove saved .env + auth (asks to confirm)
dws-lark-bench reset --yes    # skip the prompt
dws-lark-bench reset --auth   # only auth artifacts (keep .env)
dws-lark-bench reset --env    # only the .env (keep auth)
dws-lark-bench reset --runs   # also remove this project's run outputs

If you run a command before configuring, the CLI now returns needs_onboarding with a pointer to dws-lark-bench init instead of a cryptic error.

Run Tasks

Run a checked-in task file:

npx github:xdxer/dws-lark-bench run benchmarks/tasks/latest-chat-xumo.yaml

Run an ad-hoc task:

npx github:xdxer/dws-lark-bench run \
  --id latest-chat-xumo \
  --text '阅读我和须莫最新的聊天内容，DWS 和 Lark 两边都输出结果。'

Local mode (no E2B)

Pass --local to execute opencode directly on your machine instead of in an E2B sandbox. No E2B_API_KEY is required; the run uses your host's own dws / lark-cli login and the model key. Each target's CLI must be installed locally — a target whose CLI is absent reports a tool_error.

dws-lark-bench run --local --text 'Read my latest chat' --id local-smoke
dws-lark-bench run --local --json benchmarks/tasks/selftest-20260624/01-im-latest-xumo.yaml

--local cannot be combined with --controller-url. Everything else (the report, the --json envelope, status classification) is identical to sandbox runs.

Task files are YAML:

id: latest-chat-xumo
name: Latest chat with Xumo
description: >
  Read the latest visible one-to-one chat between me and 须莫.
  Use only read-only commands.
risk: read-only
targets:
  - dws
  - lark
success_criteria:
  - type: exitCode
    value: 0
limits:
  timeoutMs: 420000
  retries: 0

Reports and raw artifacts are written under:

.dws-lark-bench/runs/<task-id>/
  report.json
  dws.stdout.jsonl
  dws.stderr.log
  dws.auth-restore.stdout.log
  dws.auth-restore.stderr.log
  lark.stdout.jsonl
  lark.stderr.log
  lark.auth-restore.stdout.log
  lark.auth-restore.stderr.log

Render a report again:

npx github:xdxer/dws-lark-bench report .dws-lark-bench/runs/latest-chat-xumo

Report Semantics

The Markdown report includes:

overall outcome;
completion status per target;
duration;
tool calls;
CLI invocations;
OpenCode invocations;
input/output/reasoning/total token usage;
artifacts;
evidence snippets.

Target statuses:

| status | meaning | | --- | --- | | passed | Target command exited successfully and did not report a known blocker. | | failed | Target command exited non-zero. | | blocked_by_auth | Output indicates missing/expired auth or required user authorization. | | blocked_by_permission | Output indicates missing permission/scope. | | timeout | Target command exceeded timeout or E2B request deadline. | | tool_error | Sandbox/SDK/tool failure that is not classified as timeout. | | partial | Reserved for future evaluators. |

The classifier is intentionally conservative, but it is still text-based. When investigating a surprising status, inspect the raw *.stdout.jsonl and *.stderr.log artifacts.

Selftest Suite

The repository includes 15 read-only regression tasks:

benchmarks/tasks/selftest-20260624/
  01-im-latest-xumo.yaml
  02-contact-profile-xumo.yaml
  03-calendar-upcoming.yaml
  04-task-todo-open.yaml
  05-approval-pending.yaml
  06-attendance-recent.yaml
  07-mail-inbox-recent.yaml
  08-meeting-records-recent.yaml
  09-drive-recent-files.yaml
  10-org-feed-or-notice.yaml
  11-doc-recent-docx.yaml
  12-doc-read-latest-docx.yaml
  13-doc-wiki-recent.yaml
  14-doc-sheet-summary.yaml
  15-doc-base-summary.yaml

Run one case:

npx github:xdxer/dws-lark-bench run benchmarks/tasks/selftest-20260624/01-im-latest-xumo.yaml

Run all cases sequentially from a local clone:

for task in benchmarks/tasks/selftest-20260624/*.yaml; do
  node bin/dws-lark-bench.js run "$task"
done

Known observations from real authorized runs:

IM, calendar, tasks, approvals, attendance, drive, docs, wiki, sheets, and base cases can pass on both sides when scopes and user resources exist.
02-contact-profile-xumo can hit long-tail timeout and needs more focused task design.
10-org-feed-or-notice is not exposed by current DWS MCP services; DingTalk raw OpenAPI requires separate AppKey/AppSecret credentials.
Lark mail may authenticate successfully but still report no mailbox if the tenant has not provisioned Lark Mail for the user.

Controller Mode

Controller mode lets one long-running service own E2B execution. A local or remote client submits tasks over HTTP.

Start a local controller:

npx github:xdxer/dws-lark-bench controller start --host 127.0.0.1 --port 4317

Submit a run:

npx github:xdxer/dws-lark-bench run \
  --controller-url http://127.0.0.1:4317 \
  --id latest-chat-xumo \
  --text '阅读我和须莫最新的聊天内容，DWS 和 Lark 两边都输出结果。'

Endpoints:

| endpoint | purpose | | --- | --- | | GET /health | Non-secret readiness: E2B, model, template, auth requirement, artifact storage. | | POST /runs | Submit a benchmark task. | | GET /runs/:id/artifacts | List run artifacts. | | GET /runs/:id/artifacts/:name | Download one artifact. |

For remote deployments, set the same bearer token on both sides:

export DWS_LARK_BENCH_CONTROLLER_TOKEN=bench-controller-secret

When local auth artifacts exist under .dws-lark-bench/auth/, run --controller-url sends them with the request so the remote controller can restore auth inside its E2B sandboxes. Use HTTPS for any non-local controller.

Railway deployment is supported by railway.json:

npm run start:controller

The start script binds to 0.0.0.0; controller start reads PORT when --port is omitted.

E2B Sandbox Image

The E2B template lives in sandbox/ and installs:

Node.js;
Git/curl/jq/tar/gzip/unzip;
opencode-ai;
@ai-sdk/openai-compatible;
@e2b/cli;
dingtalk-workspace-cli;
@larksuite/cli.

Build the template:

E2B_API_KEY=e2b_... npm run sandbox:build

The image does not contain API keys or auth artifacts. Auth artifacts are uploaded per run.

Local Development

npm install
npm test
npm run typecheck
npm run build

Useful local commands:

node bin/dws-lark-bench.js --help
node bin/dws-lark-bench.js doctor
node bin/dws-lark-bench.js sandbox smoke
node bin/dws-lark-bench.js run benchmarks/tasks/read-only-status.yaml

Before committing changes, run:

npm test
npm run typecheck
npm run build
git diff --check

Troubleshooting

`blocked_by_auth`

Re-run the relevant auth flow:

npx github:xdxer/dws-lark-bench auth login --target dws
npx github:xdxer/dws-lark-bench auth login --target lark --scope '<required scopes>'

DWS may report that the refresh token is valid while access-token refresh fails. Treat that as needing a fresh dws auth login.

`blocked_by_permission`

Check whether the Lark app has the required scopes and whether the user has granted them. For newly added scopes, re-run Lark auth.

`timeout`

The harness preserves a report even when one target times out. Inspect:

.dws-lark-bench/runs/<task-id>/<target>.stdout.jsonl
.dws-lark-bench/runs/<task-id>/<target>.stderr.log

Timeouts usually mean the task prompt is too broad, the CLI command path is unclear, or the underlying API is slow. Narrow the task or add a more specific case.

Report says `passed` but evidence looks blocked

The status classifier is text-based. Trust the raw evidence, then improve the classifier with a regression test.

Security Notes

Do not commit .env.
Do not commit .dws-lark-bench/auth/.
Do not commit .dws-lark-bench/runs/ if artifacts contain private content.
Do not print real E2B_API_KEY or model API keys in logs, docs, issues, or commits.
Remote controller mode should use HTTPS and DWS_LARK_BENCH_CONTROLLER_TOKEN.