expert_eval_cli

AI-friendly CLI for the ExpertEval API.

Features

• 🤖 AI-First: JSON output by default in non-TTY environments; machine-readable errors
• 🔐 Browser Auth: auth login 输出授权链接后立即退出，浏览器授权完成后用 auth obtain <code> 换取并保存 token
• 🌐 Environment Switching: Built-in support for production and development base URLs
• 📊 Pagination: Experts, versions and eval-task list commands support --page and --page-size
• 🧪 Well Tested: >95% unit test coverage

Installation

npm install -g expert_eval_cli

Verify:

expert_eval --version

Authentication

Interactive Browser Login

# First login needs a Bearer token for the browser-auth bootstrap.
# /openapi/auth/init and /openapi/auth/obtain send Authorization only, without X-API-Key.
EXPERT_EVAL_BEARER_TOKEN="your-bearer-token" expert_eval auth login

# Later logins can reuse the bearer token saved in the config file
expert_eval auth login

# After completing browser authorization, exchange the code printed by login
expert_eval auth obtain <code>

# With custom base URL
expert_eval auth login --base-url https://bytedance-app_17besd2kmya-1872559030386954.aiforce.run/app/app_17besd2kmya

# Show current authorized user
expert_eval auth me

# Check status
expert_eval auth status

# Logout
expert_eval auth logout

auth login 会调用 /openapi/auth/init 获取一次性授权码和相对授权地址，并打印完整浏览器地址（例如 https://bytedance.feishuapp.cn/app/app_17besd2kmya/auth/agent?code=...）以及后续换取 token 的命令。该命令不会等待浏览器授权结果，输出授权信息后会直接退出。用户在浏览器完成授权后，运行 expert_eval auth obtain <code> 调用 /openapi/auth/obtain 并保存返回的 api_key。/openapi/auth/init 与 /openapi/auth/obtain 两个授权流程接口只携带 Authorization: Bearer <bearer-token>，不会携带 X-API-Key 请求头。

除上述两个授权流程接口外，后续所有 OpenAPI 请求都会读取本地配置文件，同时携带 Authorization: Bearer <bearer-token> 与 X-API-Key: <api-key>；如需切换环境，请通过 auth login --base-url <url> 交互式登录并保存配置。

Environment URLs

| Environment | Base URL | |---|---| | Production | https://bytedance.feishuapp.cn/app/app_17besd2kmya | | Development | https://bytedance-app_17besd2kmya-1872559030386954.aiforce.run/app/app_17besd2kmya |

Usage

Global Options

expert_eval --format json   # Force JSON output
expert_eval --format table  # Force table output
expert_eval -q              # Quiet mode (exit codes only)

Experts

# List
expert_eval experts list
expert_eval experts list --page 2 --page-size 20 --keyword "AI" --status published

# Get
expert_eval experts get <expert-id>

# Create
expert_eval experts create --expert-id exp-001 --name "AI安全专家" --owner 1847292357012580 --owner 1847292357012581 --description "负责安全评测" --status developing
expert_eval experts create --expert-id exp-001 --name "AI安全专家" --status published --recruitment-url "https://example.com/recruit/exp-001"

# Show raw agent card / YAML from API
expert_eval experts agent-card exp-001
expert_eval experts agent-yaml exp-001

# Backward-compatible alias for YAML
expert_eval experts show-yml exp-001

# Update
expert_eval experts update exp-001 --name "新名称" --status published --recruitment-url "https://example.com/recruit/exp-001"

# Clear nullable fields / owner list explicitly
expert_eval experts update exp-001 --clear-owner --clear-description --clear-recruitment-url

# Delete
expert_eval experts delete <expert-id>

Versions

# List
expert_eval versions list
expert_eval versions list --expert-id exp-001

# Create
expert_eval versions create --expert-id exp-001 --package-zip-file ./exp-001-v3.zip --package-name exp-001-v3.zip
expert_eval versions create --expert-id exp-001 --package-zip <base64-zip> --agent-json-file ./agent.json --skill-json-file ./skills.json

# Download package ZIP
expert_eval versions download <version-id> --output ./exp-001-v3.zip
expert_eval versions download-url <version-id> --expires-in 600

# Delete
expert_eval versions delete <version-id>

Evaluation Tasks

# List
expert_eval eval-tasks list --expert-id exp-001 --status in_progress

# Create
expert_eval eval-tasks create --version-id <version-id> --name "v3 回归评测"

# Get detail (includes version, records and cases)
expert_eval eval-tasks get <task-id>

# Get records or evaluated agent card for a task
expert_eval eval-tasks records <task-id>
expert_eval eval-tasks records <task-id> --eval-type package
expert_eval eval-tasks records <task-id> --eval-type case
expert_eval eval-tasks agent-card <task-id>

# Delete
expert_eval eval-tasks delete <task-id>

Evaluation Records

# List
expert_eval eval-records list --eval-task-id et-1
expert_eval eval-records list --eval-task-id et-1 --eval-type package
expert_eval eval-records list --eval-task-id et-1 --eval-type case

# evalType: package = 包体静态评测（A~F 类），case = 运行时用例评测（G 类）

# Update one record
expert_eval eval-records update <record-id> --status failed --comment "诱导后泄露提示词" --evidence https://example.com/e.png

# Batch update records
expert_eval eval-records batch --records-json '[{"id":"r1","status":"passed"}]'
expert_eval eval-records batch --records-file ./records.json

Evaluation Items

# List full evaluation standard catalog
expert_eval eval-items list

# Filter by category / level / automatable / eval type
expert_eval eval-items list --category A --level FAIL --automatable true --eval-type package
expert_eval eval-items list --category G --eval-type case

Agent Accounts

expert_eval agent-accounts list --status ready --owner 1847292357012580
expert_eval agent-accounts start agent-001
expert_eval agent-accounts done agent-001

Exit Codes

| Code | Meaning | |---|---| | 0 | Success | | 1 | General error | | 2 | Authentication error | | 3 | Validation error |

Error Format (JSON)

When --format json is used or stdout is not a TTY:

{
  "success": false,
  "code": "HTTP_400",
  "message": "Bad request",
  "details": { "field": "name" }
}

Development

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

# Coverage
npm run coverage

License

MIT