@dooer/dooer-test-env

One CLI to run the entire Dooer backend locally in Docker — every staging service + frontend, wired together, backed by a customer-free copy of the staging database (reference data + anonymized users, no organizations). Copy specific customers in when you need them, spin up empty test accounts, search logs across the whole stack, and measure resource usage.

Runs on an Apple-Silicon Mac (images run under emulation) — 32 GB RAM / ~150 GB free disk is comfortable. See ENVIRONMENT-PLAN.md for the full design and rationale, and REVERT-POINT.md for the last version that works without service-dooer-test-env.

npx @dooer/dooer-test-env <command>

First time here?

You need exactly two things:

1. The VPN — everything this tool talks to lives inside the cluster network.
2. An admin session for staging — remote-environment login. That is what authorizes the tool to fetch the environment spec, the registry credentials and the BankID material.

No kubectl. No cluster credential. No new-infrastructure checkout. No npm account. All of that moved behind service-dooer-test-env, which runs in each environment and exposes only the operations this tool needs — each one admin-gated and recorded. See the specification for what it replaced and why.

npm access is needed for the next step, not this one. The CLI and every one of its dependencies are public, so npx works with no npm account. You need membership of the dooer organisation on npmjs.com as soon as you install a service's own dependencies — yarn install in a service checkout, or service local, which runs that checkout instead of the container image. Only Sam Nurmi can grant it; getting-access.md §3 has the table and the wording to ask with.

⚠️ CLI 1.16.0–1.19.1 got this wrong and failed for anyone outside the org with 404 … @dooer%2fpackage-info — a private dependency the CLI never used. Fixed in 1.19.2.

• Getting access — new developer: set up the VPN and log in.
• Logins and user management — signing in as a customer, partner or admin user; creating users; partner membership; using the token from scripts.
• Issuing access — admin: how to create and hand over a VPN tunnel. Still required — a new developer cannot reach anything without one.
• System architecture — how the layers fit (browser → public-facade → service-graphql → the 64 stitched services → Postgres/S3/task-engine), a domain→service reference, and a per-HQ-view breakdown of which services each view pulls data from, with diagrams.

Quick start

# 0. Log in to staging as an ADMIN — the local stack is built from staging's live Deployments.
#    BankID cannot reach an admin account; this has to be email + password.
npx @dooer/dooer-test-env remote-environment login --env staging --email [email protected]

# 1. One-time: prereqs, registry login, environment spec, compose generation, BankID cert → keychain.
#    Needs the VPN and the admin session above.
npx @dooer/dooer-test-env setup

# 2. Bring the whole stack up (all staging services + frontends)
npx @dooer/dooer-test-env up

# 3. Load the customer-free base DB (anonymized users + reference data, zero orgs)
npx @dooer/dooer-test-env db pull --execute

# 4. Status. HQ → http://localhost:8080, GraphQL gateway → http://localhost:4000
npx @dooer/dooer-test-env status

Commands

setup                                            prereqs + registry login + compose + BankID cert → keychain
up [--profile full|booking|hq]                   create + start the stack (default full = everything staging runs)
stop | start | down [--volumes]                  stop / restart / tear down (optionally wipe the local DB)
status                                            list services (and which are served from local code)
db pull --execute                                restore the newest customer-free base DB, then provision roles
db roles                                          (re)create per-service DB roles + search_path (db pull runs it)
db snapshot <name> | db rollback <name>          local restore points
db psql [-- psql args]                            open psql on the local DB (pipe SQL in; forwards args like -A -t -c)
customer new "<name>" --owner <userId> [--env <env>] [--org-number N]  empty functional account (company+Owner+subs+dooer), any env via GraphQL
customer copy | customer purge                    copy / delete one org between environments (emails anonymized)
remote-environment login|become|token|status|logout   sign in to an environment; session → keychain
user search | get | create | partners             find/inspect/create users (local env unless --env)
partner list | users | assign | unassign          partners and who belongs to them, at which level
partner create | team create | team add-user      create partners, teams, and team memberships (GraphQL, any env)
document upload <file> --org <id> --customer-user-id <id> [--customer-sort ...]  upload a doc AS a customer, like sumify
shred                                             anonymize the LOCAL db (localhost only)
bankid pull | status | clear                      BankID cert → keychain (alias for `secret … bankid`)
secret pull <name> | status | clear               pull a per-service secret's keys (bankid, org-cert, …) → keychain, injected at up
validation on | off | status                      output-schema validation (default OFF, like staging/live)
logs <pattern> [--since 5m] [--service x] [-i]    search logs across ALL services (e.g. a transactionId). alias: search
measure                                           sample docker stats → resource-usage report
service <start|stop|restart|deploy|local|unlocal|version> <name>   per-service control

Everything that writes is dry-run by default; add --execute. Run any command with --help.

Use cases

Everything below runs from any directory (npx resolves the published CLI). Nothing is tied to a particular machine — you cd to your own checkouts, and state lives in ~/.dooer-test-env.

Where things are served

| | | | --- | --- | | localhost:4000 | public-facade — the browser-facing entrypoint (GraphQL + REST passthrough) | | localhost:4001 | service-graphql (direct; frontends go via the facade) | | localhost:8080 | frontend-hq | | localhost:8081 | frontend-back-office-neue | | localhost:8081/tools/graphiql | ↳ its GraphiQL console — handy for running queries by hand | | localhost:8082 | frontend-billing-invoice | | localhost:8083 | frontend-booking-engine | | localhost:8084 | frontend-byra | | localhost:8085 | frontend-ai-data-studio | | localhost:8086 | frontend-microservice-documentation | | localhost:8087 | frontend-reconcile-engine | | localhost:8088 | frontend-sumify-neue | | localhost:8089 | frontend-tasks | | localhost:8090 | frontend-website | | localhost:8091 | frontend-xrays | | localhost:9001 | MinIO console (minioadmin / minioadmin) | | localhost:55432 | Postgres (dooer / dooer) | | localhost:19092 | Kafka/Redpanda, host listener (containers use kafka:9092) | | localhost:21000+ | every backend service, one host port each (so host processes can reach them) |

Frontend ports are fixed per name — adding or removing a frontend, or running one from a local checkout, no longer shifts the others, so these URLs stay put. Note back office's graphiql is at 8081/tools/graphiql. status still shows the live mapping. (Backend service ports, 21000+, remain positional — see status; a shift there only matters to service local.)

Spin up an empty, ready-to-use customer

# local (default)
npx @dooer/dooer-test-env@latest customer new "Nordvik Handel AB" \
  --owner <users_pk-of-a-CUSTOMER-user> --execute

# any environment you are logged in to — same command, add --env
npx @dooer/dooer-test-env@latest customer new "Nordvik Handel AB" \
  --owner <users_pk> --env staging --org-number 5560123456   # 10 digits → stored 165560123456 --execute

# place it under a specific partner AND one of that partner's teams
npx @dooer/dooer-test-env@latest customer new "Nordvik Handel AB" \
  --owner <users_pk> --env staging \
  --partner "Dooer Devteam" --partner-team "Team Nord" --execute

Runs entirely through the public GraphQL API, authorized by your session — the same path the product itself uses, so company-created / fiscal-year-created fire and service-closing generates the accounting periods. Works against any environment (--env local|staging|production|<namespace>); local falls back to a minted service token, so no login is needed there. It creates: company + Owner + address, the Ghost-Inspector subscription set, the partner link (dooer by default — see --partner/--partner-team below), Terms of Service pre-accepted as the owner, the current-year fiscal year, and the temporal keys the platform reads (fiscalYear, hasVatRegistration, hasCompanyTax, hasEmployeeRegistration, bookkeepingMethod, vatPeriod/vatDue).

• --org-number sets the organization number (10 or 12 digits, spaces/dashes ok). A limited company's canonical form is 12-digit with the 16 legal-entity prefix, so a bare 10-digit 556… is stored as 16556…; omit the flag for a random 16556XXXXXXX placeholder.
• --partner chooses which partner the org is placed under — by domain, name, or id (default dooer).
• --partner-team additionally assigns the org to a team inside that partner — by team name or id. Omit it and the org lands in the partner's default team. A bad partner/team name fails before anything is created (it is resolved up front, even on a dry-run), and the error lists the partner's real team names.
• --no-fiscal-year leaves the fiscal year, registration fields and temporal keys blank.
• --no-task-delay sets the org setting noTaskDelays so the deliberate hi-task delays are skipped for this customer — a document normally waits a day and a transaction a week (so the matching bank transaction can sync), which makes a manual test wait for nothing. Handy for a throwaway test account you want to book immediately. (Read-merge-write of the settings object, so the other settings keep their defaults.)
• The owner must be a customer user — TOS is accepted on their behalf (only the Owner may accept it); a hi/admin owner gets a warning and no TOS acceptance.

The owner must be a customer user — a hi/admin owner cannot accept Terms of Service and the command warns you. Find one:

npx @dooer/dooer-test-env@latest user search --role customer

Create a user you can actually log in as

# an accounting professional (role `hi`) in the local env, password generated and printed once
npx @dooer/dooer-test-env@latest user create --email [email protected] --execute

# …a customer instead, with your own password and a personnummer
npx @dooer/dooer-test-env@latest user create --email [email protected] \
  --role customer --password 'hunter2hunter2' --personnummer 19900101-1239 --execute

# …or an `hi` user placed straight into a partner
npx @dooer/dooer-test-env@latest user create --email [email protected] \
  --partner "Dooer Devteam" --level admin --execute

This goes through users.create — the same mutation back-office's /users/new calls — so the service hashes the password itself and applies its own input validation. The password works on the ordinary email+password login, no BankID needed, and is shown once; there is no way to read it back. --role defaults to hi (other choices: customer, admin) and names default to something derived from the address.

The personnummer must be a real one: the API checks the Luhn digit, so an invented number is rejected (the CLI checks it first and tells you which rule failed). Creating a user needs an admin session in a remote environment — see below.

Look users up by email or id, and see where they belong:

npx @dooer/dooer-test-env@latest user search anna          # email, name, personnummer or id
npx @dooer/dooer-test-env@latest user get [email protected]   # details + partner memberships

Partners and who belongs to them

npx @dooer/dooer-test-env@latest partner list                        # every partner in the environment
npx @dooer/dooer-test-env@latest partner users "Dooer Devteam"       # its users and their level

# add / promote / remove (a partner is addressable by id, name or domain)
npx @dooer/dooer-test-env@latest partner assign [email protected] "Dooer Devteam" --level admin --execute
npx @dooer/dooer-test-env@latest partner unassign [email protected] "Dooer Devteam" --execute

--level is the membership level inside the partner (member or admin) — not the same thing as the user's --role at Dooer. Only hi users can belong to a partner; the CLI applies the same invalid-user-type check the service-accounts API does rather than writing a row HQ would choke on.

Create partners and teams

# a brand-new partner (accounting firm) — domain must be unique
npx @dooer/dooer-test-env@latest partner create "Acme Revision" \
  --domain acme --email [email protected] --env staging --execute

# a team inside a partner (partner by id, name or domain)
npx @dooer/dooer-test-env@latest partner team create "Team Nord" \
  --partner acme --tags region-nord --env staging --execute

# add a user to that team (membership). --assign also adds them to the partner as member if they aren't yet
npx @dooer/dooer-test-env@latest partner team add-user [email protected] \
  --partner acme --team "Team Nord" --assign --env staging --execute

All three are dry-run by default (--execute to apply) and run through the public GraphQL API, so they work against any environment. add-user presupposes partner membership — it errors with a hint unless the user is already a member or you pass --assign (which adds them as member first, hi-users only).

Upload a document — as a customer, exactly like sumify

npx @dooer/dooer-test-env@latest document upload ./receipt.pdf \
  --org <organizationId> \
  --customer-user-id <a-CUSTOMER-users_pk> \
  --customer-sort expense \
  --env local --execute

The upload is performed as the customer: the CLI “becomes” --customer-user-id (the same user(id){become} back-office/HQ use) and uploads with their session — so service-core-objects records a genuine customer upload, sets the document's creator, and derives customerSortedAs the way it does for a real sumify upload. (--customer-user-id must be a customer user; a service/hi session would record a non-customer upload with customerSortedAs null, which is not the same thing.)

--customer-sort is the customer's own sort — a DocumentSortOption (unsorted | income | expense | reimbursement | other), the exact buckets sumify offers (defaults to unsorted). The server derives the richer customerSortedAs/partnerSortedAs (supplierInvoice, receipt, salesInvoice, …) from it — e.g. --customer-sort expense lands as supplierInvoice. Dry-run by default. Duplicate files are rejected server-side (document-exists), same as the product.

Look at (or fix) users in a real environment

The user and partner commands reach an environment through its public GraphQL endpoint — the same one the frontends use — so staging and live need no VPN, no kubectl and no cluster credentials. They do need a session:

# BankID — what HQ does, so you get your professional (hi) account
npx @dooer/dooer-test-env@latest remote-environment login --env staging

# email + password — the only route to an ADMIN account (BankID cannot reach one)
npx @dooer/dooer-test-env@latest remote-environment login --env staging --email [email protected]

npx @dooer/dooer-test-env@latest remote-environment status --all

Then point any command at it with --env (local is the default; staging / production and aliases like dooer-staging / live resolve too):

npx @dooer/dooer-test-env@latest user search [email protected] --env staging
npx @dooer/dooer-test-env@latest partner list --env production

Which of your accounts you sign in as decides what you can do — reads work from any session, partner membership needs a partner session, and creating a user needs an admin one. See Logins and user management for the full picture: the three kinds of user, become, and the permission matrix.

Use the session token in your own scripts

remote-environment token prints the token and nothing else — no colour, no label — so it drops straight into a variable or a pipe:

TOKEN=$(npx @dooer/dooer-test-env@latest remote-environment token --env staging)

curl -s https://api.s-e032.com/graphql \
  -H "content-type: application/json" -H "authorization: Bearer $TOKEN" \
  -d '{"query":"{ currentUser { id email userType } }"}'

The local env needs no login for this — with no stored session it mints one from the shared dev keypair.

Copy a real customer into the local env

--target-local addresses this stack; the source can be any environment (--source-local goes the other way). Emails are always anonymized. Dry-run by default; add --execute.

You need an admin session in BOTH ends. A copy is two locally-authorized halves — an export authorized by admin in the source and an import authorized by admin in the target — because each environment signs its own tokens and neither service will ever validate the other's. That is a stronger check than the kubeconfig it replaces, where reaching the cluster was enough. The command is unchanged; the split is internal.

npx @dooer/dooer-test-env@latest remote-environment login --env production --email [email protected]
npx @dooer/dooer-test-env@latest remote-environment login --env staging --email [email protected]

Both halves are async, so the command prints progress while it waits. If the source could not read a table, the copy says so as a WARNING rather than quietly producing a partial org, and both ends record the operation in their audit trail (remote-environment → the service's /v1/operations).

# copy Ghost Inspector out of LIVE into the local env, as a new org owned by a local user
npx @dooer/dooer-test-env@latest customer copy \
  --source e45a3bc4-1b61-4f55-9bd6-de2705420cc2 \
  --source-namespace dooer-production \
  --target-local \
  --name "Ghost inspector (live)" \
  --owner-user <local-customer-users_pk> \
  --execute

# optional: shred the rest of the PII afterwards (localhost only)
npx @dooer/dooer-test-env@latest shred --execute

# undo: delete that org's rows again — dry-run first (omit --execute)
npx @dooer/dooer-test-env@latest customer purge --org <orgId> --local --execute

What the purge does not do: it deletes rows, not S3 objects. A purged org leaves its uploaded documents in the bucket; the command says so in its own output. Removing them needs bucket credentials the service does not hold today.

Referenced rows come with the org. A copied row can point at a row in a table the schema map does not copy: service_workflow.run is org-scoped and copied, but the workflow definition it points at is global and is not. The target usually has those already — every environment's base DB descends from the same production dump — but a base DB is a point-in-time snapshot while the source keeps moving. Production has 819 workflow definitions; a base DB built in April 2023 has 689, so a 2026 org's runs pointed at versions that were not there.

So the export carries the exact parent rows the org's rows reach, transitively (a carried definition points at a workflow in turn), and the import inserts only what is missing:

  referenced rows carried: 9 from 2 table(s), 8 inserted (the rest the target already had)

Their ids are not remapped — the copied rows reference them verbatim.

Two tables are never carried: service_accounts.users (it would move real names and personnummer from the source environment into the target — exactly what a live→staging copy must not do) and service_accounts.companies (the organization itself). The closure is capped at 50,000 rows, and both the cap and any composite-key foreign key it declines to follow are reported as warnings.

Anything that still dangles is reported after the load, because the bulk load runs with foreign-key enforcement off and Postgres never re-checks afterwards — so a dangling row would otherwise commit silently:

  44 row(s) still reference a parent that is not there:
      42  service_workflow.run.definitionId → service_workflow.definition

Naming: with neither --name nor --target the copy creates a new org that keeps the source org's own name (its short_name still gets a unique suffix). Pass --name to rename it, or --target <uuid> to copy into an org that already exists.

Many orgs at once — --sourcefile takes a file with one org uuid per line (# comments and blank lines ignored, duplicates collapsed). Each org is copied in turn and keeps its own name, so --name and --target (which describe a single target) are rejected with it:

cat > orgs.txt <<'IDS'
e45a3bc4-1b61-4f55-9bd6-de2705420cc2
0391c6ba-2c41-458b-95d2-a2d555117833
IDS

npx @dooer/dooer-test-env@latest customer copy \
  --sourcefile orgs.txt \
  --source-namespace dooer-production \
  --target-local \
  --owner-user <local-customer-users_pk> \
  --execute

One failure does not abandon the batch: failures are collected and summarised at the end (and the command exits non-zero).

It scans all ~336 tables in the source before writing, so expect a few minutes per org on a production source.

⚠ Noisy neighbour when the source is live. A bulk export reads the org and every row it references straight out of the source database in one pass — across all ~336 tables, per org. With --source-namespace dooer-production that read runs against the live production database and competes with real customer traffic; a --sourcefile batch multiplies it by the number of orgs. The command prints this warning whenever the source is production. Prefer staging as the source, or copy a large batch during off-peak hours. (Reading from staging or local carries no such risk.)

Run your own code — one service, or several at once

service local stops that service's container, starts yarn start from your checkout, and repoints the discovery router so containers call your process. Your process reaches the containerized services through their published host ports, so traffic flows both ways.

cd to your checkout — wherever you keep it — and run it there. The service name is read from that repo's package.json, and it runs in the foreground with its logs streaming to your terminal; Ctrl-C stops it and restores the container.

cd ~/wherever/service-ledger
npx @dooer/dooer-test-env@latest service local          # name inferred from package.json

# or name/point it explicitly from anywhere
npx @dooer/dooer-test-env@latest service local service-ledger ~/wherever/service-ledger
npx @dooer/dooer-test-env@latest service version service-ledger   # LOCAL code, or which image?
npx @dooer/dooer-test-env@latest service unlocal service-ledger   # back to the container image

Several services at once — e.g. test a gateway change together with a backend change, no image builds:

Each one holds its own terminal, so use two:

# terminal 1 — the backend, started FIRST
cd ~/wherever/service-accounts && npx @dooer/dooer-test-env@latest service local

# terminal 2 — the gateway
cd ~/wherever/service-graphql  && npx @dooer/dooer-test-env@latest service local

Order matters: a local process learns its peers' addresses at startup, so start the service being called before the one calling it (or restart the caller afterwards).

Now a single query through the facade spans everything — browser → public-facade (Docker) → service-graphql (your code) → service-accounts (your code) and service-core-objects (Docker):

query VerifyLocalCode {
  organization(id: "<orgId>") {
    id
    documents(first: 1) { totalCount }   # service-core-objects, in Docker
  }
}

Try it in back-office's GraphiQL at http://localhost:8081/tools/graphiql, or:

curl -s -X POST http://localhost:4000/graphql -H 'Content-Type: application/json' \
  -H "Authorization: Bearer <token>" \
  --data '{"query":"query { organization(id:\"<orgId>\") { id documents(first:1){ totalCount } } }"}'

Test a published build before it ships

npx @dooer/dooer-test-env@latest service deploy service-graphql 66.34.2   # pin an image tag

Pins it in docker-compose.override.yml and keeps DOOER_SERVICE_VERSION in sync so logs report the version actually running.

Set a persistent env var on a service

For a service whose behaviour is gated by an env var (e.g. a feature that defaults off):

npx @dooer/dooer-test-env@latest service env service-institution-browser MY_FEATURE_ENABLED=true
npx @dooer/dooer-test-env@latest service env service-institution-browser            # list its overrides
npx @dooer/dooer-test-env@latest service env service-institution-browser MY_FEATURE_ENABLED=   # unset

The override is kept in ~/.dooer-test-env/state.json and re-derived into docker-compose.override.yml, so it survives compose regeneration (unlike hand-editing the compose file) and stacks with any service deploy pin on the same service. Setting or unsetting recreates the container so it takes effect (a plain restart wouldn't — Docker fixes env at container creation).

The same override also reaches a service run from a checkout with service local — the host process picks it up from state on start (it wins over the compose value and the forced local infra defaults), so a feature you toggle on with service env is on in both container and local mode.

Find out what broke

npx @dooer/dooer-test-env@latest logs <transactionId>          # across ALL services at once
npx @dooer/dooer-test-env@latest logs "error" --service ledger --since 30m -C 2

Output-schema validation (off by default)

Off means the local env behaves like staging/live, where these checks are silent. Turn it on to hunt stale service definitions — a consumer whose bundled definition predates a producer's added fields returns 500 on perfectly valid data:

npx @dooer/dooer-test-env@latest validation on      # then reproduce; drift shows up as 500s
npx @dooer/dooer-test-env@latest validation status
npx @dooer/dooer-test-env@latest validation off

Inbound input validation always runs — this only affects response validation.

How it works

• Compose generator (lib/compose/): translates the s-e032 k8s manifests → a docker-compose spec — images verbatim from the two registries; DB/Redis/Kafka/S3 and secrets repointed at local infra.

• Discovery router (discovery-router/): answers the Consul health + session/KV API (single-node leader election), so @dooer/service discovery and doode leadership work with no Consul. service local/deploy repoint entries so peers pick up changes with no restarts.

• Auth is self-contained: the shared local dev keypair is forced on every verifier; service-accounts signs with the matching key; per-service tokens are freshly minted with it (the baked config ones expired).

• Per-service DB roles: db pull recreates each service's role + schema search_path (mirroring cli-cowboy's setup) so unqualified queries resolve — the base dump carries no cluster-global roles.

• BankID: setup/bankid pull copy the PFX from the staging secret into the macOS keychain; up injects it so local login runs production BankID like staging.

• Base DB: built in-cluster (a k8s Job with its own scratch Postgres) — the 685 GB staging DB never leaves the cluster. It schema-restores, full-copies only non-org reference tables + users, shreds + PII-audits, and uploads the artifact to an OBC bucket; devs only db pull. The CronJob + OBC manifests live with the other staging yaml in new-infrastructure/kubernetes/environments/s-e032-onprem/.

• Log caps: every container is capped at 10m x 3 json-file logs. Unbounded logging once grew the discovery-router's log to 179 GB and filled the host disk.

Safety

• shred from a dev machine is localhost-only.
• customer copy/purge and db build writing to dooer-production require --confirm-production.
• Customer emails are always anonymized (testcustomer+<id>@dooer.com).

Prerequisites

Docker (Desktop), the VPN, an admin session for staging, and Node 18+.

No kubectl, no cluster certificate, no new-infrastructure checkout — service-dooer-test-env supplies what those used to. Don't have the VPN yet? → Getting access.

DOOER_TEST_ENV_MANIFESTS=<dir> still generates from local manifest yaml instead of the live cluster, which is useful when you are working on a manifest that is not deployed yet.