@simon-says-build/harbormaster
v0.22.0
Published
A localhost daemon that leases each caller its own port block — isolated Firebase emulator sandboxes, Metro/Expo ports, private iOS simulators — reaped when the caller exits. No port collisions, no shared instances, no pkill.
Maintainers
Readme
harbormaster
A localhost daemon that leases each caller its own isolated port block — a Firebase emulator sandbox with fresh data, a Metro port, a private simulator — and reclaims it when the caller's process exits.
If you run several Firebase projects locally (dev, tests, CI, agents) you know the
pain: emulators fight over ports, a bare firebase emulators:start collides with
another suite, and pkill "fixes" it by nuking everyone's emulators. harbormaster
removes the collision entirely — two callers that both want "an impulse emulator" get
two independent suites (own ports, own data), so they never step on each other.
$ eval "$(harbormaster acquire --cwd ./my-app --only auth,firestore --env)"
$ echo $FIRESTORE_EMULATOR_HOST
127.0.0.1:11001
# ...run your tests against the suite...
# it's torn down automatically when this shell exitsRequirements
- Python 3.8+ — the daemon is a single Python script (
#!/usr/bin/env python3). npm just ships and links the executable; it must be able to findpython3on PATH. - firebase-tools (
firebase) on PATH — it boots the actual emulators. - macOS or Linux.
Install
npm install -g harbormaster # global CLI
# or add it to a project so it ships with your app/agent:
npm install harbormaster # → node_modules/.bin/harbormasterUsage
The daemon auto-starts on the first command. firebase.json ports are optional —
the daemon injects (and overrides) a pool block into a temp config per lease, so you
never hand-assign a port.
# Boot a private suite for the project in DIR, owned by the calling shell,
# reaped when it exits. --env prints exports to eval into your shell:
eval "$(harbormaster acquire --cwd ./my-app --only auth,firestore,functions --env)"
# → FIRESTORE_EMULATOR_HOST, FIREBASE_AUTH_EMULATOR_HOST, EMULATOR_<SVC>_PORT,
# GCLOUD_PROJECT (from .firebaserc)
# Run a command inside a fresh suite that tears down after (great for CI):
harbormaster exec --cwd ./my-app --only auth,firestore "npm test"
# Reserve a block WITHOUT booting — for a caller that starts its own suite:
harbormaster allocate --json # → { "leaseId": "...", "base": 11000 }
# Have the DAEMON run a long-lived process (Metro, a dev server) as the lease:
# daemon-owned, so it survives the calling shell / agent session. Liveness is the
# spawned pid (+ TTL backstop); stop it with `release`. CMD sees METRO_PORT,
# DETOX_UDID/DETOX_SIM_NAME (with --sim), FBPORTS_BLOCK_BASE, FBPORTS_LEASE_ID.
eval "$(harbormaster allocate --sim --ttl 604800 \
--spawn 'exec npx expo start --port "$METRO_PORT"' --cwd ./my-app --env)"
harbormaster release "$FBPORTS_LEASE_ID" # tears down Metro + the sim
# Reuse gate for long-lived sessions: exit 0 iff the lease is FULLY alive
# (known to the daemon, process running, sim still present, ports up) — anything
# less means clean up your session state and re-lease:
harbormaster verify "$FBPORTS_LEASE_ID" || re_lease
harbormaster status # everything the daemon has out
harbormaster down --cwd ./my-app # release this shell's lease early--json on acquire/allocate returns the ports/base as JSON for scripting.
Version skew is self-healing: the daemon reports its version on /health, and a
newer client replaces an older daemon in place (leases persist in the ledger;
every leased process runs in its own session, so nothing is torn down by the
handover). A stale node_modules copy can no longer pin the machine to an old
daemon.
Leased simulators are part of a lease's liveness: if a sim is deleted externally
(Xcode cleanup, simctl delete, a device reset), the reaper notices within
FBPORTS_SIM_CHECK_INTERVAL (60s) and reaps the lease whole — killing its
spawned process too — so a half-alive session (sim gone, Metro still squatting
its port) can never linger for callers to "reuse".
How it works
- Caller-owned. One primitive:
acquire {cwd, owner_pid} → {ports, projectId}. Issuance is unique per caller, not per project — so concurrent callers of the same project get independent suites. - Reaped on exit. The daemon kills the suite (the whole process group, so the JVM
children die too) when
owner_piddies. The caller's death is the release — nothing to remember, and never a reason topkill. A TTL and emulator-liveness are backstops. - Pool ports. Blocks of 20 from 11000–14999. Every port a suite binds is on its block, including the ones firebase-tools starts unasked (see "Every port a suite binds is on its block"); offsets 12–14 are left for the caller.
- No registry, no shared instances. A "project" is just the
firebase.jsonincwd; the project id comes from.firebaserc.
See DESIGN.md for the full architecture.
License
MIT — see LICENSE.
The daemon's environment is nobody's test environment
The daemon is started by whichever client first needs it and would otherwise
inherit that shell. Suites it boots inherit the daemon's environment (that is
how --fn-env reaches the functions emulator runtime), so a variable exported
by the first caller would reach every later caller's suite for the life of the
daemon. Harbormaster therefore strips test-only and emulator-pointing variables
(TEST_NOW_MS, LLM_CACHE*, *_EMULATOR_HOST, *_EMULATOR_PORT,
GCLOUD_PROJECT, and the values a lease hands out) both when spawning the
daemon and when it starts serving. A suite that needs one of these gets it
explicitly: acquire --fn-env TEST_NOW_MS=....
A block docker holds is not free (0.10.0)
Free-block selection also asks docker ps which host ports are published. On
Linux, dockerd publishes a container's ports through iptables and, without the
userland proxy, nothing listens on them, so a connect finds the port quiet
while a container holds it. Simon Says' local Supabase stacks are containers
on harbormaster blocks. On simon-linux-1 (2026-10-09), four jobs were handed a
block a stack still held. No docker means the empty set, never a failure.
python3 test/docker_ports.py.
A managed broker (0.11.0)
harbormaster serve --managed is for a broker that a service manager
(systemd, launchd) runs and restarts. It keeps a marker in the state dir
fresh, touched every minute. While the marker is fresh, clients never replace
the broker, even an older one, and never spawn their own. If it is briefly
down, they wait up to 2 minutes for the service manager to bring it back. Upgrades
are the service manager's job: restart it after installing a new version. A
stale marker means the service is gone, and clients spawn as before. Without
this, the broker belonged to whichever client happened to start it: on
simon-linux-1 that was a CI job, whose orphan cleanup killed the broker and
every emulator it ran for other jobs mid-suite. python3 test/managed.py.
A project's own suite env (0.12.0)
--fn-env is per caller, so a setting every suite of a project needs had to
be repeated by every caller (session tests, e2e, agent runs), and the one that
forgot ran against production. A project now names it once, in
harbormaster.env next to its firebase.json:
# every emulator suite of this project runs under the TEST service account
GOOGLE_APPLICATION_CREDENTIALS=./impulse-test-bq.jsonKEY=VALUE lines, # comments. A value starting with ./ or ../ is made
absolute against the project, since the emulator runs functions (and
extensions) in other directories. acquire: on top of the daemon's scrubbed
env, under the caller's explicit --fn-env. exec: on top of the caller's
ambient env, so a key exported in some shell can't leak into the suite.
python3 test/project_env.py.
A function env value can name a leased port (0.14.2)
--fn-env KEY={port:firestore}, or the same in harbormaster.env, becomes
this lease's firestore port, inside a value too (127.0.0.1:{port:auth}). A
caller can't know its ports before the lease, and an app whose functions read
their own variable for the emulator (Tapuat's EMULATOR_PORT_FIRESTORE ||
"8080") otherwise talks to whatever holds the default. An unknown service is
left as written. python3 test/port_env.py.
Every port a suite binds is on its block (0.15.0)
firebase-tools starts more than firebase.json declares: the hub always; the
UI whenever a UI-supported emulator runs (declared or not) and logging with
it; eventarc and tasks whenever functions has a backend (an extension counts);
firestore's websocket (9150) and dataconnect's postgres (5432). Any port it is
not given it takes from the default and, if that is held, probes upward —
check-then-bind, so two suites booting in the same seconds pick the same port
and the second dies with "An unexpected error has occurred." (SimonSays' Mac
Studio, 2026-10-21: logging 4501 and 4502 taken twice; a walk2gether-expo CI
suite holding tasks 9501.) The injected config now sets all of them, plus
hosting, apphosting and dataconnect wherever the project uses them:
| +0 auth | +5 hub | +10 firestore websocket | +15 hosting | |---|---|---|---| | +1 firestore | +6 logging | +11 Metro | +16 apphosting | | +2 functions | +7 pubsub | +12 caller | +17 dataconnect | | +3 storage | +8 eventarc | +13 caller | +18 dataconnect postgres | | +4 ui | +9 database | +14 caller | +19 tasks |
+12..+14 are never bound by a suite. A declared emulator with no offset is
refused by name instead of left to its default. Teardown and the world sweep
already cover the whole block. python3 test/injected_config.py; test/e2e.sh
checks a live suite's listeners.
An Android emulator's ports are a lease (0.15.0)
acquire --android / allocate --android also lease an emulator console/adb
pair: an even console port in 5554..5584 (the range adb discovers; adb =
console + 1; FBPORTS_ANDROID_CONSOLE_FIRST/_LAST move it), unique among
live leases and free when handed out. The caller (and a --spawn process)
gets ANDROID_CONSOLE_PORT, ANDROID_ADB_PORT, ANDROID_SERIAL=emulator-<console>:
harbormaster allocate --android --owner auto \
--spawn 'exec emulator -avd Pixel -port "$ANDROID_CONSOLE_PORT" -read-only -no-window' --envThe pair is outside the pool, where a hand-booted emulator lands by default, so
teardown kills only what the lease spawned (its group and recorded tree) and
reports anything else still on the pair; the world sweep reports unleased
listeners on the range and evicts only what it can attribute.
python3 test/android_ports.py.
A lease never reads the hosting config it doesn't start (0.14.1)
A lease whose --only leaves hosting out boots from a config with no
hosting block. firebase-tools 15.33 reads hosting even when it starts no
hosting emulator, and a web-framework target ("source": ".", Next.js)
makes it refuse every emulator without the webframeworks experiment.
Tapuat's lease on simon-linux-1 died there on 2026-10-21; 15.28 on the
studio never checked. An --only that asks for hosting, or no --only,
keeps it. python3 test/injected_config.py.
An idle simulator nobody leases is shut down (0.13.0)
The world sweep (every FBPORTS_SWEEP_INTERVAL, 5 min) now shuts down a
booted simulator that meets all three conditions:
- no lease holds it;
- booted longer than
FBPORTS_IDLE_SIM_HOURS(default 6;0turns this off), measured by itslaunchd_sim; - no app is running in it.
It is shut down only, never deleted, so whoever booted it can boot it again.
On 2026-10-20 "Impulse-agent" had sat on the Mac Studio for 32 hours, booted outside any lease by an app's pinned simulator name, holding two of its ten cores with no app installed. Leases stopped only what they created.
status reports what the sweep last saw and did: idleSims {booted, unleased,
shutDown[]}. The CLI prints a simulators: line. Tested by
python3 test/idle_sims.py.
The box's simulator budget holds for every caller (0.22.0)
harbormaster makes every private simulator on the box, so a budget set here
holds for every caller: SSB's runs and rehearsals, CI's Detox suites, and a
suite run by hand. Set it with FBPORTS_MAX_SIMS, or with sim-budget.json
({"max": N}) in the state dir. SSB's host agent writes that file from the
host's maxSimulators. With no budget set, or one that can't be read, there is
no gate, because an unknown budget is not "full".
What counts: every booted simulator, leased or not, plus leases still making theirs. A leftover or someone's own device takes the same cores as a leased one. A leased device that is shut down costs nothing and does not count.
When there's no room: an acquire/allocate with --sim that would pass
the budget gets 503 {"wait": "simulators 4 of 4 in use on this box"} right
away, before anything is made. The client prints waiting for a simulator: …
on stderr and asks again every FBPORTS_SIM_WAIT_POLL seconds (default 15).
If FBPORTS_SIM_WAIT seconds pass (default 1800), it exits 75 (EX_TEMPFAIL,
"try later"), which SSB re-queues. Leases without a simulator are never gated.
On 2026-11-06 the Mac Studio ran 6–7 simulators beside their emulator stacks,
with CPU at 75–98% all day and swap 98% full, and simctl install/addmedia
stalled for minutes. SSB's scheduler had a budget, but CI and hand-run suites
had none.
status reports simBudget {max, waits}. Tested by python3 test/sim_budget.py.
An idb companion whose simulator is gone is ended (0.14.0)
idb starts one idb_companion --udid <UDID> per simulator it drives, with a
socket at /tmp/idb/<UDID>_companion.sock. A lease deletes its simulator, but
nothing ended that simulator's companion. On 2026-10-20 the Mac Studio had 19
companions running for devices that no longer existed, some from Oct 19, and
109 stale sockets. (A wedged companion also cost a walk2gether run a phone the
same day; the driver now restarts one that refuses, simon-says #1782.)
The world sweep now ends a companion that meets both conditions:
- its UDID is not in
simctl list, read again just before ending it; - it is older than 10 minutes, so a device created after the inventory read keeps its companion.
It also removes a socket for a gone device when no companion holds it and the
file is older than 10 minutes. Nothing else in the directory is touched
(FBPORTS_IDB_DIR, default /tmp/idb). When simctl list fails or comes back
empty, the sweep skips this step: an unreadable inventory never counts as
"every device is gone".
status reports companions {running, ended, socketsRemoved}, and the CLI
prints an idb companions: line when the last sweep ended or removed anything.
Tested by python3 test/companions.py.
A live owner's block survives its suite restarting (0.15.1)
A reserved block whose caller boots its own suite is reaped when its ports stay
down — after 45s with no owner, but after FBPORTS_OWNED_RECLAIM_GRACE (900s)
when its owner is alive. On 2026-10-22 a resumed story's suite was dark for over
45s while it re-imported its checkpoint; the block was reaped under the live
daemon that owned it, re-leased, and the next run's port sweep killed the
story's emulators mid-scene. The owner's death still reaps at once. Tested by
python3 test/owned_grace.py.
A run takes its block over from the daemon that leased it (0.16.0)
harbormaster reown <leaseId> [--owner PID] (POST /reown) hands a live lease
to another live owner. A daemon leases a run's block before spawning the run, so
the daemon owned it; on 2026-10-22 a fleet restart mid-story let the next daemon
adopt the run, but the block was reaped the moment the old daemon exited and the
port sweep killed the story's emulators in scene 3. The run now reowns its block
at startup, so the block lives exactly as long as the run. Tested by
python3 test/reown.py.
An allocate never waits behind a teardown (0.17.0)
The broker's lock now guards only the ledger. Teardown (group kills, simctl
delete, temp dirs), the world sweep, a suite's boot and a simulator's create
all run outside it, so allocate answers in milliseconds however busy the
reaper is. A block being torn down is draining (status lists them) and
returns to the pool when its teardown ends. If the caller closes its
connection before the grant (its own timeout fired), the broker refuses the
grant and tears down whatever it made, instead of reserving a block nobody
will release. On 2026-10-22 the studio's allocates queued behind the reaper
past the SSB daemon's 30s timeout, seven times, and each was granted anyway.
Tested by python3 test/teardown_lock.py.
A broker that loses the port touches nothing (0.17.1)
serve binds the port before it builds the broker. Building the broker loads
the ledger, reaps dead leases (deleting their simulators) and saves, so until
0.17.1 every client that raced to start one after a reboot did all of that,
then died on "Address already in use" beside the broker that won. On the
studio (2026-10-23) those losers' writes corrupted the ledger, the winner
quarantined it, and its world sweep deleted four simulators that live runs had
just leased. The port is now the broker's mutex: a loser exits before touching
the ledger or the managed marker. The ledger and sims-registry temp files are
per writer, so no two writers can interleave into one.
python3 test/serve_bind_first.py.
With a managed service installed, a client never starts its own broker (0.18.0)
The managed marker says a managed broker is serving now, and goes stale
whenever the service is down. On the studio (2026-10-23, after a reboot) a
client found it stale and started a plain serve. That broker held port 4999
from then on, and launchd's managed broker crash-looped behind it for 50
minutes. serve --managed --unit <path> now records the service's unit file
(managed-service.json in the state dir). While that file exists, a client:
- never starts a broker of its own: it waits up to 2 minutes for the service, then fails naming the unit;
- never replaces a broker that is serving, managed or not: evicting a stray is the installer's job.
Deleting the unit (uninstalling the service) restores the old behaviour.
python3 test/managed_unit.py.
A suite with functions is ready when its triggers are (0.21.0)
firebase-tools binds every emulator's port first and only then lets the
functions emulator load its source and register its triggers with the Firestore
(and other) emulators. Until 0.21.0 acquire returned on open ports, so a write
made in that window was never delivered to any trigger, and nothing reported
it. On 2026-11-05 a walk2gether voice-walk simulation lost about 18s of writes
this way. When functions is in the suite, acquire now also waits for the
CLI's own "All emulators ready!" line, which firebase-tools prints once every
emulator's connect() (trigger registration included) has returned. If the line
hasn't appeared FBPORTS_FUNCTIONS_READY_TIMEOUT seconds (default 300) after the
ports opened, the suite is killed and acquire fails, saying so. A suite without
functions is ready on open ports, as before. python3 test/functions_ready.py.
