Leverege Build Tools

A collection of build, deployment, and infrastructure tools for the Leverege platform.

Installing

npm install -g @leverege/build-tools

Prerequisites

Most tools assume the following are installed and configured:

• gcloud (authenticated, with appropriate project access)
• kubectl (with a valid cluster context)
• helm (v3+)
• fzf (for interactive pod selection in k8x and klog)
• Node.js 22+ - prefer 24
• npm 11.10+

Tools

Core

build-tools — Utility for locating the build-tools installation and exposing shared bash functions.

build-tools --bashfun      # output path to bash-funcs for sourcing
build-tools --reporoot     # print the repo root path
build-tools --version      # print installed version

bash-funcs — Shared bash function library. Source it in any shell script:

. `build-tools --bashfun`

Provides: color(), errorExit(), warnOnError(), exitOnError(), generatePassword(), showInstalling(), bindWorkloadIdentity(), createNamespaceIfNeeded(), waitForSA(), waitForIAM(), ifPluggedIn(), and more.

Helm & Kubernetes Deployment

The "field is immutable" error and helmup --mutate

A routine helmup will occasionally fail with an error like:

Error: UPGRADE FAILED: cannot patch "my-service" with kind Deployment:
Deployment.apps "my-service" is invalid: spec.selector: Invalid value: ...:
field is immutable

Why it happens. Kubernetes enforces immutability on certain fields after a resource is first created. The most common is spec.selector on a Deployment — the label selector that determines which pods belong to the workload. Once set at creation time, it can never be patched in place; any attempt to change it is rejected by the API server outright, regardless of how the change arrives (kubectl apply, helm upgrade, etc.). Helm has no built-in way to recover from this on its own. (Kubernetes docs — label selector updates)

At Leverege, this most commonly surfaces when a service chart is upgraded from the older self-contained flat chart structure to the newer Leverege Base Chart library approach (leverege-base-charts). As the platform's chart conventions have evolved and improved over the years, migrating a chart with init-my-chart can alter the rendered selector labels, leaving the live Deployment's immutable spec.selector incompatible with what helm now wants to apply.

What helmup --mutate does. Before invoking helm, helmup deletes the Deployment object for the service by name (kubectl delete deployments.apps <service>) and waits for the deletion to complete. Helm then runs its normal upgrade, which recreates the Deployment from scratch with the new selector. Because the object is gone before helm runs, there is no immutable-field conflict to hit.

helmup will ask for confirmation before deleting and prints a warning that brief downtime is coming. The upgrade proceeds automatically once you confirm.

Side effects to be aware of:

• Brief downtime — pods are terminated when the Deployment is deleted and do not come back until helm recreates the object. The duration is typically a few seconds.
• LoadBalancers and static IPs are unaffected — only the Deployment object is deleted; Services (and their external IPs) remain untouched throughout.
• StatefulSets are not handled — --mutate targets deployments.apps only. If you hit an immutable field on a StatefulSet (e.g. spec.volumeClaimTemplates), you must delete the StatefulSet manually using --cascade=orphan to preserve PVCs, then re-run helmup normally. (Kubernetes docs — StatefulSet limitations)

In short: reach for --mutate whenever you see the immutable field error on a Deployment. It is targeted, explicit, and safe — the only cost is the brief downtime while the object is recreated.

helmup — Context-safe Helm deployment tool. Installs or upgrades one or more services.

helmup <service> [service...]   # deploy one or more services
helmup platform                 # deploy all platform services
helmup system                   # deploy all system services

Aliases: helmboot (rerun bootstrap), helmcycle (down then up), helminit (clean reinstall), helmwhat (dry run). Handles workload identity binding, namespace creation, SA creation via service-man, and GCS bucket setup automatically.

helmdn — Uninstalls Helm releases. Requires confirmation before proceeding.

helmdn <service> [service...]

overwhelm — Kubernetes context validator and YAML template processor. The configuration backbone of helmup — replaces ${VARIABLE} placeholders in .ovh files, validates the active k8s context, and generates values.yaml files.

overwhelm --genvals            # generate values.yaml from values-local.yaml
overwhelm --context            # print the current k8s context

helm-audit — Displays all deployed Helm releases with chart versions, app versions, and deployed image tags. Highlights version mismatches.

helm-audit

chart-compass — Locates Helm charts in GCP Artifact Registry for use by helmup.

chart-compass --service <name>
chart-compass --init           # generate chart-compass.yaml

init-my-chart — Initializes a new Helm chart from Leverege leaf chart templates, or upgrades an existing chart to the current template structure.

init-my-chart [--dry-run] [--debug]

chart-to-registry — Pushes Helm charts to GCP Artifact Registry with version checking and dependency resolution. Warns if appVersion in Chart.yaml does not start with v (bypass with --ignore-appv).

chart-to-registry --chart <dir> [--project <id>] [--repository <name>] [--force] [--dry-run] [--ignore-appv]

push-my-chart — Publishes Helm charts to both GCS Chartmuseum (legacy) and GCP Artifact Registry. Displays chart name, version, and appVersion before pushing; warns if appVersion does not start with v.

push-my-chart [--chart <dir>] [--no-registry]

Service & Infrastructure Management

service-man — Manages GCP service accounts, IAM roles, and secrets for Kubernetes cluster bootstrapping. Runs during helmup for each service, and standalone for initial cluster setup.

service-man --create-service <name>   # create SA, IAM roles, and k8s SA
service-man --setup-secrets           # write all secrets to GCP Secret Manager
                                      # and apply ESO ExternalSecrets
service-man --setup-roles             # bind IAM roles
service-man --fetch-versions          # display deployed service versions
service-man --sync-secrets <sm>=<k8s> # sync SM secret to k8s (legacy)

Secrets are defined as JSON files in src/service-man/config/secrets/. See the Secrets README for the full format reference.

Docker Image Management

docker-to-registry — Builds and pushes Docker images (Node.js/general) to GCP Artifact Registry with git tagging. Any non-semver version string (e.g. latest) is treated as a beta build and skips git tagging.

docker-to-registry v1.2.3            # tagged release build
docker-to-registry latest            # beta build — skips git tagging
docker-to-registry v1.2.3 --dry-run  # skip confirmation prompt
docker-to-registry guide-me          # print helm/registry migration guidance
docker-to-registry v1.2.3 --debug    # enable debug logging

build-cnpg-image — Builds a custom CloudNative PostgreSQL image with jsquery and timescaledb extensions using GCP Cloud Build.

build-cnpg-image <sub-tag> [--pg-version <version>]
# e.g. build-cnpg-image final --pg-version 17.6

Database & Snapshot Management

snapshot-cnpg — Creates a volumeSnapshot backup of a CNPG cluster.

snapshot-cnpg <cluster-name>

clone-cnpg-from-snapshot — Clones a CNPG database cluster from a snapshot, optionally across GCP projects or regions. Generates YAML and scripts; requires --execute for real operations.

clone-cnpg-from-snapshot \
  --snapshot <name> \
  --source-project <proj> \
  --dest-project <proj> \
  --dest-region <region> \
  [--execute] [--dry-run] [--cleanup]

gcp-snapshots — Lists or deletes old GCP VM/disk snapshots.

gcp-snapshots --project <id> [--age <days>] [--list|--delete]
# default age threshold: 180 days

Kubernetes Utilities

k8s — Unified Kubernetes toolkit with cluster-context safety via overwhelm. All subcommands verify the active cluster context before acting to prevent cross-cluster contamination.

k8s fwd <target>                    # port-forward to a named cluster endpoint
k8s exe [target]                    # exec into a pod — shortcut or fuzzy match
k8s scale <up|dn> [deployments...]  # scale deployments up or down
k8s roll [deployments...]           # rolling restart of one or more deployments
k8s log [service]                   # stream pod logs with pino-pretty formatting
k8s cryo init                       # generate k8s-cryo.yaml from live cluster state
k8s cryo hibernate                  # scale all workloads to zero and drain node pools
k8s cryo wake                       # restore node pools and workload replica counts

k8s fwd — Port-forwards a named cluster service to localhost. Handles Ctrl+C cleanly.

k8s fwd elastic    # localhost:9200 → elasticsearch8:9200
k8s fwd redis      # localhost:6379 → redis-master:6379
k8s fwd valkey     # localhost:6379 → valkey-master:6379
k8s fwd prom       # localhost:9090 → prometheus-operated:9090
k8s fwd alert      # localhost:9093 → alertmanager-operated:9093
k8s fwd grafana    # localhost:3000 → prometheus-stack-grafana:3000
k8s fwd <target> --local <port>     # override the local port

k8s exe — Execs into a running pod. Named shortcuts resolve to the right namespace and CLI automatically; any other string is used as a fuzzy filter against pods in the default namespace.

k8s exe               # interactive pod picker (all namespaces)
k8s exe cnpg          # bash on a cnpg-operands pod
k8s exe es8           # bash on an elasticsearch8 node
k8s exe grafana       # bash on grafana (prometheus namespace)
k8s exe grafold       # bash on legacy grafana (monitoring namespace)
k8s exe prometheus    # bash on prometheus-operator pod
k8s exe psql          # psql -U postgres on a CNPG node
k8s exe redis         # redis-cli on redis master
k8s exe valkey        # valkey-cli on valkey master
k8s exe <pattern>     # fuzzy match against pod names in default namespace

k8s scale — Scales one or more deployments in a namespace. Notifies Slack on each result. Interactive picker if no deployment names are given.

k8s scale up api-server message-processor    # scale to 1 replica
k8s scale dn api-server                      # scale to 0
k8s scale up                                 # interactive: pick from listed deployments
k8s scale dn -n monitoring grafana           # scale in a specific namespace

k8s roll — Rolling restart of one or more deployments (kubectl rollout restart). Notifies Slack on each result. Interactive picker if no names given.

k8s roll api-server authz-server
k8s roll                          # interactive picker
k8s roll -n monitoring grafana

k8s log — Streams pod logs with pino-pretty formatting. Follows the stream until Ctrl+C. Can tee raw output to a timestamped file.

k8s log                             # interactive pod picker (all namespaces)
k8s log api-server                  # stream by app= label selector, default namespace
k8s log api-server -n staging       # specify namespace
k8s log api-server --tee            # also save raw logs to api-server-<timestamp>.log
k8s log api-server --tee my.log     # save to a named file
k8s log api-server --no-pretty      # raw output, skip pino-pretty
k8s log api-server --tail 200       # start with last 200 lines (default 50)

k8s cryo — Cluster hibernation and wake tool. Uses k8s-cryo.yaml in the current directory to define shutdown order, dependency phases, node pool sizing, and CNPG operator awareness. Intended to be run from a cluster directory (e.g. plat-k8s/tst-builder/).

k8s cryo init                 # scan live cluster and generate k8s-cryo.yaml
k8s cryo init --force         # overwrite existing k8s-cryo.yaml
k8s cryo hibernate            # stop all workloads and drain node pools
k8s cryo wake                 # restore nodes and scale workloads back up
k8s cryo hibernate ./path/to/k8s-cryo.yaml   # explicit config path

k8s cryo init scans the live cluster for deployments, StatefulSets, CNPG clusters and poolers, node pools, and system namespaces (traefik, cert-manager, external-secrets, velero, estafette, prometheus, monitoring). It queries live replica counts for variable-size StatefulSets (timescale slave, valkey replicas, elasticsearch data) and emits a complete k8s-cryo.yaml. Review the generated file — especially the Project phase member list and any cluster-specific replica counts — before committing.

k8x (deprecated — use k8s exe) — Opens an interactive shell (or specific CLI) on a running pod, with fuzzy finder support for pod selection. Requires fzf.

k8x               # open bash on any pod (fuzzy select)
k8x cnpg          # bash on a cnpg cluster node
k8x es8           # bash on an elasticsearch8 node
k8x grafana       # bash on grafana
k8x prometheus    # bash on prometheus
k8x redis         # redis-cli on redis master
k8x valkey        # valkey-cli on valkey master

k8scale (deprecated — use k8s scale and k8s roll) — Scales Kubernetes deployments or performs rolling restarts.

k8scale up [services]    # scale up to minReplicas
k8scale dn [services]    # scale down to zero
k8scale roll [services]  # rolling restart

klog (deprecated — use k8s log) — Streams logs from a pod with pino-pretty formatting. Saves to a timestamped log file. Requires fzf for interactive selection.

klog                     # fuzzy-select pod and stream logs
klog <app-name>          # stream logs for named app
klog <app-name> <ns>     # specify namespace

Package & Release Management

hook-and-release — Manages git pre-commit hooks and version consistency for npm package releases. Runs automatically as the npm prepare lifecycle hook.

prepack — Pre-publication build step that compiles code and creates git tags.

prepack [--compile] [--manual-tag] [--force-tag] [--push-tag]

refresh-npm-token — Fetches the current npm token and rc file templates from GCP Secret Manager and overwrites ~/.npmrc and ~/.yarnrc.yml. DevOps controls the full content of both files via three secrets: REFRESH_NPM_TOKEN (raw token), REFRESH_NPM_NPMRC, and REFRESH_NPM_YARNRC (templates with ${REFRESH_NPM_TOKEN} placeholder). Skips refresh if ~/.npmrc is less than 24 hours old. On first run, existing rc files are preserved as .LEGACY before being overwritten.

refresh-npm-token                        # refresh if older than 24h
refresh-npm-token --force                # always refresh
refresh-npm-token --beta                 # use _BETA template secrets (migration testing)
refresh-npm-token --dry-run              # fetch and validate without writing files
refresh-npm-token --no-version-check     # warn on version requirements but continue
refresh-npm-token --no-update-yarnrc     # skip ~/.yarnrc.yml update
refresh-npm-token --npmrc-age 1h         # custom age threshold

repo-panopticon — Multi-repo health dashboard. Checks each repository against a suite of indicators in a compact 80-column table. Monorepo-aware: workspace packages are auto-discovered, grouped under their parent repo label, and checked individually.

repo-panopticon <dirs...>                 # check one or more repos / globs
repo-panopticon @plt/* @kud/* @srv/*      # typical full-org sweep
repo-panopticon --no-pull @plt/*          # skip git pull/fetch
repo-panopticon --limit 40 @srv/*         # cap displayed rows (default 80)
repo-panopticon --concurrency 10 @srv/*   # increase parallelism (default 5)

Columns: mgr (npm/yrn) · pkg (package.json version) · git (latest semver tag) · scop (package scope) · cir (CI status — ✘ missing · L legacy · 1 orb v1 · 2 orb v2 · ★ orb v2 + GHA ci.yml) · ESM ("type":"module") · har (.har hooks directory) · lnt (ESLint major version) · bbl (Babel presence). Repo name shown in orange when dirty or sync failed.

pkglint — Verifies required documentation files (README.md, LICENSE.md) are present before publishing.

pkglint

generate-docs — Generates JSDoc API documentation from source files.

generate-docs

count-lines-of-code — Counts lines of code in a project or monorepo workspaces.

count-lines-of-code [--src-only]

CI/CD Integration

circle-orb-it — Installs or updates CircleCI configuration (Leverege orb v2.0.0) and GitHub Dependabot settings for a repository. Auto-detects npm vs yarn-berry and writes the appropriate template. Preserves existing customizations.

circle-orb-it [--dry-run] [--debug]

github-actions-it — Installs or updates GitHub Actions CI configuration for a Leverege library repository. Reads existing .circleci/config.yml to preserve settings (coverage threshold, cache config, GCP context) when migrating from CircleCI. Generates .github/workflows/ci.yml using the shared library-ci.yml reusable workflow.

github-actions-it [--dry-run] [--debug]

send-to-slack — Sends messages to a Slack channel, retrieving the webhook URL from GCP Secret Manager.

send-to-slack --project <id> --config <secret-name> --message <text> [--dry-run]

Git Utilities

dirty-git — Shows the status of multiple git repositories (clean/dirty/stashed/unpushed).

dirty-git <dir1> [dir2] ...

prune-git — Garbage-collects and prunes remote branch references in clean repositories.

prune-git <dir1> [dir2] ...

pull-git — Pulls latest changes in clean repositories, skipping dirty ones.

pull-git <dir1> [dir2] ...

git-tar — Creates a tarball of a git repository with exclusion list support.

git-tar    # run from within the repository

Secrets & Encryption

encrypt-secrets — Encrypts .env and secrets/ directory using GCP KMS.

encrypt-secrets

decrypt-secrets — Decrypts KMS-encrypted files, restoring .env and secrets/.

decrypt-secrets

Helm Charts

The src/helm-charts/ directory contains bootstrapped chart configurations for all platform, auxiliary, and system services. Each chart directory may contain:

• helmup.bootstrap — one-time setup script run on first deploy (SA creation, IAM, secrets)
• helmup.plugin — helm install/upgrade logic
• helmdn.plugin — teardown logic
• *.yaml.ovh — overwhelm-processed Kubernetes manifests
• values-*.yaml — helm values overrides

Platform services: api-server, authz-server, db-curator, emailer, imagine, message-processor, messenger, reason, resource-server, rule-engine, scheduler, transponders (bq/dh/rt/tsdb)

Auxiliary services: argocd, fota-server, geotile-server, pgbouncer, pubsub-pulse, push-notifier, pusher, vin-decoder-server

System services: cnpg-operator, cnpg-db-psql-stack, cnpg-db-tsdb-basic, cnpg-db-tsdb-dense, elasticsearch8, eso, prom-operator, traefik, valkey, velero, pg-repack

Configuration

For additional debug logging, set BUILD_TOOLS_DEBUG=1:

BUILD_TOOLS_DEBUG=1 docker-to-registry

Deprecated Tools

| Tool | Replacement | |------|-------------| | k8x | k8s exe | | k8scale | k8s scale / k8s roll | | klog | k8s log | | pkgck | repo-panopticon | | docker-to-registry-py | docker-to-registry | | chart-to-museum | push-my-chart | | refresh-py-idx | No longer needed | | k8cryo | snapshot-cnpg / clone-cnpg-from-snapshot | | reflector (helm) | ESO ExternalSecrets |

Authors

• DevOps and Friends