alchemy-k8s-operator

alchemy-k8s-operator is a TypeScript framework for building Kubernetes operators whose custom resources expand into Alchemy resource graphs. One graph can reconcile Kubernetes objects and external resources such as Cloudflare DNS, AWS databases, or SaaS configuration.

The project has two layers:

• The core owns CRD generation, list/watch loops, a keyed reconcile queue, finalizers, status conditions, drift sync, observed-child watches, provider configuration, and Alchemy state.
• A concrete operator supplies an OperatorDefinition: its TypeBox spec schema, explicit Kubernetes RBAC, provider layer, and a trusted TypeScript expand() function. defineOperator() is an identity helper that preserves inference and validates invariants at module load time.

TypeScript is the configuration language deliberately. The file runs with the operator's authority, so it must be reviewed and baked into the image (or mounted from a trusted ConfigMap); it is not user-authored CR data and is not a sandbox boundary.

Define an operator

import * as Kubernetes from "alchemy/Kubernetes";
import * as Effect from "effect/Effect";
import { Type } from "typebox";
import { defineOperator } from "alchemy-k8s-operator";

const spec = Type.Object({ image: Type.String() });

export default defineOperator({
  crd: {
    group: "apps.example.com",
    version: "v1alpha1",
    kind: "App",
    plural: "apps",
    scope: "Namespaced",
    specSchema: spec,
  },
  providers: Kubernetes.providers(),
  kubernetesPermissions: [
    {
      apiGroups: ["apps"],
      resources: ["deployments"],
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"],
    },
  ],
  expand: Effect.fn(function* (resource) {
    return yield* Kubernetes.Manifest("Deployment", {
      cluster: resource.kubernetesCluster,
      manifest: {
        apiVersion: "apps/v1",
        kind: "Deployment",
        metadata: {
          name: resource.metadata.name,
          namespace: resource.metadata.namespace,
        },
        spec: { /* desired state */ },
      },
    });
  }),
});

The public contract is an interface plus defineOperator(), which is the usual TypeScript pattern for externally supplied implementations: structural typing keeps client modules simple, while the identity helper preserves literal and schema inference better than an explicit type annotation. Runtime validation catches mistakes when loading JavaScript where types no longer exist.

Install

Helm or static manifests

Generate normal operator distribution artifacts without contacting a cluster:

bunx alchemy-k8s-operator generate \
  --config ./operator.config.ts \
  --image ghcr.io/acme/app-operator@sha256:<digest> \
  --version 1.0.0 \
  --out ./dist

helm upgrade --install app-operator ./dist/chart \
  --namespace app-operator-system --create-namespace
# or: kubectl apply -f ./dist/install.yaml

This follows the common Operator SDK installation shape: CRD, service account, RBAC, and a single-replica Deployment. The generator is intentionally not an imperative installer, so artifacts can be reviewed, signed, committed, or used by GitOps. The chart follows Helm's CRD lifecycle guidance and puts the CRD in crds/; Helm installs but does not upgrade or delete CRDs, so upgrades that change the schema must apply the generated crd.yaml explicitly. OLM bundles are deferred until the APIs and upgrade policy stabilize.

As part of an Alchemy cluster graph

Use installOperator() from alchemy-k8s-operator/alchemy, passing any Kubernetes.ClusterLike. It yields Kubernetes Manifest resources and can either mount an existing credential Secret or create one from Effect Redacted/Config.redacted values. CRDs and namespaces use a retain removal policy. See the reference wrapper.

Credentials and secrets

There are two distinct credential moments:

1. Bootstrap credentials are whatever the installer needs to reach the cluster. Helm and kubectl use the caller's kubeconfig; an Alchemy install uses its cluster connection. They are never copied into the operator.

2. Runtime provider credentials belong to the operator. Prefer workload identity through ServiceAccount annotations. Otherwise create a dedicated Secret whose keys are provider environment names, for example:

   kubectl -n app-operator-system create secret generic app-operator-credentials \
     --from-literal=CLOUDFLARE_ACCOUNT_ID='...' \
     --from-literal=CLOUDFLARE_API_TOKEN='...'

The Secret is mounted as files. Every reconcile builds a fresh Effect ConfigProvider, so projected Secret rotation is picked up without injecting secret values into process environment variables. File changes enqueue all known CRs, and known values are scrubbed from surfaced error messages. Make the volume non-optional when missing credentials should prevent pod readiness; leave it optional for workload identity. For Helm, put identity annotations in serviceAccount.annotations; installOperator() accepts the same setting as serviceAccountAnnotations.

Restrict Secret read/write access to the operator namespace, use separate provider credentials with least privilege, enable Kubernetes encryption at rest (ideally KMS envelope encryption), and avoid putting secrets in CR specs.

State and lifecycle

Each CR gets an independent Alchemy stack. State is sharded across Kubernetes Secrets in the operator namespace and labeled by operator, stack, stage, and owner. Sharding avoids the single-object 1 MiB limit and isolates update conflicts. A finalizer keeps the CR until alchemy destroy has removed managed resources. Status contains only progress and conditions.

State intentionally does not live in CR metadata:

• annotations have a much smaller practical size budget and are commonly exposed to readers, logs, policies, and UIs;
• Alchemy state may include sensitive provider attributes and needs Secret RBAC/encryption semantics;
• controller state writes would contend with users, admission, and other controllers updating the CR;
• sharded Secrets support independent writes and cleanup without bloating the public API object.

The cost is more Kubernetes objects and a namespace-scoped Secret permission. The framework labels and hashes their names, and never adds cross-namespace owner references (which Kubernetes forbids). The CR finalizer performs explicit state cleanup.

By default the controller checks drift every ten minutes with ±10% jitter. observe(output) can identify Kubernetes children whose watch events should trigger immediate reconciliation. The controller currently runs one replica; leader election is the natural prerequisite before horizontal scaling.

Reference operator

examples/cloudflare-webapp defines a WebApp CR that creates a Kubernetes Deployment/LoadBalancer Service and a proxied Cloudflare CNAME. It demonstrates a mixed internal/external graph, child watching, explicit RBAC, a derived image, an Alchemy installer wrapper, and a sample CR.

Development

mise install
mise exec -- bun install --frozen-lockfile
mise exec -- bun run release:check
mise exec -- bun run generate:example

GitHub Actions runs the same release:check gate for pull requests, main, and release tags. It type-checks and tests the project, then dry-packs the npm package and verifies that only the public TypeScript sources and package documentation are included.

Local K3s end-to-end test

With Docker running, exercise the current working tree in a disposable K3s cluster backed by k3d:

mise install
mise exec -- just e2e-local

The harness provisions the cluster through alchemy-docker-k3s, builds and imports the root operator image, mounts a trusted TypeScript definition from a ConfigMap, and installs it with installOperator(). It then verifies CRD and RBAC installation, status and finalizers, Kubernetes Secret state, spec updates, repair of changed and deleted children, restart recovery, and complete CR deletion. The operator install and cluster stacks are destroyed even when a check fails. A stable alchemy-k8s-operator-e2e:cache image tag is retained so subsequent working-tree builds can reuse the production dependency layer.

To validate an already-published public image instead of the working tree, set its exact tag or digest. The cluster pulls this image, so no local build/import is performed:

OPERATOR_E2E_IMAGE=ghcr.io/toolbar23/alchemy-k8s-operator:main \
  mise exec -- just e2e-local

Operator container image

.github/workflows/operator-image.yml builds the generic root image for Linux AMD64 and ARM64. Pull requests build it without publishing; pushes to the default branch and v* tags publish to ghcr.io/<owner>/<repository> using GitHub's package token. Published images include BuildKit provenance and an SBOM. The first package publication may need its GHCR visibility changed to public before an unauthenticated K3s cluster can pull it.

The image intentionally contains the core runtime but no concrete operator definition. Use configMap in installOperator() to mount reviewed TypeScript, as the local E2E does, or derive an image that bakes in the definition and its additional dependencies.

Releases

The repository pins Bun, Node.js, npm, and just with mise. Preview the exact npm package and run every release gate locally with:

mise install
mise exec -- just release-dry-run

Actual releases are published by npm-package.yml using npm trusted publishing. Configure the alchemy-k8s-operator package on npm with this trusted publisher before creating the first automated release:

• provider: GitHub Actions;
• organization or user: toolbar23;
• repository: alchemy-k8s-operator;
• workflow filename: npm-package.yml;
• environment: npm;
• allowed action: npm publish.

Create a GitHub environment named npm; add required reviewers and a deployment tag rule such as v* if you want releases to require approval. No npm token is stored in GitHub: the workflow receives a short-lived credential through OIDC.

To release, update version in package.json, merge that revision to main, and create and push an exactly matching tag. For example, version 0.1.1 must be tagged v0.1.1:

jj git fetch
jj tag set v0.1.1 -r main@origin
jj git push --tag v0.1.1

The workflow reruns all release checks before publishing; stable versions receive npm's latest tag and prereleases receive next.

The local just release command remains available as an authenticated emergency fallback. Normal releases should use GitHub because npm binds the publish to the specific repository, workflow, and environment and automatically records provenance for public packages.