@ship.zone/ci-compiler

@ship.zone/ci-compiler is the reference compiler of the ship.zone CI standard, @ship.zone/ci-spec. It owns the deterministic parts of the standard that coordinators and runners share: RFC 8785 canonical JSON and digests, the specification version rule, the strict reading of ci_actions.yml and its check against the schema and the workflow rules, branch and tag pattern matching, run compilation into compiled jobs, plan digests, image publications, the cache namespace, and the concurrency key, the insertion of secret values into the job of a lease, validation of compiled runner jobs, and candidate image index assembly.

Every function is pure. The only I/O is loading the normative JSON Schemas from the installed @ship.zone/ci-spec once, when the module loads; no function reads a clock, randomness, the environment, the filesystem, or the network, and equal inputs give byte-equal outputs.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Install

pnpm add @ship.zone/ci-compiler

Specification Version

The compiler implements exactly one specification version: the version of the @ship.zone/ci-spec it pins exactly. A coordinator or runner that uses the compiler pins the same @ship.zone/ci-spec version, so the version it negotiates is the version it compiles and checks against.

import { acceptCiSpec, implementedCiSpecVersion, lowerCiSpec } from '@ship.zone/ci-compiler';

implementedCiSpecVersion; // '2.1.2'
acceptCiSpec(implementedCiSpecVersion, '2.0.0'); // 'accepted'
acceptCiSpec(implementedCiSpecVersion, '2.2.0'); // 'newer'
acceptCiSpec(implementedCiSpecVersion, '1.2.0'); // 'major-mismatch'
acceptCiSpec(implementedCiSpecVersion, '2.0'); // 'malformed'
lowerCiSpec('2.1.0', '2.0.5'); // '2.0.5'

acceptCiSpec and lowerCiSpec throw CiCompilerUsageError for a version that must be canonical and is not; lowerCiSpec also throws for versions of different major versions, which never form a session.

Canonical JSON and Digests

import {
  canonicalizeCiJson,
  digestCiCompiledPlan,
  digestCiJson,
  projectCiCompiledPlan,
  sha256CiHex,
} from '@ship.zone/ci-compiler';

canonicalizeCiJson({ b: 1, a: [true, null] }); // '{"a":[true,null],"b":1}'
digestCiJson(requestBody); // canonicalRequestDigest: lowercase SHA-256 of the RFC 8785 bytes
projectCiCompiledPlan(job); // the job without compiledPlanDigest, every secrets value replaced by true
digestCiCompiledPlan(job); // compiledPlanDigest
sha256CiHex(bytes); // lowercase hex SHA-256 of bytes or of a UTF-8 string

Canonicalization throws CiCanonicalJsonError for a value without an RFC 8785 serialization: a lone UTF-16 surrogate in a string or member name, a non-finite number, a value that is not JSON, an array with holes, or a cycle.

Workflow Check

checkCiWorkflow reads ci_actions.yml bytes as a coordinator reads them from the source commit and checks the five layers of ci-actions, Compilation Failures, that depend only on the bytes: source, yaml, version, schema, and workflow. Every run of one commit fails these layers alike. It returns the typed document with the admitted trigger kinds of every job, or the failure of the first layer with a violation:

import { checkCiWorkflow } from '@ship.zone/ci-compiler';

const result = checkCiWorkflow(bytes);
if (result.ok) {
  result.value.spec; // '2.0.0'
  result.value.workflowDigest; // lowercase hex SHA-256 of the bytes as read
  result.value.document; // ICiWorkflow
  result.value.mappingKeyOrder.get('/jobs/test/matrix'); // matrix dimensions in declaration order
  result.value.admittedKinds.get('test'); // ['push', 'tag']: the run kinds that do not skip the job
} else {
  // result.failure: { code, message, location?, issues? }
}

| Layer | Codes | Rule | | --- | --- | --- | | source | workflow_too_large, workflow_encoding_invalid | at most 262,144 bytes, a leading byte order mark included, and well-formed UTF-8 | | yaml | workflow_yaml_document_count, workflow_yaml_directive, workflow_yaml_duplicate_key, workflow_yaml_key_type, workflow_yaml_anchor, workflow_yaml_merge_key, workflow_yaml_tag, workflow_number_invalid, workflow_yaml_syntax | YAML 1.2 with the core schema: one document as YAML 1.2 counts them, no directive, unique string keys, no anchor, alias, merge key, or explicit tag, and numbers that are finite and, when integral, within ±9,007,199,254,740,991; the character, byte order mark, escape, and nesting depth rules and any other parser error or warning are workflow_yaml_syntax | | version | spec_incompatible | a canonical spec that is newer or major-mismatch for the implemented version | | schema | workflow_schema_invalid | schemas/ci_actions.schema.json; issues carries the schema errors, sorted, at most 32 | | workflow | the 20 codes from trigger_pattern_invalid to publish_reference_invalid | the rules ci-actions applies in every run, whatever its kind; see below |

The yaml layer follows ci-actions, Parsing, on the syntax tree of the yaml package:

• A line feed, a carriage return followed by a line feed, and a carriage return alone each end a line, as in YAML 1.2, and every line break in scalar content, a quoted scalar's included, reads as a line feed. The yaml package breaks lines only at a line feed, so the text is read with every line break as a line feed before it is parsed; the size limit and the workflow digest cover the bytes as read.
• A C0 control character other than tab, line feed, and carriage return fails anywhere, a quoted scalar included. DEL, C1 control characters other than NEL, U+FFFE, U+FFFF, and U+FEFF are content inside a single- or double-quoted scalar token and fail anywhere else.
• Runs of U+FEFF at the start of a line are byte order marks, not content, unless the line lies within a document's content: after the line on which the document begins and before the end of its last content token. A line that begins with U+FEFF ends a block scalar, a root scalar of content indentation 0 included: after a keep-chomped |+ or >+ scalar, the empty lines before it are content and the lines from it on are not. Such a line after a document's content is read as a document end marker ..., which begins no document. The file's own leading byte order mark is the first of them. Any other U+FEFF outside a quoted scalar fails.
• In a double-quoted scalar, a \u high surrogate escape directly followed by a \u low surrogate escape is one character; every other surrogate escape, the \U form included, fails.
• Collections nest at most 64 deep, the root collection at depth 1; a deeper collection fails and is not read.

Within the yaml layer a rule of the subset is reported before a syntax violation, and among the rules the first in the text. Positions in messages are lines and columns of the text as parsed: a carriage return alone ends a line, and columns do not count the byte order marks at the start of a line, except on a line read as .... A failure's code is normative; its location and message are informative. Integer tokens keep their exact value, float tokens are converted to binary64, negative zero becomes zero, and quoted scalars are always strings. The document is built from the YAML syntax tree in declaration order. JavaScript enumerates integer-like object keys first, so mappingKeyOrder carries the declaration order of every mapping by JSON Pointer; matrix expansion follows it. The function throws CiCompilerUsageError when its argument is not a Uint8Array.

Workflow Rules

The workflow layer checks its rules in the order of the Compilation Failures table and reports the first code whose rule the workflow violates, wherever in the file the violation is. location points at the first violation of that rule, with jobs in declaration order.

| Code | Rule | | --- | --- | | trigger_pattern_invalid | a branch or tag filter outside the pattern grammar | | needs_unknown | needs names an undeclared job | | needs_cycle | needs forms a cycle, a self-dependency included | | expansion_limit_exceeded | more than 256 expanded jobs: one per matrix combination or build platform, skipped jobs included, and no candidate assembly node | | needs_trigger_wider | a job admits a kind a job it needs does not admit | | trigger_kind_unused | a kind in on that no job admits | | permission_escalation | a job permission above the workflow permission, images: publish counting as candidate | | permission_insufficient | an artifact without artifacts: write, a cache without caches: read, a read-write cache without caches: read-write, or a build job without images: candidate | | environment_conflict | a SHIPZONE_CI_* name in an environment map or as a secret target, two secret references with one target, or a target that is a name of a step's merged environment | | environment_limit_exceeded | a step receives more than 128 names: the union of its merged environment, its job's secret targets, and the reserved names, each counted once | | command_limit_exceeded | a command argument above 16,384 UTF-8 bytes or a command above 131,072 | | transfer_name_duplicate | two artifacts or two caches of a job with one name | | cache_path_overlap | two cache paths of a job whose segments, without empty and . segments, are a prefix of each other | | resources_invalid | a declared sharedMemoryBytes above a declared memoryBytes | | network_host_duplicate | an egress host listed twice | | npm_read_invalid | an npmRead registry listed twice, compared as written, a scope under two registries, a registry host and port the job's egress allowlist does not permit, a *. entry permitting every name below its domain, or, with npmRead in a job of any profile, NPM_CONFIG_USERCONFIG in a step's merged environment or as a secret target, or a build secret with id npmrc | | build_input_conflict | a build secret naming no job secret target, a target no build secret names, or a build argument named like a target or build secret id | | candidate_reference_invalid | a candidate of a job that is not a build job, or of a build job the referencing job does not need | | candidate_platform_missing | a pinned runner.platform or a consuming build platform the producing build job does not list | | publish_reference_invalid | an image publication of a job that is not a build job, an artifact reference to a matrix job or to an artifact not declared with required: true and when: success, or a referenced job that does not admit tag |

Branch and Tag Patterns

matchCiRefPattern decides whether a branch or tag filter matches a short ref name under the grammar of ci-actions, Triggers: anchored, case-sensitive, over Unicode scalar values, with *, **, ?, and the escapes \*, \?, and \\. It runs in time proportional to the product of the two lengths and throws CiCompilerUsageError for a pattern that checkCiWorkflow rejects with trigger_pattern_invalid.

import { matchCiRefPattern } from '@ship.zone/ci-compiler';

matchCiRefPattern('release/*', 'release/1'); // true
matchCiRefPattern('release/*', 'release/next/1'); // false
matchCiRefPattern('release/**', 'release/next/1'); // true

Run Compilation

compileCiRun compiles one run of a checked workflow. The run context and the deployment policy have the shapes of conformance/compile-cases.json, and the compiler validates them against that schema:

import { checkCiWorkflow, compileCiRun, matchCiTrigger } from '@ship.zone/ci-compiler';

const checked = checkCiWorkflow(bytes);
if (checked.ok && matchCiTrigger(checked.value, trigger)) {
  // Prepare the source archive, whose descriptor every compiled job carries, then compile.
  const outcome = compileCiRun(checked.value, {
    repository: { tenantId, repositoryId, url },
    sourceObjectId,
    source, // { format: 'tar.gz', sha256, sizeBytes, extractedSizeBytes, entryCount }
    trigger, // { kind: 'push', ref: 'refs/heads/main' }, { kind: 'tag', ref }, { kind: 'pullRequest', pullRequest, targetBranch }, or { kind: 'manual', ref, inputs }
    trustClass, // 'trusted' | 'untrusted'
    protectedTagRun,
    secrets, // [{ name, protected, valueBytes }], never values
  }, {
    limits, // source and log limits, workspace defaults
    ociIsolation: 'container', // or 'microvm'
    maximumTimeoutMs, // optional
    maximumRetentionDays, // optional
    deniedEgressHosts, // optional, exact hosts
    deniedNpmRegistries, // optional, exact registries
  });
  if (outcome.outcome === 'compiled') {
    outcome.plan.nodes; // [{ id: 'test[0]', kind: 'job', needs, job, secretBindings }, { id: 'image', kind: 'assembly', needs }, …]
  }
}

compileCiWorkflow(bytes, context, policy) does both steps in one call: a workflow that fails the first five layers fails every run of its commit with that failure, whether or not the event matches its triggers.

The outcome is compiled with the plan, not-triggered when on does not declare the event's kind or its filter does not match, or failed with one code:

1. A job that does not admit the run's kind is skipped: it and all its expansions are left out of the run, and its secrets are never evaluated.
2. The policy layer checks, in table order: the source archive against the deployment source limits (policy_limit_exceeded), each job's timeoutMs or its default of 3,600,000 (policy_timeout_exceeded), each artifact's and cache's retentionDays or its default of 30 or 7 days (policy_retention_exceeded), and the denied egress hosts and npm registries. The workspace values of the policy are defaults that a declared resources.workspaceBytes or workspaceEntries replaces, not maxima.
3. The run layer reports the first code in table order whose rule the run violates, wherever in the workflow:

| Code | Rule | | --- | --- | | manual_input_invalid | a dispatch value that is undeclared, of another type, outside its choice options, or a missing required input | | tag_version_required | a tag run without a tag version, of a workflow whose publish block has a version alias or an npm publication | | alias_invalid | on a tag run, an expanded alias that is not a valid OCI tag, [A-Za-z0-9_][A-Za-z0-9._-]{0,127}, for example a tag version with + build metadata | | alias_duplicate | on a tag run, two aliases of one destination that expand to one tag | | secret_untrusted | an untrusted run in which a job references a secret | | npm_read_untrusted | an untrusted run in which a job declares npmRead | | secret_unknown | a reference to a secret the repository does not hold | | protected_secret_denied | a run that is not a protected tag run references a protected secret | | secret_too_short | a referenced secret whose stored value is shorter than 8 bytes |

The publication rules apply to every tag run of a workflow with a publish block, whether or not it is a protected tag run. The secret and grant rules are evaluated over every reference of the jobs the run does not skip before a code is chosen. An untrusted run fails before any repository secret is looked up, so its failure, message included, is the same whatever the repository holds. Only the protected mark and the value length of a repository secret are read; the compiler never receives a value during compilation.

Every job that is not skipped expands to nodes in plan order, by job name in Unicode code point order: a job to the node test, a matrix job to test[0], test[1], … in expansion order, the first-declared dimension varying slowest, and a build job to image[linux/amd64], … in build.platforms order followed by its assembly node image. A node that needs a job depends on all its job nodes, or on the assembly node of a build job, in plan order. deriveCiTagVersion gives the tag version of a tag short name, which labels the builds of a tag run as org.opencontainers.image.version.

if (outcome.outcome === 'compiled') {
  const plan = outcome.plan;
  plan.workflowDigest;
  plan.tagVersion; // on a tag run with a tag version
  plan.nodes; // [{ id: 'sign', kind: 'job', needs, job, secretBindings: { SIGNING_KEY: 'signing-key' } }, { id: 'image', kind: 'assembly', needs }, …]
  plan.images; // on a protected tag run that publishes images: [{ candidate: 'image', registry, repository, tags: ['v1.2.3', 'latest'] }, …]
  plan.cacheNamespace; // { tenantId, repositoryId, trustClass, protectedTagRun }
  plan.concurrency; // when declared: { key: { tenantId, repositoryId, trustClass, protectedTagRun, ref, group }, keyDigest, cancelInProgress }
}

• Job nodes. Every job node carries its planned compiled job, whose secrets map every target to true, as in the plan digest projection, with its compiledPlanDigest, and secretBindings, the repository secret whose value each target receives. Every planned job resolves, with placeholder values, to a coherent runner job.
• Publications. images lists every destination of every image publication in declaration order with its aliases expanded: a version alias is its prefix, the tag version, and its suffix; a literal alias its value. It is present exactly on a protected tag run whose workflow publishes images; every other run publishes nothing. npm and release-asset publications are taken from document.publish of the checked workflow on a protected tag run; the plan fixes their tag version.
• Cache namespace. Caches are shared only within one namespace, so untrusted runs, trusted runs, and protected tag runs of one repository never read each other's caches.
• Concurrency. The key holds the run's ref: { kind: 'ref', ref } with the full ref name of a branch push, tag push, or manual run, or { kind: 'pullRequest', pullRequest } for every run of one pull request. group is compared exactly. keyDigest is the SHA-256 of the key's RFC 8785 bytes, equal exactly when the keys are equal.

The specification defines no digest of a run as a whole, and the plan has none: every compiled job carries its own compiledPlanDigest. An incoherent context throws CiCompilerUsageError: a context or policy outside the schema, a protected tag run that is not a trusted tag run, a trusted pull request run, a repository secret listed twice, or a ref name that is not well-formed Unicode.

Secret Resolution

resolveCiRunnerJob is the only place where secret values enter a job. At lease time, the coordinator passes the job node from the stored plan and the value of every repository secret the node binds, by secret name:

import { resolveCiRunnerJob } from '@ship.zone/ci-compiler';

const result = resolveCiRunnerJob(node, new Map([['signing-key', signingKeyValue]]));
if (result.ok) {
  lease.job = result.value; // secrets: { SIGNING_KEY: signingKeyValue }, the planned compiledPlanDigest
} else {
  // result.failure.code: 'job_secret_inadmissible' for a value shorter than 8 UTF-8 bytes, …
}

The resolved job is checked with checkCiRunnerJob, so it has the digest of the planned job, and a stored plan whose job no longer matches its compiledPlanDigest fails with job_digest_mismatch. The planned job is not changed. A missing value, a value for a secret the node does not bind, a value that is not a well-formed Unicode string, and an assembly node throw CiCompilerUsageError. No failure or error message contains a value. A coordinator evaluates the protected tag run conditions again before it leases a job that references a protected secret; the repository secret of every target is in secretBindings.

Compiled Runner Jobs

checkCiRunnerJob decides whether a value is a coherent compiled runner job with its secret values inserted (runner protocol, Job Coherence), as the coordinator checks it before enqueueing and the runner before acceptance:

import { checkCiRunnerJob } from '@ship.zone/ci-compiler';

const result = checkCiRunnerJob(JSON.parse(body));
if (!result.ok) {
  // result.failure: { code, message, location?, issues? }
}

It checks the rows of the Job Coherence table in order and reports the job failure code of the first row the job violates; the codes are the specification's runnerJobFailureCodes:

| Code | Rule | | --- | --- | | spec_incompatible | a canonical spec that is newer or major-mismatch for the implemented version | | job_schema_invalid | schemas/runner-job.schema.json; issues carries the schema errors, sorted, at most 32 | | job_secret_incoherent | in an oci-image job, a build secret names no secrets member, or a member is named by no build secret | | job_limit_incoherent | source metadata, a declared artifact or cache maximum, or a declaration count above requirements.limits | | job_resources_incoherent | resources.sharedMemoryBytes above resources.memoryBytes | | job_environment_incoherent | a step without SHIPZONE_CI_MATRIX_JSON and SHIPZONE_CI_INPUTS_JSON as RFC 8785 JSON object text, another SHIPZONE_CI_* name, a secret target with that prefix, or step environment names, secret targets, and reserved names that repeat or number more than 128 | | job_command_limit_exceeded | a step command argument above 16,384 UTF-8 bytes, which the schema bounds only in characters, or a step command above 131,072 UTF-8 bytes | | job_secret_inadmissible | a secret value shorter than 8 UTF-8 bytes | | job_digest_mismatch | compiledPlanDigest is not the digest of the plan projection |

A planned job, whose secrets values are true, fails the schema: values enter a job only for a lease. Matching a job against a runner's capability snapshot and checking lease coherence are coordinator and runner concerns outside the job and are not part of this check. No failure message contains a secret value.

Candidate Index Assembly

assembleCiCandidateIndex builds the multi-platform OCI image index of a build job from the accepted image records of its per-platform build jobs, in the declared platform order:

import { assembleCiCandidateIndex } from '@ship.zone/ci-compiler';

const candidate = assembleCiCandidateIndex(['linux/amd64', 'linux/arm64/v8'], imageRecords);
candidate.index; // { schemaVersion: 2, mediaType, manifests }
candidate.bytes; // the RFC 8785 bytes to push
candidate.digest; // 'sha256:…'

It throws CiCompilerUsageError unless every platform is a distinct linux platform with exactly one image record and every record belongs to a platform.

Conformance

The tests run the conformance data of the installed @ship.zone/ci-spec: every case of version-cases.json and digest-cases.json, every case of runner-job-cases.json with its exact job failure code, which covers every runner-jobs fixture, the job of every runner-messages.json lease, and every compiled job of compile-cases.json, whose plan digest is recomputed and which is checked with placeholder secret values. Every workflow of compile-cases.json and every ci-actions fixture is checked by checkCiWorkflow: a case of the source, yaml, version, schema, or workflow layer fails with its exact code, every other case passes with its workflow digest, and every valid fixture yields the values of the YAML in declaration order. Every patternCases vector is matched by matchCiRefPattern and, when invalid, fails a workflow in each filter with trigger_pattern_invalid. Every compile case with a run context is compiled by compileCiRun and reproduces its outcome, and a compiled case its nodes, plan digests, tag version, and image destinations byte for byte; every planned job resolves with resolveCiRunnerJob to a runner job with its planned digest. Every tagVersionCases vector gives its tag version, and every valid fixture and the example compile for every kind they admit. A coverage gate runs all 186 compile cases through checkCiWorkflow and compileCiWorkflow, with nothing deferred, and requires every failure code of the specification to be produced, except policy_denied, a deployment's refusal outside the other policy rules, which the policy shape of compile-cases.json cannot express.

Verification

pnpm run build
pnpm run test:types
pnpm test

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the repository license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at [email protected].

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.