AlignTrue Conformance Testkit v1

Conformance testkit for validating implementations of Align Spec v1. Provides language-agnostic JSON test vectors and a TypeScript runner for the AlignTrue ecosystem.

Purpose

The testkit ensures that any implementation of Align Spec v1 produces byte-identical outputs for:

• Canonicalization - YAML → JSON → JCS (RFC 8785) transformation
• Hashing - SHA-256 integrity computation
• Validation - Schema and integrity verification
• Check runners - Machine-checkable rule execution

Contents

Test Vectors (JSON)

Language-agnostic test vectors in vectors/:

• canonicalization.json - 17 canonicalization edge cases
• checks/file-presence.json - 4 file presence check vectors
• checks/path-convention.json - 3 path convention check vectors
• checks/manifest-policy.json - 3 manifest policy check vectors
• checks/regex.json - 5 regex check vectors
• checks/command-runner.json - 3 command runner check vectors
• integration.json - References to 11 production aligns from AlignTrue/aligns

Total: 40 test vectors

Golden Aligns (YAML)

Synthetic minimal aligns in golden/ demonstrating specific behaviors:

1. minimal-valid.aligntrue.yaml - Absolute minimum valid align
2. canonicalization-edge-cases.aligntrue.yaml - Unicode, floats, nested structures
3. all-five-check-types.aligntrue.yaml - One rule of each check type
4. severity-levels.aligntrue.yaml - MUST, SHOULD, MAY severities
5. dependency-chain.aligntrue.yaml - Align dependencies

All golden aligns include:

• Computed integrity hashes
• Inline comments explaining what they test
• Valid schema structure
• Under 50 lines for clarity

TypeScript Runner

The @aligntrue/testkit package provides:

• runCanonVectors(vectors, impl) - Test canonicalization implementation
• runCheckVectors(vectors, impl) - Test check runner implementation
• runGoldenAligns(aligns, validator) - Test align validation
• runAllVectors(...) - Complete conformance suite

Usage (Internal)

Run the full conformance suite against the AlignTrue implementation:

pnpm verify

Or from the workspace root:

pnpm --filter @aligntrue/testkit test

This validates that our implementation conforms to Align Spec v1.

Usage (External Implementations)

1. Parse JSON Vectors

Read and parse the vector files:

# Python example
import json

with open('vectors/canonicalization.json') as f:
    canon_vectors = json.load(f)

for vector in canon_vectors:
    input_value = vector['input']
    expected_jcs = vector['expected_jcs']
    expected_sha256 = vector['expected_sha256']

    # Test your implementation
    actual_jcs = your_canonicalize(input_value)
    actual_sha256 = your_hash(actual_jcs)

    assert actual_jcs == expected_jcs, f"JCS mismatch: {vector['name']}"
    assert actual_sha256 == expected_sha256, f"Hash mismatch: {vector['name']}"

2. Canonicalization Vectors

Each vector has:

• name - Unique test case name
• description - What this tests
• input - JSON value to canonicalize
• expected_jcs - JCS (RFC 8785) canonical JSON string
• expected_sha256 - SHA-256 hash (hex) of JCS output

Contract: input → canonicalize() → JCS string → hash() → SHA-256 hex

3. Check Runner Vectors

Each vector has:

• name - Unique test case name
• description - What this tests
• check_type - One of: file_presence, path_convention, manifest_policy, regex, command_runner
• rule - Complete Align rule structure
• file_tree - Virtual file system (path → content map)
• expected_findings - Array of findings (empty if rule passes)
• allow_exec - (Optional) Whether command execution is allowed

Contract: Apply rule to file_tree and verify findings match expected_findings.

4. Golden Aligns

Validate complete Align aligns:

# Python example
import yaml

with open('golden/minimal-valid.aligntrue.yaml') as f:
    align_yaml = f.read()

# Your validator should:
# 1. Parse YAML to JSON
# 2. Validate against JSON Schema
# 3. Compute integrity hash
# 4. Verify stored hash matches computed hash

result = your_validate_align(align_yaml)
assert result.schema_valid
assert result.integrity_valid

5. Integration Vectors

The integration.json file references 11 production aligns from the AlignTrue/aligns repository. To use:

1. Clone https://github.com/AlignTrue/aligns
2. Read the align files at the specified paths
3. Verify your implementation computes matching integrity hashes

Vector Formats

Canonicalization Vector

{
  "name": "stable-key-ordering",
  "description": "Verifies that object keys are sorted lexicographically",
  "input": { "z": 1, "a": 2, "m": 3 },
  "expected_jcs": "{\"a\":2,\"m\":3,\"z\":1}",
  "expected_sha256": "ebba85cfdc0a724b6cc327ecc545faeb38b9fe02eca603b430eb872f5cf75370"
}

Check Runner Vector

{
  "name": "passes-when-files-exist",
  "description": "Verifies that check passes when files matching pattern exist",
  "check_type": "file_presence",
  "rule": {
    "id": "test-file-presence",
    "severity": "MUST",
    "check": {
      "type": "file_presence",
      "inputs": {
        "pattern": "**/*.test.ts"
      },
      "evidence": "Missing test file"
    }
  },
  "file_tree": {
    "src/foo.test.ts": "test content",
    "src/bar.test.ts": "test content"
  },
  "expected_findings": []
}

Integration Vector

{
  "id": "aligns/base/base-testing",
  "repo": "https://github.com/AlignTrue/aligns",
  "path": "aligns/base/base-testing.aligntrue.yaml",
  "expected_integrity": "FETCH_FROM_REPO"
}

Adding Vectors

To add new test vectors:

1. Add the vector to the appropriate JSON file in vectors/
2. For canonicalization vectors, compute expected JCS and SHA-256
3. Run pnpm verify to ensure it passes
4. Submit a PR with the new vector

New golden aligns:

1. Create a new .aligntrue.yaml file in golden/
2. Use the aligns/testkit/* namespace for IDs
3. Add inline comments explaining what it tests
4. Run npx tsx scripts/compute-golden-hashes.ts to stamp the hash
5. Run pnpm verify to ensure it passes

CI Integration

Add testkit verification to your CI pipeline:

# GitHub Actions example
- name: Run conformance testkit
  run: pnpm verify

This ensures the implementation stays compliant as the codebase evolves.

Coverage

The testkit covers:

• Canonicalization edge cases: Unicode, floats, nested structures, key ordering, empty values, YAML anchors
• All 5 check types: file_presence, path_convention, manifest_policy, regex, command_runner
• All 3 severity levels: MUST, SHOULD, MAY
• Schema validation: Required fields, pattern matching, type checking
• Integrity verification: Hash computation and comparison
• Dependency chains: Align dependencies and resolution

Exit Codes

• 0 - All tests passed
• 1 - One or more tests failed

Troubleshooting

All canonicalization tests fail

• Verify you're using JCS (RFC 8785), not standard JSON.stringify()
• Check key ordering (lexicographic sort)
• Verify floating point handling (0.0 → 0, preserve precision)

Hash mismatches

• Ensure you're hashing the JCS string, not the original input
• Use SHA-256 with hex output (lowercase)
• Check for trailing newlines or encoding issues

Golden align validation fails

• Verify you're excluding integrity.value when computing the hash
• Parse YAML to JSON before canonicalization
• Check that your schema matches packages/schema/schema/align.schema.json

Check vector failures

• Verify you're implementing the FileProvider abstraction correctly
• Check that glob patterns match the spec (minimatch syntax)
• For command_runner, ensure execution is properly gated by allow_exec

License

MIT

See Also

• Align Spec v1
• packages/schema - JSON Schema and canonicalization
• packages/checks - Check runner implementation
• AlignTrue/aligns - Production aligns