Sensitive Data Sanitizer

An offline JavaScript library, detector framework, and CLI for redacting credentials and personal data from logs, AI sessions, and text files, including logs with invalid UTF-8 bytes. It combines required Secretlint and 221 bundled Gitleaks text rules with contextual credentials, default multilingual name/PII recognition, known values and bounded encoding detection. Optional local Gitleaks, TruffleHog, and Presidio detectors extend the union.

Library calls without a profile replace every actionable finding completely with [REDACTED]. Personal findings below the default confidence threshold of 0.5 remain inspectable without replacing ordinary log numbers; credentials always qualify. Publication methods verify full redaction and block output if a required detector fails or finds a residual. Optional typed transformations, compatible partial masks and encoding preservation are described in the API. Findings contain offsets and rule metadata; they omit matched values and source excerpts.

For publishing sessions, use createSanitizer({profile: 'publication'}): the personal threshold is 0.3, identity values use the fixed first/last-two *** mask, and fake-identity exemptions remain disabled. CLI redact defaults to this profile. Explicit configuration such as identityMask: false selects complete redaction. Credentials always receive complete redaction unless an explicit credential transformation is selected.

Detection has limits: the default 17-locale name gazetteer has finite vocabulary; unfamiliar names need an additional NER model or known-personal dictionary. New token formats, unsupported encodings, and unconfigured languages can escape detection. Successful verification means the enabled detectors found no residual, not that the input is universally anonymous. See coverage and limits and the issue #1 case study.

For real session files whose personal values are confirmed later, use atomic two-pass publication:

sensitive-data-sanitizer redact session.jsonl --stream --jsonl --two-pass --output safe.jsonl

For synchronous log hooks, createSyncSanitizer supports custom detectors and verified callback transformations. Rule/type controls, caller vocabulary, structured field selectors and compileKnown configure reusable policies; see API examples and the runnable log policy. The optional /ner entry adds offline Russian/English symbolic recognition with published coverage and cost.

Quick Start

For stored prose that can tolerate extra redaction, opt into structural proper names independently for each string value:

sanitize('Работал в Ромашке менеджером по продажам.', {
  profile: 'documents',
  properNames: 'capitalized',
}).text;
// [REDACTED] в [REDACTED] менеджером по продажам.

The rule defaults to Cyrillic or structural data triggers, keeps JSON keys and numbers, and supports caller-selected Unicode scripts. It is off by default. See activation rules and limits and the prose-log example. CLI: redact input.jsonl --jsonl --proper-names capitalized --profile documents.

Readable placeholders and document profiles

Use the opt-in documents profile for CVs and similar prose. It adds bounded employer, school, city and region detection across the existing 17 locales. Default and publication log profiles keep their existing detection behavior.

import { sanitize, restore } from '@link-foundation/sensitive-data-sanitizer';

const result = sanitize('Jane Doe worked at Acme Ltd in Springfield.', {
  profile: 'documents',
  transformation: { mode: 'placeholder' },
});
// result.text: Person-1 worked at Organization-1 in Location-1.
// Keep result.map private. Share only result.text.
restore(result.text, result.map); // exact original document
restore('Ask Person-1 about Organization-1.', result.map); // derived answer

The restore map is personal data. It contains original spellings and private occurrence metadata; never commit it, send it to a model, or publish the whole result object. The CLI writes maps separately with mode 0600, and publication checks reject serialized restore maps. Reuse result.map in the next transformation.map to keep IDs stable across files. JSON and Links Notation serializations carry the same versioned map. Credentials stay fully redacted and out of the map unless explicitly opted in.

sensitive-data-sanitizer redact cv.txt --profile documents --placeholder --map private.lino --output safe.txt
sensitive-data-sanitizer restore safe.txt --map private.lino --output restored.txt
sensitive-data-sanitizer redact cv.txt --profile documents --placeholder --no-map

knownOnly: true (CLI --known-only) limits replacement to knownPersonal, knownSecrets and reused map identities, preserving fictional examples. Placeholder results report irreversibleRedactions; a nonzero count means some replacements, such as credentials excluded from the map, cannot be restored.

Known Russian names match in all six cases and in any word order, with ё/е spelling variants and initials (с Петровым Данилом Сергеевичем, Петров Д. С.). A part of a longer name is matched alone only when it is listed itself.

For irreversible output, choose transformation: { mode: 'placeholder', map: false }: no restore map is returned or written. Detection limits still apply in either mode. See the placeholder API, coverage, and runnable example.

Sanitizer APIs require Node.js 22.13 or later. Template arithmetic utilities are available under @link-foundation/sensitive-data-sanitizer/legacy; the existing universal example uses that browser-compatible entry. From this checkout:

npm ci
node bin/sensitive-data-sanitizer.js redact session.jsonl --output session.safe.jsonl
node bin/sensitive-data-sanitizer.js scan ./logs
node bin/sensitive-data-sanitizer.js history ./repository

The npm package is @link-foundation/sensitive-data-sanitizer; the installed executable is sensitive-data-sanitizer. This PR prepares its release through Changesets.

First npm publication currently requires owner setup; see the verified release blockers and exact setup steps.

import {
  createSanitizer,
  knownSecretsFromEnv,
} from '@link-foundation/sensitive-data-sanitizer';

const sanitizer = createSanitizer({
  profile: 'publication',
  knownSecrets: knownSecretsFromEnv(), // Explicitly include local credentials.
});
const result = await sanitizer.sanitize('password=short [email protected]');
console.log(result.text); // password=[REDACTED] [REDACTED]

createSanitizer() requires the pinned Secretlint recommended rules to succeed. The synchronous sanitize() and inspect() functions use the native rules only. redact() applies supplied findings without a residual scan. Prefer the asynchronous sanitizer for publication.

CLI

printf 'password=short\n' | sensitive-data-sanitizer redact -
sensitive-data-sanitizer redact input.txt --in-place
sensitive-data-sanitizer scan logs --config /private/policy.json
sensitive-data-sanitizer redact input.txt --gitleaks --trufflehog --output safe.txt
sensitive-data-sanitizer redact input.txt --presidio /path/presidio-bridge.py --model en_core_web_sm --language en
sensitive-data-sanitizer redact large-session.jsonl --jsonl --stream --output safe-session.jsonl
sensitive-data-sanitizer history rewrite ./repository --output ./private-preview
sensitive-data-sanitizer --help

--output creates a new file without clobbering an existing one. --in-place replaces the input atomically after verification. Intermediate and output files use mode 0600 on POSIX. Symlinks are refused. NUL-containing input is refused by default; --allow-binary explicitly preserves NUL bytes in plain logs while credentials are still sanitized. JSON/JSONL remain strict. Plain logs preserve invalid UTF-8 bytes exactly; JSON/JSONL and configuration files require valid UTF-8. Directory scans omit .git and node_modules; skipped binary/malformed-structured/oversized files make the report incomplete. Stdout is written after the complete operation succeeds.

| Exit | Meaning | | ---- | ------------------------------------------------------------------------ | | 0 | Redaction succeeded, or a complete scan found nothing | | 1 | A complete scan/history audit found sensitive data | | 2 | Error, detector failure, or incomplete scan; redaction output is blocked |

Defaults: 32 MiB whole-text input bytes, 100,000 findings and 10,000 files. --stream permits 1 GiB total with plain-text records up to the total input limit or 8 MiB JSONL records, bounded workers and atomic publication. --worker-timeout-ms N sets the worker inactivity/per-batch deadline (default 60000 ms); verified file progress resets the deadline, so large files can run longer overall. --max-bytes sets the byte limit; JSON config sets the other sanitizer limits. --paranoid adds entropy heuristics and increases false positives. --native-only explicitly disables Secretlint.

Configuration

For AI sessions, --jsonl preserves JSON records and declares id, type, name, and description as metadata. --json handles a single JSON document; use JSONL for streaming. Known private values and credential formats still override declarations. Set structuralFields in private config to choose your own property names, or use an empty array to scan every field normally. The library provides structured: 'json' | 'jsonl' and engine.sanitizeJsonl(text) with an empty structural policy by default.

For input above the whole-text limit, pass an iterable to sanitizeStream(source, {structured: 'jsonl', profile: 'publication'}) or use sanitizeStreamToFile for atomic output. JSONL defaults to two ordered record workers. See the streaming example.

Native coverage includes tolerant passport MRZ fragments, country passport formats, PNR/ticket/visa identifiers, contextual birth dates, national IDs and messenger phone numbers. identityMask: true (the publication/CLI redact default) uses 2+2 characters with a fixed *** marker. Opt-in fake identity policies retain audited specimen/synthetic values while credentials remain protected. See the API and all issue 13 requirements and research.

transformation: {mode: 'fake', key} generates deterministic names, valid dates/MRZs/checksummed IDs and reserved email/domain destinations. Credentials remain fully redacted by default. Keep the key private: this preserves relations and structure and is pseudonymisation. Russian mentions of one known person or entity share one fake identity declined to match the source (Петров … С Петровым becomes Баранов … С Барановым), and pools overrides the entity names. See realistic fakes and the complete issue 24 requirements, research and solution plans.

Keep policies containing real private values outside Git. Built-in exact public knowledge and narrowly scoped role/domain heuristics are enabled offline. Explicit private literals override public exemptions. Keep custom public policies reviewed against evidence; --verify-public separately opts into Wikidata name lookup:

{
  "knownSecrets": [],
  "knownPersonal": [{ "type": "PERSON", "value": "Private Person" }],
  "publicEntities": [
    {
      "type": "EMAIL",
      "value": "[email protected]",
      "source": "https://example.org/contact",
      "reviewedAt": "2026-10-07"
    }
  ]
}

Custom policies preserve exact reviewed PERSON, ORGANIZATION and EMAIL detections. Built-in public resolvers/documentation IPs are also exempt. Credential detections always win, including a public email used as a password or authorization value. Evidence is an explicit caller decision and does not make every occurrence harmless.

Extend detection

import {
  createSanitizer,
  createGitleaksDetector,
  createTrufflehogDetector,
  createPresidioDetector,
} from '@link-foundation/sensitive-data-sanitizer';

const sanitizer = createSanitizer({
  detectors: [
    createGitleaksDetector(),
    createTrufflehogDetector(),
    createPresidioDetector({ args: ['/absolute/path/presidio-bridge.py'] }),
  ],
});

Install external binaries and language models separately. TruffleHog runs with verification and updates disabled. The Presidio example checks for an installed model and never downloads it at analysis time. Each selected detector is required, including during residual verification. See the API and adapter contract, combined scanners, and the local NER bridge.

The framework also accepts Gitleaks, TruffleHog, detect-secrets, Presidio, or explicitly typed offset reports. Cloud reports can be converted locally; the package itself makes no cloud requests.

For CLI name recognition, --presidio PATH selects a local Python JSON bridge. --python COMMAND selects its interpreter, and --model / --language select an already installed model. These options require the bridge and participate in final-output verification.

Git history

history audits reachable blobs, commit messages/authors, annotated tags, and reflog-reachable objects without changing refs. It can find deleted files. It reports object hashes and safe findings, and refuses limits or reports skipped binary objects as incomplete. The history rewrite SOURCE --output NEW_DIRECTORY command previews private replacement/mailmap files in a fresh mirror clone; add --apply to rewrite only that clone and re-audit blobs, metadata, paths and refs. It never pushes. Remote caches, forks, artifacts and unreachable objects require separate remediation; see the history guide.

Contributing

npm test
npm run check
bun test --timeout 30000
deno test --allow-read --allow-env

See contributing. add, multiply and delay live in the legacy subpath, with the sensitive-data-sanitizer-legacy CLI and retained universal app; inherited pipeline instructions are in the template guide.

Auto-regenerated preview screenshots

The universal app's preview images regenerate on pushes to the default branch. Run npm run example:web:preview-images to regenerate them locally; see the template guide for setup.

Automatic --env-secrets discovery skips authentication configuration names such as GIT_AUTH_TRANSPORT, AUTH_MODE and AUTH_METHOD. Explicit knownSecretsFromEnv(env, names) selections still include their exact values.

For shared tool logs, --credentials-only --env-secrets --gh-config preserves public account details while redacting credential patterns and exact local secret values. Environment discovery matches credential-related variable names, including AI-provider keys; --gh-config reads every host/account token from the local GitHub CLI hosts.yml. Both local sources are opt-in. Literal references such as ${{ secrets.NPM_TOKEN }} and $(gh auth token) remain readable; actual credential values are still redacted.

Long plain-text records are held until a safe boundary, including credentials longer than a 64 KiB input chunk. Memory grows with the largest held record; set --max-record-bytes for a smaller deployment budget. Crossing an explicit record or total limit blocks atomic publication. See the tool-log integration example and API byte handling.

License

Released under the Unlicense. External components keep their own licenses; see component provenance.