json-shredder

Reduce token tax in AI development by shrinking JSON payloads safely.

json-shredder preprocesses JSON for LLM prompts by compacting structure, trimming low-value content, and redacting sensitive fields before data ever reaches a model.

Why this package exists

Teams often send raw logs, API responses, and nested objects directly to LLMs. That increases:

• Token cost
• Latency
• Context window pressure
• Risk of leaking secrets

json-shredder addresses all four by default.

Research basis

Design decisions for this package were informed by:

• OpenAI token guidance: token usage drives cost and context limits; rough rule is 1 token is approximately 4 characters.
• OpenAI token-counting cookbook: practical token estimation and message-overhead awareness.
• Anthropic prompt engineering docs: establish measurable success criteria and iteratively optimize prompts.
• OWASP logging guidance: never log secrets or access tokens directly; prefer masking/redaction.

Install

npm install json-shredder

Quick start

const { shredJson } = require("json-shredder");

const payload = {
  event: "checkout",
  authorization: "Bearer sensitive",
  user: {
    id: "u_42",
    bio: "A very long profile string ...",
  },
  logs: ["...many lines..."],
};

const result = shredJson(payload, {
  maxStringLength: 140,
  maxArrayItems: 20,
  dropPaths: ["logs"],
  keepPaths: ["event", "user.id"],
});

console.log(result.json);
console.log(result.report);

CLI usage

Shred a file:

json-shredder shred --in payload.json --out payload.shred.json --report

Shred from stdin:

cat payload.json | json-shredder shred --report

Estimate token savings:

json-shredder estimate --in payload.json

What gets optimized

1. Key sorting for stable, compact JSON output.
2. String truncation with explicit omission markers.
3. Array trimming with omitted item summary marker.
4. Depth limiting to prevent oversized nested payloads.
5. Key/path filtering to keep only prompt-relevant fields.
6. Secret redaction by default key set plus custom keys/paths.

Default security behavior

The following key names are redacted by default (case-insensitive):

• access_token
• api_key
• apikey
• authorization
• cookie
• id_token
• password
• refresh_token
• secret
• set-cookie
• token

Replacement value is [REDACTED] by default.

API

shredJson(input, options)

Returns:

• data: shredded JSON object
• json: serialized JSON string
• stats: before/after and operation counters
• report: human-readable summary

Options:

• maxArrayItems (default 24)
• maxDepth (default 8)
• maxStringLength (default 240)
• keepKeys (array)
• removeKeys (array)
• redactKeys (array)
• keepPaths (array of dot paths)
• dropPaths (array of dot paths)
• redactPaths (array of dot paths)
• removeEmptyStrings (boolean)
• removeNulls (boolean)
• dropEmptyObjects (default true)
• appendArraySummary (default true)
• sortKeys (default true)
• pretty (boolean)
• redactReplacement (string)

Path wildcard support:

• * matches any single segment.
• Example: events.*.raw.

shredToString(input, options)

Shortcut that returns only shredded JSON string.

estimateTokens(input)

Character-based estimate using ceil(chars / 4).

Use model-specific tokenizers for strict accounting in production billing systems.

Token-reduction workflow recommendation

1. Keep only fields needed for the model task.
2. Drop high-volume debug/raw data by path.
3. Redact secret and regulated fields.
4. Truncate long strings and cap arrays.
5. Track before/after estimate in CI to prevent prompt bloat regressions.

Security notes

• Redaction is best-effort and key/path based; validate against your own data classification rules.
• Avoid passing untrusted shredded output into code-execution contexts.
• Keep logs and prompt artifacts access-controlled.
• Review SECURITY.md for threat model and disclosure policy.

Development

Run tests:

npm test

Run audit:

npm run audit

Check publish contents:

npm run pack:check

Publish:

npm publish --access public

License

MIT