pdf2gep

Convert a PDF document into GEP (Genome Evolution Protocol) assets suitable for retrieval inside the EvoMap network.

pdf2gep fetches a PDF (local path or URL), splits the text into chunks, and writes one GEP bundle per chunk:

• An explore Gene -- a compact retrieval pointer (category: "explore").
• A reference Capsule (source_type: "reference") -- the chunk text itself, carried as reference material.

Both assets are fully schema-valid GEP (validated against @evomap/gep-sdk) and carry a real, Hub-recomputable asset_id. Earlier versions used sentinel values (category: "knowledge_reference", outcome.status: "knowledge_reference", a _source side-channel) that did not validate against the strict protocol schema — see the v2 note below.

Honest scope note (please read before using)

pdf2gep is a retrieval-oriented protocol adapter. It does not produce the kind of Capsule that proves a Gene works.

• A standard GEP Capsule is an auditable record of one real execution of a Gene (execution_trace with exit codes, non-zero blast_radius, etc.). PDFs contain knowledge, not executions, so pdf2gep marks its capsules with the protocol's own reference marker: source_type = "reference", an empty execution_trace, and a zero blast_radius. outcome.status = "success" here means only "the reference chunk was extracted", not that any task passed. Treating these as proof-of-validation is a misuse.
• The paper that motivates GEP -- Wang, Ren, Zhang, "From Procedural Skills to Strategy Genes: Towards Experience-Driven Test-Time Evolution" (arXiv:2604.15097) -- validates Gene-as-control-interface on 45 scientific code-solving tasks with Gemini 3.1 Pro and Flash Lite. That result does not carry over automatically to retrieval-style knowledge Genes. The Gene emitted by this tool is explicitly a retrieval pointer, not a control interface.
• Chunk quality is naive: fixed-width ~4000-code-point slices, without splitting surrogate pairs. Whitespace-only slices are skipped as layout separators; retained slices are unchanged. This is fine for retrieval-by-topic, but it is not a structured extraction. Do not expect the output to replace a proper RAG ingestion pipeline.

Downstream consumers (EvoMap hub, local agents) should filter on source_type === "reference" and treat these Capsules as reference material only.

Install

v2 release contract: @evomap/[email protected] pins the published @evomap/[email protected] contract from SDK PR #21. Clean registry-backed dependency installation, Node 18/22 regression tests, and installed CLI verification have passed. pdf2gep 1.x does not include this reference-only contract or the isolated output layout; read the migration notes before upgrading. Hub/Evolver acceptance still requires their coordinated reference-only support; generating files does not publish them.

From npm

npm install -g @evomap/pdf2gep

This installs the pdf2gep CLI globally. Requires Node.js 18+ (for built-in fetch).

For one-off use, npx works without a global install:

npx @evomap/pdf2gep "https://arxiv.org/pdf/2604.15097.pdf"

Alternative: from source

git clone https://github.com/EvoMap/pdf2gep.git
cd pdf2gep
npm install

Usage

After npm install -g @evomap/pdf2gep:

# From a URL (arXiv, etc.)
pdf2gep "https://arxiv.org/pdf/2604.15097.pdf"

# From a local file
pdf2gep "./manual.pdf"

When working from a source checkout, the equivalent is node index.js "<url-or-path>".

Each conversion is staged and atomically published as an isolated run_<timestamp>_<id>/ directory under temp/evomap_assets/ (or the directory passed to --output-dir). A completed run contains batch.json, manifest.json, and one gene_<asset-id>.json plus one capsule_<asset-id>.json file per pair. Runs do not share mutable files. This intentionally replaces the earlier flat output layout; consumers that globbed the output root must discover run_*/batch.json instead. The CLI prints the completed batch and manifest paths.

• Failure and retry: handled write/rename failures remove the private staging directory and preserve earlier runs. If cleanup itself fails, both errors are reported. SIGKILL or a host crash can leave a hidden .run_*.tmp directory; consumers must ignore it. A retry creates a separate run and never deletes another run's staging. Remove an orphan only after confirming that its writer is no longer running. Atomic rename guarantees visibility of complete runs, not power-loss durability (fsync is not performed).
• Provenance: local paths and file: URIs become basename-only references. HTTP(S) references omit credentials, query parameters, and fragments. If a query identifies a distinct document, set a non-secret opaque --source-ref, such as manual-v1. This sanitizes provenance metadata, not the document's text; review content before publishing.
• Options: --chunk-size must be a positive integer. Missing option values and unknown options fail before reading the PDF.

pdf2gep ./manual.pdf --chunk-size 4000 --output-dir ./out --source-ref manual-v1

Library API

pdf2gep also exposes its building blocks for programmatic use:

const {
  chunkText,
  createGene,
  createReferenceCapsule, // alias: createKnowledgeCapsule (kept for back-compat)
  processChunk,           // async — computes the asset_id via @evomap/gep-sdk
} = require('@evomap/pdf2gep');

createGene / createReferenceCapsule are pure, synchronous builders (they take a schemaVersion argument and do not set asset_id). processChunk is async: it loads @evomap/gep-sdk, stamps each asset's schema_version from the SDK's SCHEMA_VERSION, and computes a Hub-valid asset_id via computeAssetId. The exported helpers are documented inline in index.js.

Output schema

Assets validate against the pinned @evomap/[email protected] Gene/Capsule schemas; schema_version comes from the SDK at runtime. Schemas are loaded once. The local validator implements the keywords used by this pinned contract and rejects unknown keywords rather than silently ignoring future constraints; it is not a general-purpose JSON Schema engine. processChunk additionally invokes the SDK reference-only classifier to verify UTF-8 hash/byte-size evidence.

Gene (category: "explore")

{
  "type": "Gene",
  "schema_version": "<from @evomap/gep-sdk SCHEMA_VERSION>",
  "id": "gene_pdf2gep_<slug>_chunk<N>_<sha8>",
  "category": "explore",
  "signals_match": ["knowledge_lookup", "pdf_reference", "<slug>"],
  "preconditions": ["Agent needs to consult the source document to answer or plan."],
  "strategy": [
    "Retrieve the backing reference Capsule (source_type=reference) to read the chunk verbatim.",
    "Treat the chunk as reference material only -- it is NOT a validated procedure."
  ],
  "constraints": { "max_files": 1, "forbidden_paths": [".git", "node_modules"] },
  "validation": ["node -e \"<self-contained SHA-256 verifier>\" <chunk_sha256>"],
  "summary": "Reference pointer for <slug> chunk #<N> (sha256:<sha12>) extracted from <source>.",
  "asset_id": "sha256:<64 hex>"
}

validation is a self-contained Node.js reference-integrity check (pipe the chunk in, confirm its sha256 matches); it does not require pdf2gep to be installed on the consuming machine. The helper validates the interpolated digest before generating the command, so caller-controlled shell text cannot enter it. This proves the reference is intact, not that a task ran.

Reference Capsule (source_type: "reference")

{
  "type": "Capsule",
  "schema_version": "<from @evomap/gep-sdk SCHEMA_VERSION>",
  "id": "cap_pdf2gep_<chunk_sha12>_<idkey>",
  "gene": "<gene.id>",
  "trigger": ["knowledge_lookup", "pdf_reference", "<slug>"],
  "summary": "PDF chunk #<N> from <name> (reference material).",
  "confidence": 1,
  "blast_radius": { "files": 0, "lines": 0 },
  "outcome": { "status": "success", "score": 1 },
  "success_reason": "Reference chunk extracted verbatim and attested by content hash.",
  "env_fingerprint": { "platform": "...", "node": "..." },
  "source_type": "reference",
  "strategy": ["...copied from the Gene..."],
  "content": {
    "text": "<chunk text verbatim>",
    "mime": "text/plain",
    "source_ref": "file:manual.pdf",
    "source_sha256": "<sha256 of the whole pdf>",
    "chunk_index": 0,
    "chunk_sha256": "<sha256 of this chunk>",
    "claims_outside_scope": "knowledge_extraction"
  },
  "evidence_mode": "reference_only",
  "proof_of_work": { "kind": "artifact_hash", "artifact_hash": { "sha256": "<sha256 of UTF-8 content.text>", "mime": "text/plain", "size": 123 } },
  "execution_trace": [],
  "asset_id": "sha256:<64 hex>"
}

Key invariants validators can rely on:

• source_type === "reference" — the canonical marker for extracted/cited knowledge.
• execution_trace is empty and blast_radius is { files: 0, lines: 0 } — no Gene was executed.
• outcome.status === "success" means "reference extracted", not "task validated"; always read it together with source_type.
• The chunk text and provenance live inside the content object (a real object, not a bare string).
• asset_id recomputes correctly under @evomap/gep-sdk's verifyAssetId.

Publishing to EvoMap

Use the EvoMap publisher with the generated batch file as its input. The exact publisher command is version-specific; this package does not ship or invoke the publisher.

v2 migration note

@evomap/pdf2gep v2 changes the output format to be strictly schema-valid GEP. If you have a consumer built against v1:

| v1 (sentinel, non-conforming) | v2 (schema-valid) | |---|---| | gene.category: "knowledge_reference" | gene.category: "explore" | | gene._source.{...} | provenance moved into capsule.content.{...} | | gene.validation: [], max_files: 0 | a real integrity check; max_files: 1 | | capsule.outcome.status: "knowledge_reference" | capsule.outcome.status: "success" + source_type: "reference" | | capsule.content: "<string>" | capsule.content: { text, ... } (object) | | capsule.source_type: "pdf_knowledge" | capsule.source_type: "reference" | | capsule.blast_radius.chunk_chars | dropped (use content.text.length) | | no asset_id | real asset_id via @evomap/gep-sdk | | flat batch_<timestamp>.json plus shared manifest.json | isolated run_<timestamp>_<id>/batch.json plus per-run manifest |

Filter on source_type === "reference" instead of "pdf_knowledge".

See also:

• Protocol reference: https://evomap.ai/wiki/16-gep-protocol
• Skill store (where the Gene shows up): https://evomap.ai/wiki/31-skill-store

Relationship to other tools

• skill2gep -- protocol adapter that converts SKILL.md into Gene+ExecutionCapsule bundles. That tool is for procedural knowledge where the Capsule's execution_trace comes from real runs. pdf2gep is complementary: it covers reference knowledge and deliberately does not fabricate execution evidence.
• kitchen-engineer42/pdf2skills -- prior art that inspired this tool. pdf2skills targets Claude Code's SKILL.md format; pdf2gep targets the GEP protocol and is explicit about being retrieval-only.

License

MIT. See LICENSE.