📄 PDF-to-JSON Converter

A revolutionary, high-fidelity, color-aware PDF parsing engine built for Node.js and the Browser. Specifically designed to transform raw PDFs into cleanly segmented, structurally perfect JSON layouts for modern PDF Editing applications.

📖 The Story: Why We Built This (And Why You Need It)

If you've ever tried to build a browser-based PDF Editor, you know the struggle.

We started exactly where you probably are right now: we wanted to create a professional, web-based PDF editing interface. We quickly hit a massive, expensive wall. True enterprise-grade PDF parsers (like muPDF) are heavily restricted behind commercial licenses, creating an impossible barrier to entry for solo-developers and small indie teams.

"No problem," we thought. "We'll just use open-source tools like pdfjs-dist to extract the text, and patch the background!"

We were wrong. It was a nightmare.

We quickly discovered that standard open-source extraction tools are inherently flawed for editing purposes because of how the PDF specification itself is designed. PDFs are not DOMs; they are chaotic streams of archaic drawing operators.

Here are the catastrophic roadblocks we faced:

1. The Invisible Color Tragedies: PDF.js aggressively chunks text based on font dictionaries. If a document has a sentence with a single red word, but that red word uses the exact same base font dictionary as the rest of the black sentence, the PDF parser essentially erases the color transition. The red text is silently swallowed by the black text into a singular, monolithic, un-splittable block.
2. The Bounding-Box Decapitations: Mathematical height provided by standard PDF text extractors does not account for glyph extremities. Traditional bounding boxes repeatedly "decapitated" tall capitals (like T, H) and clipped low-hanging descenders (like y, g, p).
3. The Multi-Column Catastrophe: Multi-column layouts and tabular data (like Payslips or Invoices) would merge horizontally. Address column 1 would fuse directly into Earnings column 2, destroying the structural integrity of the document.

We realized something profound: We cannot trust the PDF text stream dictionary. We must trust the pixels.

So, we separated our parsing logic from the editor and built this custom engine. pdf-to-json-converter is our gift to the open-source community. It represents weeks of bleeding-edge geometric math, canvas pixel-sweeping, and rendering pipelines designed to give developers a free, incredibly accurate parser so you can build your own editing layers.

📸 Asset Generation Guide (Instructions for the Maintainer)

💡 Note: You can remove this block of text once you upload the images. To make this README look world-class, please capture and link the following images in the placeholders below:

Image 1: The Broken Extraction (Before) How to generate: Take a screenshot of a complex PDF (like the payslip) where a colored block of text (like "PDF ONLY" in red) is forcibly merged with the surrounding black text into one massive, unseparated bounding box.

Image 2: The Perfect Segmentation (After) How to generate: Run parsePdfToJson with { debugOutline: true, patchOpacity: 0 }. Take a screenshot showing how WhatsApp:, 9932131990, and (PDF ONLY) now have their own perfectly separated red debug bounding boxes.

Image 3: The Blank Patch Canvas How to generate: Run parsePdfToJson with { patchOpacity: 1, debugOutline: false }. Take a screenshot showing the rendered background image where the text is flawlessly "erased" (patched) matching the surrounding background variance.

The Problem vs The Solution

![Insert Image 1 Link Here]
Before: Standard PDF extraction relies on archaic font dictionaries, gluing different colors and columns together into unmanageable blocks.

![Insert Image 2 Link Here]
After: pdf-to-json-converter literally sweeping the physical canvas pixels to mathematically and accurately split the text into logical, independent blocks.

Flawless In-Painting

![Insert Image 3 Link Here]
The native output background layer. The engine analyzes background variance and generates precise opacity patches to erase the baked-in text without destroying dotted-lines, borders, or layout integrity.

⚙️ How It Actually Works (The Physics of Parsing)

pdf-to-json-converter doesn't just blindly read the underlying text stream—it deeply analyzes both the geometric text layer and the visual pixel data.

graph TD
    A[Raw PDF Blob] --> B[PDF.js Base Extraction]
    B --> C[Background Canvas Rendering]
    C --> D[Visual Pixel Sweeping]
    D --> E[Geometric Text Clustering]
    E --> F[Statistical In-Painting]
    F --> G[Beautiful JSON & Clean Canvas]
    
    style A fill:#2d3436,stroke:#b2bec3,stroke-width:2px,color:#fff
    style G fill:#00b894,stroke:#55efc4,stroke-width:2px,color:#fff

1. Visual Pixel Sweeping (The Secret Sauce)

To combat PDF.js hiding inline color overrides, we realized we had to bypass the text dictionary entirely. Inside the pipeline, we pre-render a high-resolution version of the document on a hidden HTML5 Canvas. We then mathematically sweep the physical bounding boxes of the text, filtering out transparent space. We run Euclidian distance calculations against the localized background to discover the true, physical rendered color of the text chunk. This allows us to force a split whenever the physical color changes, perfectly isolating words even if they use the same underlying font object.

2. Geometric Text Clustering

We map scattered individual characters into horizontal TextLine vectors. By analyzing baseline variance, we calculate extreme-precision paddings: 1.05x ascent multipliers and 0.30x descent multipliers. This absolute mathematical padding ensures that glyphs are permanently protected from being clipped by bounding boxes. Large horizontal spacing triggers gap algorithms to separate tabular data into independent columns.

3. Statistical In-Painting (Patches)

Once the text is isolated, the engine analyzes the pixels behind and around the bounding box. It runs a statistical color variance analysis to categorize the background as uniform, semi-uniform, or complex. It then dynamically generates precise bounding "patches" to erase the original baked-in text, yielding a perfectly clean, native-looking background image layer ready for an editor's DOM elements to sit on top of.

🎯 Use Cases: What Can You Build with This?

Because this engine natively decouples the text layout from the document background, you can power deeply complex applications:

• Browser-Based PDF Editors: Map the extracted JSON coordinates directly to React/Vue <input> wrappers or Fabric.js elements. The background image acts as the canvas, while the JSON builds the interactive DOM layer. (This is what it was originally built for!).
• Invoice & Receipt Data Extraction: With strict column-splitting, tabular data (like payslips and invoices) retains its tabular structure in the JSON arrays, making programmatic scraping highly reliable.
• Intelligent Auto-Translation: Swap the text field in the JSON with a translated string, and render it back over the patched background image. You get a fully translated document that looks like the original.
• Redaction Tools: Accurately locate PII or sensitive text nodes and render blackout rectangles over the exact [x, y, w, h] arrays.

🚀 Installation & Setup

Install via NPM:

npm install pdf-to-json-converter pdfjs-dist

Environment Warning: Node.js vs Browser

Because pdf-to-json-converter relies on high-resolution pixel sampling to guarantee color and formatting accuracy, it requires a Canvas environment. Standard web browsers provide this natively out-of-the-box.

If you are running this in a Node.js architecture (e.g., Express handlers, background workers, Next.js server actions), you must install a canvas backend:

# Recommended (Pre-built Rust binaries, insanely fast, no native compilation tools needed)
npm install @napi-rs/canvas

# Alternative (Requires system build tools like python & node-gyp)
npm install canvas

💻 Usage Code

In the Browser (React / Next.js Client / Vue)

import parsePdfToJson, { configurePdfWorkerSrc } from 'pdf-to-json-converter';

// IMPORTANT: Set up the PDF.js web worker securely
configurePdfWorkerSrc('https://cdnjs.cloudflare.com/ajax/libs/pdf.js/4.4.168/pdf.worker.min.mjs');

async function handleFileUpload(file: File) {
  try {
    const jsonOutput = await parsePdfToJson(file, {
      renderScale: 2,         // Resolution of the background image
      patchOpacity: 1,        // Fully hide the baked-in text
      useJpegBackground: true,// Reduces payload size
      jpegQuality: 0.85,
    });

    console.log("Segmented Layout:", jsonOutput);
    // You can now render `<input>` or text layers over the `jsonOutput.pages[0].background`
  } catch (error) {
    console.error("Failed to parse PDF:", error);
  }
}

In Node.js (Express / Edge / Serverless)

The library automatically detects the Node environment and injects @napi-rs/canvas or canvas.

import fs from 'fs';
import parsePdfToJson from 'pdf-to-json-converter';

async function processPdfServerSide() {
  const fileBuffer = fs.readFileSync('./payslip.pdf');
  
  const result = await parsePdfToJson(fileBuffer, {
    binaryBackground: true, // Returns the background image as a Uint8Array instead of a heavy base64 string
    renderScale: 2
  });

  fs.writeFileSync('./output.json', JSON.stringify(result, null, 2));
}

processPdfServerSide();

🛠 API & Debugging Options

The parsePdfToJson(input, options) function accepts a configuration object to tune the extraction algorithm:

| Option | Type | Default | Description | |--------|------|---------|-------------| | renderScale | number | 2 | Resolution multiplier for the extracted background image and canvas pixel sampling. Higher scale = superior color detection accuracy but heavier CPU/RAM usage. | | patchOpacity | number | 1.0 | Opacity of the patches covering the original PDF text. Set to 0 or 0.5 to make patches transparent for visually verifying your extraction alignments. | | debugOutline | boolean | false | Crucial Debugger: Draws distinct red bounding boxes (1px) around every generated sub-region on the background image. Let's you visually see exactly how the engine sliced the text! | | useJpegBackground| boolean | false | Encodes the resulting background layer as a JPEG instead of a PNG for huge payload size reductions. | | binaryBackground | boolean | false | If true, returns the background image as a raw Uint8Array rather than a base64 Data URL. Ideal for efficient backend storage or proxying. | | verbose | boolean | false | Dumps extensive coordinate, variance, gap sizes, and color samples into your system console. |

📊 Output JSON Schema

The engine outputs a highly structured, CSS-friendly document map.

{
  "meta": {
    "pageCount": 1
  },
  "pages": [
    {
      "width": 612,
      "height": 792,
      "rotation": 0,
      "background": {
        "type": "image",
        "data": "data:image/jpeg;base64,...",
        "format": "jpeg"
      },
      "texts": [
        {
          "text": "Amount Due: $1,500.00",
          "x": 46.4,
          "y": 136.4,
          "width": 133.9,
          "height": 19.2,
          "fontSize": 12,
          "fontFamily": "Helvetica, Arial, sans-serif",
          "fontWeight": "bold",
          "fontStyle": "normal",
          "color": "#121212",
          "transform": [12, 0, 0, 12, 50, 640],
          "confidenceScore": 1.0,
          "items": [ /* Raw chunks */ ]
        }
      ]
    }
  ]
}

🌟 Contributing & Community

We believe the open-source community deserves access to high-fidelity PDF editing tools without being strong-armed by massive commercial licensing fees. This project is a labor of love to break down those walls.

If you use this engine and feel the same sense of triumph when you finally see those flawless red bounding boxes segmenting your data perfectly, we'd love your help to keep making it better!

Known areas for improvement:

• Heavily vectorized (non-text) PDFs or image-based scanned PDFs will not yield text items and require an OCR bridge (like Tesseract).
• Background patching attempts logical variance inference. Highly complex, hyper-gradient graphics behind text may reveal slight artifacting when patched. Let's improve the statistical variance mapping together.
• Geometric gap detection for better complex CJK language line-wrapping.

Drop a star, open an issue, or fork it to start conquering PDFs!

(Powered by modern Web APIs and the incredible roots of PDF.js)