pdf-inspector README

The project's README from github.com/firecrawl/pdf-inspector. As one markdown file: README.md.

pdf-inspector

Crates.io npm PyPI License: MIT

Fast Rust library for PDF classification and text extraction. By default it detects whether a PDF is text-based or scanned, extracts text with position awareness, and converts to clean Markdown without OCR. Native Rust and CLI consumers can opt into selective OCR. Includes bindings for Python, Node.js, and browser WebAssembly.

Built by Firecrawl to handle text-based PDFs locally in under 200ms, skipping expensive OCR services for the ~54% of PDFs that don't need them.

Features

Benchmark

Evaluated on the opendataloader-bench corpus (200 PDFs). Only local engines without model-based PDF parsing are shown; OCR was disabled. Scores are 0-1, higher is better.

Engine Overall Reading Order (NID) Tables (TEDS) Headings (MHS) Speed (200 docs)
pdf-inspector 0.875 0.915 0.814 0.788 0.470s
liteparse 0.873 0.913 0.693 0.811 0.750s
opendataloader 0.831 0.902 0.489 0.739 2.569s
pymupdf4llm 0.735 0.886 0.401 0.424 17.117s
markitdown 0.589 0.844 0.273 0.000 16.165s

Results were refreshed on July 31, 2026, on an Apple M4 Pro. Engine versions were pdf-inspector 0.2.6, LiteParse 2.10.1, OpenDataLoader 2.2.1, PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.5. Speed is the median of five alternating or rotating complete corpus runs after an excluded warm-up run, with each parser processing documents sequentially in a single process.

The complete parser configuration, per-document predictions, evaluator output, and generated charts are available in the reproducible results branch.

Best fit: Native-text PDFs where speed, reading order, and table structure matter. In this comparison, pdf-inspector delivered the higher overall, reading-order, and table scores, along with the fastest complete run. That makes it a strong local default for reports, research papers, financial documents, invoices, and legal PDFs that need clean, structured Markdown without adding OCR latency or infrastructure.

Use the paired benchmark harness to compare two local builds against the exact same corpus and evaluator revision.

Quick start

Python

pip install pdf-inspector

import pdf_inspector

result = pdf_inspector.process_pdf("document.pdf")
print(result.pdf_type)   # "text_based", "scanned", "image_based", "mixed"
print(result.markdown)   # Markdown string or None

# Selective OCR; clean text PDFs do not load the external OCR runtime.
ocr = pdf_inspector.process_pdf_with_ocr("document.pdf")
print(ocr.pages_routed_to_ocr)

Full API reference: docs/python.md

Node.js

npm install @firecrawl/pdf-inspector

import { readFileSync } from 'fs';
import { processPdf, processPdfWithOcr } from '@firecrawl/pdf-inspector';

const pdf = readFileSync('document.pdf');
const result = processPdf(pdf);
console.log(result.pdfType);   // "TextBased", "Scanned", "ImageBased", "Mixed"
console.log(result.markdown);  // Markdown string or null

const ocr = await processPdfWithOcr(pdf); // selective OCR, off the event loop
console.log(ocr.pagesRoutedToOcr);

Full API reference: napi/README.md

Browser WebAssembly

npm install @firecrawl/pdf-inspector-wasm

import init, { processPdf } from '@firecrawl/pdf-inspector-wasm';

await init();
const response = await fetch('/document.pdf');
const pdf = new Uint8Array(await response.arrayBuffer());
const result = processPdf(pdf);

console.log(result.pdfType);
console.log(result.markdown);

Full API reference: wasm/README.md

Rust

Install from crates.io:

cargo add pdf-inspector

Or add it manually:

[dependencies]
pdf-inspector = "1"

use pdf_inspector::process_pdf;

let result = process_pdf("document.pdf")?;
println!("Type: {:?}", result.pdf_type);
if let Some(markdown) = &result.markdown {
    println!("{}", markdown);
}

Full API reference: docs/rust-api.md

CLI

# Install the CLI tools
cargo install pdf-inspector

# Convert PDF to Markdown
pdf2md document.pdf

# JSON output (for piping)
pdf2md document.pdf --json

# Positioned TextItem JSON (coordinates relative to the visible page box): axis-aligned box, rotation, font, underline metadata
pdf2md document.pdf --items-json

# Raw markdown only (no headers)
pdf2md document.pdf --raw

# Token-efficient output (collapses long dot leaders and similar source padding)
pdf2md document.pdf --compact

# Insert page break markers (<!-- Page N -->)
pdf2md document.pdf --pages

# Process only specific pages
pdf2md document.pdf --select-pages 1,3,5-10

# Detection only (no extraction)
detect-pdf document.pdf
detect-pdf document.pdf --json

# Detection + layout analysis (tables, columns)
detect-pdf document.pdf --analyze --json

Rust and CLI consumers opt into OCR at build time:

cargo install pdf-inspector --features ocr --bin pdf2md
PDFIUM_LIB_PATH=/path/to/libpdfium ORT_DYLIB_PATH=/path/to/libonnxruntime \
  pdf2md scan.pdf --ocr auto --json

The OCR JSON envelope is versioned and reports routed pages, per-page source and confidence, warnings, and pages recommended for the hosted document pipeline. Native Python and Node packages expose the same pipeline without a source-build feature. All native entry points still require separately installed PDFium and ONNX Runtime libraries only when OCR is routed. See the OCR runtime setup guide for pinned downloads, platform support, model-cache behavior, and hosted-fallback integration. See the Rust API guide for lower-level controls.

From a source checkout, use cargo run --bin pdf2md -- document.pdf or cargo run --bin detect-pdf -- document.pdf instead.

Architecture

PDF bytes
  │
  ├─► detector         → PdfType (TextBased / Scanned / ImageBased / Mixed)
  │
  └─► extractor
        ├─ fonts        → font widths, encodings
        ├─ content_stream → walk PDF operators → TextItems + PdfRects
        ├─ xobjects     → Form XObject text, image placeholders
        ├─ links        → hyperlinks, AcroForm fields
        └─ layout       → column detection → line grouping → reading order
              │
              ├─► tables
              │     ├─ detect_rects      → rectangle-based tables (union-find)
              │     ├─ detect_heuristic  → alignment-based tables
              │     ├─ grid              → column/row assignment → cells
              │     └─ format            → cells → Markdown table
              │
              └─► markdown
                    ├─ analysis     → font stats, heading tiers
                    ├─ preprocess   → merge headings, drop caps
                    ├─ convert      → line loop + table/image insertion
                    ├─ classify     → captions, lists, code
                    └─ postprocess  → cleanup → final Markdown

The document is loaded once via load_document_from_path / load_document_from_mem and shared between the detection and extraction stages, so there's no redundant parsing.

Project structure

src/
  lib.rs                — Public API, PdfOptions builder, convenience functions
  python.rs             — PyO3 Python bindings
  types.rs              — Shared types: TextItem, TextLine, PdfRect, ItemType
  text_utils.rs         — Character/text helpers (CJK, RTL, ligatures, bold/italic)
  process_mode.rs       — ProcessMode enum (DetectOnly, Analyze, Full)
  detector.rs           — Fast PDF type detection without full document load
  glyph_names.rs        — Adobe Glyph List → Unicode mapping
  tounicode.rs          — ToUnicode CMap parsing for CID-encoded text
  extractor/            — Text extraction pipeline
  tables/               — Table detection and formatting
  markdown/             — Markdown conversion and structure detection
  bin/                  — CLI tools (pdf2md, detect_pdf)
napi/                   — Node.js/Bun bindings (napi-rs)
wasm/                   — Browser bindings (wasm-bindgen)

How classification works

  1. Parse the xref table and page tree (no full object load)
  2. Select pages based on ScanStrategy (default: all pages with early exit)
  3. Look for Tj/TJ (text operators) and Do (image operators) in content streams
  4. Classify based on text operator presence across sampled pages

This detects 300+ page PDFs in milliseconds. The result includes pages_needing_ocr — a list of specific page numbers that lack text, enabling per-page OCR routing instead of all-or-nothing.

Scan strategies

Strategy Behavior Best for
EarlyExit (default) Scan all pages, stop on first non-text page Pipelines routing TextBased PDFs to fast extraction
Full Scan all pages, no early exit Accurate Mixed vs Scanned classification
Sample(n) Sample n evenly distributed pages (first, last, middle) Very large PDFs where speed matters more than precision
Pages(vec) Only scan specific 1-indexed page numbers When the caller knows which pages to check

Markdown output

The converter handles:

Element How it's detected
Headings (H1-H4) Font size tiers relative to body text, with 0.5pt clustering
Bold/italic Font name patterns (Bold, Italic, Oblique)
Bullet lists •, -, *, ○, ●, ◦ prefixes
Numbered lists 1., 1), (1) patterns
Letter lists a., a), (a) patterns
Code blocks Monospace fonts (Courier, Consolas, Monaco, Menlo, Fira Code, JetBrains Mono) and keyword detection
Tables Rectangle-based detection from PDF drawing ops + heuristic detection from text alignment
Financial tables Token splitting for consolidated numeric values
Captions "Figure", "Table", "Source:" prefix detection
Sub/superscript Font size and Y-offset relative to baseline
URLs Converted to Markdown links
Hyphenation Rejoins words broken across lines
Page numbers Filtered from output
Drop caps Large initial letters merged with following text
Dot leaders TOC-style dots collapsed to " ... "

Use case: smart PDF routing

pdf-inspector was built for pipelines that process PDFs at scale. Instead of sending every PDF through OCR:

PDF arrives
  → pdf-inspector classifies it (~20ms)
  → TextBased + high confidence?
      YES → extract locally (~150ms), done
      NO  → send to OCR service (2-10s)

This saves cost and latency for the majority of PDFs that are already text-based (reports, papers, invoices, legal docs).

Debugging

See docs/debugging.md for RUST_LOG environment variable usage.

License

MIT

pdf-inspector (firecrawl/pdf-inspector)

Searching Database…

19.7K ·GitHub
Pdf-inspector
这是什么?Firecrawl 出品的一个高速 Rust 库,用来检查 PDF:它会把每个文件归类为文本型、扫描型、图片型或混合型,提取带位置信息的文本,并默认在不使用 OCR 的情况下把它转换成干净的 Markdown。
它是一个 Rust 库、一个命令行工具,还是我到底要安装什么?一个 Rust 核心,提供多种形态:来自 crates.io 的 crate、带 Python 和 Node.js 绑定的原生包、供浏览器使用的 WebAssembly 构建,以及一个带 pdf2md 和 detect-pdf 命令的小型命令行工具。
怎么在我自己的项目里开始用它?Rust 用户直接从 crates.io 拉取;Python、Node.js 和浏览器构建则以独立的包发布,并各自带有 API 文档。如果是源码检出,可以立刻试试 cargo run --bin pdf2md -- document.pdf。
我还需要额外安装别的东西吗?默认路径什么都不需要——Rust 和浏览器构建都是纯提取。只有可选的选择性 OCR 模式需要单独安装 PDFium 和 ONNX Runtime 库,而且即便如此,也只在某一页真的被路由到 OCR 时才会用到。
遇到扫描版 PDF 会怎样?默认情况下它绝不进行 OCR:它会把文件标记为 Scanned 或 Mixed 并给出置信度分数,同时返回缺少文本的精确页码,由你的流程来决定怎么处理。原生包可以选择启用基于 PP-OCRv6 Small 的本地 OCR。
Firecrawl 为什么做这个,而不是把每个 PDF 都送去 OCR?因为大约 54% 的 PDF 本身就是文本型的,OCR 对它们毫无帮助。先分类、只路由真正需要 OCR 的页面,能在批量处理 PDF 时显著降低成本和延迟——报告、论文、发票和法律文档可以直接走快速的本地提取。
它是怎么这么快就分辨出文本型 PDF 和扫描型 PDF 的?它只解析 xref 表和页面树,然后抽样检查内容流里的文本操作符 (Tj/TJ) 和图片操作符 (Do),而不会加载每一个对象。这样几毫秒内就能完成 300 多页 PDF 的分类,扫描检测策略 (EarlyExit、Full、Sample(n)、Pages) 可以在速度和精度之间权衡。
它能完全在浏览器里运行、不需要服务器吗?可以。同一个 Rust 解析器能编译成 WebAssembly,在浏览器和 Web Worker 中运行,内置 CMap,不需要任何服务器往返;和 Rust 构建一样,默认的浏览器构建也只做纯提取。
What are similar projects?liteparse, opendataloader-pdf, anydoc, MinerU, zerox, nougat, pypdf, wordcraft
Which topics is it in?text-extraction, pdf-parser, parser, onnx, ocr, document-processing, wasm, pdf, markdown, rust, nodejs, library
Creator
firecrawl
Size
14.8 MB
Stars
19.7K
Updated
CommitOct 07Releasev1.25.2Sep 28
Age
8mo
Commits · 90 Days