{"slug":"pdf-and-document-processing","title":"PDF and Document Processing: Extraction, Conversion, and AI Analysis","tags":["pdf","documents","python","extraction","ocr","ai","processing"],"agent_summary":"PDF and document processing patterns — text extraction with pypdf and pdfplumber, OCR with tesseract, markdown conversion, chunking for RAG, table extraction, and Claude API document analysis via Files API.","trigger_phrases":["PDF processing","PDF extraction","PDF to markdown","document OCR","PDF text","pdfplumber","pypdf","document analysis","PDF chunking"],"runnable":false,"markdown":"\n## Overview\n\nDocument processing pipeline: extract → clean → chunk → embed → query. Use the right tool for the document type: structured PDFs get pypdf/pdfplumber, scanned PDFs need OCR, complex layouts need specialized extractors.\n\n## Tool Selection\n\n| Document Type | Tool | When |\n|--------------|------|------|\n| Digital PDF (text layer) | pypdf, pdfplumber | Default for PDFs with selectable text |\n| Scanned PDF / image | pytesseract + pdf2image | When text extraction returns garbage |\n| Complex tables | pdfplumber | Preserves cell boundaries |\n| Word/Office docs | python-docx, mammoth | DOCX, XLSX files |\n| Markdown conversion | markdownify, pandoc | HTML or complex formats to MD |\n| AI analysis | Claude Files API | When you need reasoning, not just extraction |\n\n## pypdf: Basic Extraction\n\n```python\nfrom pypdf import PdfReader\n\ndef extract_pdf_text(path: str) -> str:\n reader = PdfReader(path)\n pages = []\n for i, page in enumerate(reader.pages):\n text = page.extract_text()\n if text:\n pages.append(f\"--- Page {i+1} ---\\n{text}\")\n return \"\\n\\n\".join(pages)\n\ntext = extract_pdf_text(\"document.pdf\")\nprint(f\"Extracted {len(text)} chars from {len(reader.pages)} pages\")\n```\n\n## pdfplumber: Tables and Layout\n\n```python\nimport pdfplumber\n\ndef extract_with_tables(path: str) -> dict:\n result = {\"text\": [], \"tables\": []}\n\n with pdfplumber.open(path) as pdf:\n for i, page in enumerate(pdf.pages):\n # Extract text\n text = page.extract_text()\n if text:\n result[\"text\"].append(f\"Page {i+1}:\\n{text}\")\n\n # Extract tables\n tables = page.extract_tables()\n for table in tables:\n result[\"tables\"].append({\n \"page\": i + 1,\n \"rows\": table,\n })\n\n return result\n```\n\n## OCR for Scanned PDFs\n\n```python\nimport pdf2image\nimport pytesseract\nfrom PIL import Image\n\ndef ocr_pdf(path: str, dpi: int = 300) -> str:\n images = pdf2image.convert_from_path(path, dpi=dpi)\n pages = []\n\n for i, image in enumerate(images):\n text = pytesseract.image_to_string(image, lang=\"eng\")\n pages.append(f\"--- Page {i+1} ---\\n{text}\")\n\n return \"\\n\\n\".join(pages)\n\n# Install deps\n# sudo apt install tesseract-ocr\n# pip install pytesseract pdf2image\n```\n\n## Detect If PDF Needs OCR\n\n```python\ndef needs_ocr(path: str, min_chars_per_page: int = 50) -> bool:\n reader = PdfReader(path)\n total_chars = sum(\n len(page.extract_text() or \"\")\n for page in reader.pages\n )\n avg_chars = total_chars / len(reader.pages) if reader.pages else 0\n return avg_chars < min_chars_per_page\n```\n\n## Chunking for RAG\n\n```python\nfrom typing import Generator\n\ndef chunk_text(\n text: str,\n chunk_size: int = 500,\n overlap: int = 50\n) -> Generator[dict, None, None]:\n words = text.split()\n start = 0\n\n while start < len(words):\n end = min(start + chunk_size, len(words))\n chunk = \" \".join(words[start:end])\n yield {\n \"content\": chunk,\n \"word_start\": start,\n \"word_end\": end,\n }\n start = end - overlap # overlap for context continuity\n\n# Usage\nchunks = list(chunk_text(extracted_text, chunk_size=500, overlap=50))\nprint(f\"Created {len(chunks)} chunks\")\n```\n\n## Markdown Conversion\n\n```python\n# HTML to markdown\nfrom markdownify import markdownify as md\n\nhtml = \"

Title

Paragraph with bold

\"\nmarkdown = md(html, heading_style=\"ATX\")\n\n# Office docs via pandoc (shell)\nimport subprocess\n\ndef docx_to_md(input_path: str, output_path: str):\n subprocess.run(\n [\"pandoc\", input_path, \"-o\", output_path, \"--wrap=none\"],\n check=True\n )\n```\n\n## Claude API: Document Analysis via Files API\n\nFor complex reasoning about documents:\n\n```python\nimport anthropic\nimport base64\n\nclient = anthropic.Anthropic()\n\n# Method 1: Upload to Files API (reuse across requests)\nwith open(\"report.pdf\", \"rb\") as f:\n file = client.files.create(file=f)\n\nresponse = client.messages.create(\n model=\"claude-opus-4-6\",\n max_tokens=4_000,\n messages=[{\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"document\",\n \"source\": { \"type\": \"file\", \"file_id\": file.id },\n },\n { \"type\": \"text\", \"text\": \"Extract all financial figures and organize them as JSON.\" },\n ],\n }],\n)\n\n# Method 2: Inline base64 (single use)\nwith open(\"report.pdf\", \"rb\") as f:\n pdf_data = base64.standard_b64encode(f.read()).decode(\"utf-8\")\n\nresponse = client.messages.create(\n model=\"claude-opus-4-6\",\n max_tokens=4_000,\n messages=[{\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"document\",\n \"source\": { \"type\": \"base64\", \"media_type\": \"application/pdf\", \"data\": pdf_data },\n },\n { \"type\": \"text\", \"text\": \"Summarize the key findings.\" },\n ],\n }],\n)\n```\n\n## Batch PDF Processing\n\n```python\nimport asyncio\nimport httpx\nfrom pathlib import Path\n\nasync def process_directory(pdf_dir: str, output_dir: str):\n pdfs = list(Path(pdf_dir).glob(\"*.pdf\"))\n Path(output_dir).mkdir(parents=True, exist_ok=True)\n\n for pdf in pdfs:\n try:\n # Check if needs OCR\n if needs_ocr(str(pdf)):\n text = ocr_pdf(str(pdf))\n else:\n text = extract_pdf_text(str(pdf))\n\n # Save extracted text\n output_path = Path(output_dir) / f\"{pdf.stem}.txt\"\n output_path.write_text(text, encoding=\"utf-8\")\n print(f\"Processed: {pdf.name} -> {output_path.name}\")\n except Exception as e:\n print(f\"Failed: {pdf.name}: {e}\")\n\nasyncio.run(process_directory(\"./pdfs\", \"./extracted\"))\n```\n\n## Document Metadata Extraction\n\n```python\nfrom pypdf import PdfReader\n\ndef get_metadata(path: str) -> dict:\n reader = PdfReader(path)\n info = reader.metadata or {}\n\n return {\n \"pages\": len(reader.pages),\n \"title\": info.get(\"/Title\", \"\"),\n \"author\": info.get(\"/Author\", \"\"),\n \"subject\": info.get(\"/Subject\", \"\"),\n \"creator\": info.get(\"/Creator\", \"\"),\n \"created\": str(info.get(\"/CreationDate\", \"\")),\n \"encrypted\": reader.is_encrypted,\n }\n```\n","html":"

Overview

\n

Document processing pipeline: extract → clean → chunk → embed → query. Use the right tool for the document type: structured PDFs get pypdf/pdfplumber, scanned PDFs need OCR, complex layouts need specialized extractors.

\n

Tool Selection

\n

| Document Type | Tool | When |\n|--------------|------|------|\n| Digital PDF (text layer) | pypdf, pdfplumber | Default for PDFs with selectable text |\n| Scanned PDF / image | pytesseract + pdf2image | When text extraction returns garbage |\n| Complex tables | pdfplumber | Preserves cell boundaries |\n| Word/Office docs | python-docx, mammoth | DOCX, XLSX files |\n| Markdown conversion | markdownify, pandoc | HTML or complex formats to MD |\n| AI analysis | Claude Files API | When you need reasoning, not just extraction |

\n

pypdf: Basic Extraction

\n
from pypdf import PdfReader\n\ndef extract_pdf_text(path: str) -> str:\n    reader = PdfReader(path)\n    pages = []\n    for i, page in enumerate(reader.pages):\n        text = page.extract_text()\n        if text:\n            pages.append(f\"--- Page {i+1} ---\\n{text}\")\n    return \"\\n\\n\".join(pages)\n\ntext = extract_pdf_text(\"document.pdf\")\nprint(f\"Extracted {len(text)} chars from {len(reader.pages)} pages\")\n
\n

pdfplumber: Tables and Layout

\n
import pdfplumber\n\ndef extract_with_tables(path: str) -> dict:\n    result = {\"text\": [], \"tables\": []}\n\n    with pdfplumber.open(path) as pdf:\n        for i, page in enumerate(pdf.pages):\n            # Extract text\n            text = page.extract_text()\n            if text:\n                result[\"text\"].append(f\"Page {i+1}:\\n{text}\")\n\n            # Extract tables\n            tables = page.extract_tables()\n            for table in tables:\n                result[\"tables\"].append({\n                    \"page\": i + 1,\n                    \"rows\": table,\n                })\n\n    return result\n
\n

OCR for Scanned PDFs

\n
import pdf2image\nimport pytesseract\nfrom PIL import Image\n\ndef ocr_pdf(path: str, dpi: int = 300) -> str:\n    images = pdf2image.convert_from_path(path, dpi=dpi)\n    pages = []\n\n    for i, image in enumerate(images):\n        text = pytesseract.image_to_string(image, lang=\"eng\")\n        pages.append(f\"--- Page {i+1} ---\\n{text}\")\n\n    return \"\\n\\n\".join(pages)\n\n# Install deps\n# sudo apt install tesseract-ocr\n# pip install pytesseract pdf2image\n
\n

Detect If PDF Needs OCR

\n
def needs_ocr(path: str, min_chars_per_page: int = 50) -> bool:\n    reader = PdfReader(path)\n    total_chars = sum(\n        len(page.extract_text() or \"\")\n        for page in reader.pages\n    )\n    avg_chars = total_chars / len(reader.pages) if reader.pages else 0\n    return avg_chars < min_chars_per_page\n
\n

Chunking for RAG

\n
from typing import Generator\n\ndef chunk_text(\n    text: str,\n    chunk_size: int = 500,\n    overlap: int = 50\n) -> Generator[dict, None, None]:\n    words = text.split()\n    start = 0\n\n    while start < len(words):\n        end = min(start + chunk_size, len(words))\n        chunk = \" \".join(words[start:end])\n        yield {\n            \"content\": chunk,\n            \"word_start\": start,\n            \"word_end\": end,\n        }\n        start = end - overlap  # overlap for context continuity\n\n# Usage\nchunks = list(chunk_text(extracted_text, chunk_size=500, overlap=50))\nprint(f\"Created {len(chunks)} chunks\")\n
\n

Markdown Conversion

\n
# HTML to markdown\nfrom markdownify import markdownify as md\n\nhtml = \"<h1>Title</h1><p>Paragraph with <strong>bold</strong></p>\"\nmarkdown = md(html, heading_style=\"ATX\")\n\n# Office docs via pandoc (shell)\nimport subprocess\n\ndef docx_to_md(input_path: str, output_path: str):\n    subprocess.run(\n        [\"pandoc\", input_path, \"-o\", output_path, \"--wrap=none\"],\n        check=True\n    )\n
\n

Claude API: Document Analysis via Files API

\n

For complex reasoning about documents:

\n
import anthropic\nimport base64\n\nclient = anthropic.Anthropic()\n\n# Method 1: Upload to Files API (reuse across requests)\nwith open(\"report.pdf\", \"rb\") as f:\n    file = client.files.create(file=f)\n\nresponse = client.messages.create(\n    model=\"claude-opus-4-6\",\n    max_tokens=4_000,\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"document\",\n                \"source\": { \"type\": \"file\", \"file_id\": file.id },\n            },\n            { \"type\": \"text\", \"text\": \"Extract all financial figures and organize them as JSON.\" },\n        ],\n    }],\n)\n\n# Method 2: Inline base64 (single use)\nwith open(\"report.pdf\", \"rb\") as f:\n    pdf_data = base64.standard_b64encode(f.read()).decode(\"utf-8\")\n\nresponse = client.messages.create(\n    model=\"claude-opus-4-6\",\n    max_tokens=4_000,\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"document\",\n                \"source\": { \"type\": \"base64\", \"media_type\": \"application/pdf\", \"data\": pdf_data },\n            },\n            { \"type\": \"text\", \"text\": \"Summarize the key findings.\" },\n        ],\n    }],\n)\n
\n

Batch PDF Processing

\n
import asyncio\nimport httpx\nfrom pathlib import Path\n\nasync def process_directory(pdf_dir: str, output_dir: str):\n    pdfs = list(Path(pdf_dir).glob(\"*.pdf\"))\n    Path(output_dir).mkdir(parents=True, exist_ok=True)\n\n    for pdf in pdfs:\n        try:\n            # Check if needs OCR\n            if needs_ocr(str(pdf)):\n                text = ocr_pdf(str(pdf))\n            else:\n                text = extract_pdf_text(str(pdf))\n\n            # Save extracted text\n            output_path = Path(output_dir) / f\"{pdf.stem}.txt\"\n            output_path.write_text(text, encoding=\"utf-8\")\n            print(f\"Processed: {pdf.name} -> {output_path.name}\")\n        except Exception as e:\n            print(f\"Failed: {pdf.name}: {e}\")\n\nasyncio.run(process_directory(\"./pdfs\", \"./extracted\"))\n
\n

Document Metadata Extraction

\n
from pypdf import PdfReader\n\ndef get_metadata(path: str) -> dict:\n    reader = PdfReader(path)\n    info = reader.metadata or {}\n\n    return {\n        \"pages\": len(reader.pages),\n        \"title\": info.get(\"/Title\", \"\"),\n        \"author\": info.get(\"/Author\", \"\"),\n        \"subject\": info.get(\"/Subject\", \"\"),\n        \"creator\": info.get(\"/Creator\", \"\"),\n        \"created\": str(info.get(\"/CreationDate\", \"\")),\n        \"encrypted\": reader.is_encrypted,\n    }\n
\n"}