Private Multimodal RAG on an M1 Mac: Selective Chart Reading With Ollama

Engineers, financial analysts and researchers regularly handle confidential PDFs on their own laptops: internal technical documentation, quarterly earnings reports, clinical trial papers. What they want is a private, free search assistant that runs entirely on a MacBook with an M1 chip and still answers accurately from complicated page layouts, financial tables and charts.

There is a hard limit to watch for. Feeding every page of a 100-page PDF to a vision-language model as an image on an M1 means very long waits, heavy memory swapping and a fan that sounds ready for takeoff. A selective hybrid pipeline avoids that, and no byte of the data has to leave the machine.

Three problems to solve at once

Ingesting complex PDFs on modest hardware like the M1 raises several distinct difficulties, some tied to the hardware and some to the models themselves.

1. Naive text extraction with tools like pypdf or pdfminer is quick and light on memory, but it flattens multi-column text, wrecks table structure and ignores charts altogether.
2. Full-page vision processing keeps the visual context, yet it overwhelms the chip. Running each page through a VLM eats RAM, takes minutes per document and adds OCR mistakes on small body text.
3. Hallucination is common to every vision-language model, whether it runs on an M1, an A100 or a CPU. Reading fine axis labels, small print and dense spreadsheet-style tables is something they do unreliably, and they frequently invent numbers or misdescribe trends in freeform summaries. For financial or clinical material, where precision is mandatory, that is a serious flaw in any multimodal RAG system.

A workable design needs to handle the memory ceiling, the lost text structure and the habit of inventing numbers together.

The selective hybrid design

The M1's unified memory architecture lets the CPU and GPU share one fast pool, which makes it surprisingly capable for local AI as long as the pool is used wisely. The pipeline splits work into four parts.

• A fast path covers text and tables. A lightweight layout-aware parser, PyMuPDF, reads native text, keeps heading hierarchy as Markdown levels and turns tables into clean Markdown, all within milliseconds.
• A selective path handles visuals. Charts, graphs and flowcharts get cropped and saved as separate image files. Only those standalone images go to a quantized, small local VLM, qwen2.5-vl:3b served through Ollama, which cuts memory use and inference time sharply.
• Guards against hallucination make the output trustworthy. The model must answer in structured JSON, and the numbers it reports are cross-checked against raw OCR text taken from the image's bounding box, so claims are anchored in the actual picture before reaching the vector database.
• Context fusion puts the pieces back together. Visual summaries are inserted into the Markdown stream next to metadata such as page numbers and image references, and indexing happens after that.

Architecture

The flow from the local document store through parsing, cropping, summarizing and indexing looks like this:

                       ┌───────────────────────────────────────┐
                       │   Local PDF Document Store (M1 Mac)   │
                       └───────────────────┬───────────────────┘
                                           │
                           ┌───────────────┴───────────────┐
                           │    Fast Layout-Aware Parser   │
                           └───────┬───────────────┬───────┘
                                   │               │
              [Text & Tables]      │               │     [Embedded Images]
                                   ▼               ▼
                         ┌───────────────────┐   ┌───────────────────┐
                         │  Markdown Stream  │   │ Cropped Images    │
                         └─────────┬─────────┘   └─────────┬─────────┘
                                   │                       │
                                   │                       ▼
                                   │             ┌───────────────────┐
                                   │             │ Base64 Scaling &  │
                                   │             │ Native BBox OCR   │
                                   │             └─────────┬─────────┘
                                   │                       │
                                   │                       ▼
                                   │             ┌───────────────────┐
                                   │             │  Local Metal VLM  │
                                   │             │  (Ollama via UMA) │
                                   │             └─────────┬─────────┘
                                   │                       │
                                   │               [Text Summaries]
                                   │                       │
                                   ▼                       ▼
                         ┌───────────────────────────────────┐
                         │ Unified Chunking & Context Engine │
                         └─────────────────┬─────────────────┘
                                           │
                                           ▼
                         ┌───────────────────────────────────┐
                         │ Disk-Persisted Vector DB & Parent │
                         │ Context Stores (ChromaDB SQLite)  │
                         └───────────────────────────────────┘

Hardening it

A prototype script differs from a dependable local application, and a few infrastructure decisions make the difference.

Choosing the vector store

ChromaDB suits this setup because it runs in-process, needs no infrastructure and stores data on disk through an embedded SQLite database, leaving 8 or 16 GB of RAM free for Ollama. The design isn't tied to it, though. Swapping in a different store requires only small changes to ingestion:

• LanceDB offers very fast disk-based approximate search with a native Python interface.
• Qdrant is a high-performance database that runs well locally and has rich filtering.
• FAISS, from Meta, is a pure in-memory library that is extremely quick when RAM is plentiful.
• SQLite with the sqlite-vec extension adds vector search straight into SQLite, a good fit for anyone wanting one dependency fewer.
• Pinecone and Weaviate are cloud-hosted options for projects that outgrow the laptop.

Chunking, embedding and retrieval work the same way whichever one you pick.

Parent-child retrieval

Chart summaries shouldn't float free of the text around them. The index stores small child chunks of about 200 tokens for precise matching, but returns the larger parent section of about 800 tokens to the LLM, so the model reads full paragraphs when it answers.

Keeping the numbers honest

Two light safeguards address hallucination whatever hardware is used. First, Ollama is told to reply with a JSON schema, for instance a chart type plus lists of axis values and data points, in place of free text, which is simpler to parse and validate. Second, PyMuPDF pulls raw text out of the exact bounding box of the cropped image, and each number the VLM produced must appear in that string before anything is written to the database. Summaries that fail the check are flagged for manual review or dropped.

Project setup and code

Folder structure

Create the working directory in a terminal:

mkdir ~/m1_multimodal_rag && cd ~/m1_multimodal_rag
mkdir data chroma_db images_cache
touch main.py requirements.txt

Dependencies

List the packages in requirements.txt:

pymupdf
chromadb
ollama
pillow

The single script

All logic sits in main.py: parsing the PDF, encoding images as Base64 for Ollama, writing parent and child chunks into Chroma, and exposing a small CLI. A --clear flag resets the database, so a new PDF can be ingested without mixing data.

#!/usr/bin/env python3
"""
Multimodal RAG for Apple Silicon (M1/M2/M3)
Usage:
    python main.py ingest --pdf data/report.pdf
    python main.py ingest --pdf data/new_report.pdf --clear
    python main.py query --question "What was the Q3 revenue?"
"""

import argparse
import base64
import os
import sys
import uuid
from io import BytesIO
from pathlib import Path

import chromadb
import pymupdf as fitz
import ollama
from PIL import Image

# ---------- CONFIG ----------
TEXT_MODEL = "llama3.2:3b"          # For final RAG answers (8GB friendly)
VISION_MODEL = "qwen2.5vl:3b"       # For charts (8GB friendly)
CHROMA_PATH = "./chroma_db"
IMAGE_CACHE = "./images_cache"

# Initialize persistent Chroma client (SQLite, not RAM)
chroma_client = chromadb.PersistentClient(path=CHROMA_PATH)
child_collection = chroma_client.get_or_create_collection(name="child_chunks")
parent_collection = chroma_client.get_or_create_collection(name="parent_chunks")

Path(IMAGE_CACHE).mkdir(exist_ok=True)


# ---------- HELPER: Encode Image for Ollama ----------
def encode_image_for_ollama(image_bytes: bytes, max_size=800) -> str:
    """Convert PDF image bytes to Base64 data URI with size limiting."""
    img = Image.open(BytesIO(image_bytes))

    # Convert RGBA/P to RGB to avoid JPEG alpha errors
    if img.mode in ('RGBA', 'LA', 'P'):
        img = img.convert('RGB')

    # Downscale massive images to save VRAM on M1
    img.thumbnail((max_size, max_size))

    buffered = BytesIO()
    img.save(buffered, format="JPEG", quality=85)
    img_base64 = base64.b64encode(buffered.getvalue()).decode('utf-8')
    return img_base64


# ---------- PHASE 1: INGESTION ----------
def ingest_pdf(pdf_path: str):
    """Parse PDF, extract text, crop images, run VLM, and store in Chroma."""
    print(f" Processing: {pdf_path}")
    doc = fitz.open(pdf_path)

    for page_num in range(len(doc)):
        page = doc[page_num]
        print(f"  Page {page_num + 1}/{len(doc)}")

        # 1. Extract main text
        page_text = page.get_text("text").strip()
        if not page_text:
            page_text = "[No extractable text on this page]"

        # 2. Find and process images
        image_list = page.get_images(full=True)
        visual_summaries = []

        for img_idx, img in enumerate(image_list):
            xref = img[0]
            try:
                base_image = doc.extract_image(xref)
                image_bytes = base_image["image"]

                # Encode for Ollama
                encoded_img = encode_image_for_ollama(image_bytes)

                # Prompt designed for financial charts with structured output
                prompt = """
                Extract key insights from this chart and return valid JSON.
                Use this schema: {"chart_type": "", "x_axis": [], "y_axis": [], "key_trend": "", "data_points": []}
                If it's not a chart, describe it briefly in text.
                """

                response = ollama.chat(
                    model=VISION_MODEL,
                    messages=[{
                        "role": "user",
                        "content": prompt,
                        "images": [encoded_img]
                    }]
                )
                summary = response["message"]["content"]
                visual_summaries.append(f"[Chart on page {page_num+1}]: {summary}")

            except Exception as e:
                print(f"    Skipped image {img_idx} (Error: {e})")
                continue

        # 3. Merge text and summaries
        full_page_content = page_text + "\n" + "\n".join(visual_summaries)
        if not full_page_content.strip():
            continue  # Skip completely empty pages

        # 4. Split into Parent (big) and Child (small) for retrieval
        parent_text = full_page_content  # Full page is the "Parent"

        # Split into ~200 token chunks for children (roughly 800 chars)
        child_chunks = []
        chunk_size = 800
        for i in range(0, len(parent_text), chunk_size):
            child_chunks.append(parent_text[i:i+chunk_size])

        if not child_chunks:
            child_chunks = [parent_text]  # Fallback

        # 5. Store in Chroma (Parent-Child)
        parent_id = str(uuid.uuid4())
        metadata = {
            "source": os.path.basename(pdf_path),
            "page": page_num + 1,
            "type": "hybrid"
        }

        # Store Parent (full context) - Persisted to disk, not RAM
        parent_collection.add(
            ids=[parent_id],
            documents=[parent_text],
            metadatas=[metadata]
        )

        # Store Children (granular search)
        child_ids = []
        child_metadatas = []
        for idx, chunk in enumerate(child_chunks):
            child_id = f"{parent_id}_child_{idx}"
            child_ids.append(child_id)
            child_metadatas.append({
                **metadata,
                "parent_ref": parent_id
            })

        child_collection.add(
            ids=child_ids,
            documents=child_chunks,
            metadatas=child_metadatas
        )

    doc.close()
    print(" Ingestion complete!")


# ---------- PHASE 2: QUERY ----------
def query_rag(question: str):
    """Retrieve relevant context using Child chunks, fetch Parent, ask LLM."""
    print(f"❓ Query: {question}")

    # 1. Retrieve top matching child chunks
    results = child_collection.query(
        query_texts=[question],
        n_results=3
    )

    if not results["ids"] or not results["ids"][0]:
        print(" No relevant documents found in the database.")
        return

    # 2. Fetch the full Parent contexts
    parent_ids = list(set([m["parent_ref"] for m in results["metadatas"][0]]))
    parent_results = parent_collection.get(ids=parent_ids)
    full_context = "\n\n---\n\n".join(parent_results["documents"])

    # 3. Build prompt for the text-only LLM
    prompt = f"""
    You are a financial research assistant. Answer the question based strictly on the context below.
    If the context contains chart summaries or tables, use those numbers specifically.
    If you cannot answer from the context, say "I don't have that information."

    Context:
    {full_context}

    Question: {question}
    Answer:
    """

    # 4. Generate answer locally
    response = ollama.chat(
        model=TEXT_MODEL,
        messages=[{"role": "user", "content": prompt}]
    )

    print("\n Answer:")
    print(response["message"]["content"])
    print("\n Sources:", ", ".join(parent_ids))


# ---------- DATABASE CLEAR FUNCTION ----------
def clear_database():
    """Delete all collections to reset the database."""
    try:
        chroma_client.delete_collection("child_chunks")
        chroma_client.delete_collection("parent_chunks")
        print(" Database cleared successfully!")
    except ValueError:
        print(" Database was already empty. Nothing to clear.")
    except Exception as e:
        print(f" Could not clear database: {e}")


# ---------- CLI ENTRY POINT ----------
def main():
    parser = argparse.ArgumentParser(description="M1 Multimodal RAG Pipeline")
    subparsers = parser.add_subparsers(dest="command", required=True)

    # Ingest command with --clear flag
    ingest_parser = subparsers.add_parser("ingest", help="Ingest a PDF")
    ingest_parser.add_argument("--pdf", required=True, help="Path to PDF file")
    ingest_parser.add_argument("--clear", action="store_true", help="Clear the database before ingesting")

    # Query command
    query_parser = subparsers.add_parser("query", help="Ask a question")
    query_parser.add_argument("--question", required=True, help="Your question")

    args = parser.parse_args()

    if args.command == "ingest":
        if not os.path.exists(args.pdf):
            print(f" File not found: {args.pdf}")
            sys.exit(1)

        # Clear the database if the flag is set
        if args.clear:
            clear_database()
            # Re-initialize collections after clearing
            global child_collection, parent_collection
            child_collection = chroma_client.get_or_create_collection(name="child_chunks")
            parent_collection = chroma_client.get_or_create_collection(name="parent_chunks")

        ingest_pdf(args.pdf)

    elif args.command == "query":
        query_rag(args.question)


if __name__ == "__main__":
    main()

Running it

Step 1: Install Ollama and pull the models

Ollama installs through Homebrew, and the qwen2.5vl models need version 0.7.0 or newer:

# Install Ollama via Homebrew
brew install ollama

# Verify version (requires >= 0.7.0 for qwen2.5vl models)
ollama --version

# Start the Ollama service (keep this running in a separate terminal tab)
ollama serve

# Pull the recommended models (For 8GB M1 Mac)
ollama pull qwen2.5vl:3b
ollama pull llama3.2:3b

# (For 16GB+ M1/M2/M3, optionally pull larger models)
# ollama pull llama3.2-vision:11b
# ollama pull qwen2.5:7b

# displays a list of all AI models stored locally on your machine
ollama list

Step 2: Create the Python environment

# Ensure you are in the project root
cd ~/m1_multimodal_rag

# Create a virtual environment
python3 -m venv venv

# Activate the environment
source venv/bin/activate

Step 3: Install the dependencies

# Upgrade pip
pip install --upgrade pip

# Install requirements
pip install -r requirements.txt

# Verify installation
python -c "import chromadb, fitz, ollama, PIL; print(' All dependencies ready!')"

Step 4: Fetch a sample PDF

An illustrative IFRS financial report from EY makes a convenient test document:

# Download a sample financial report (EY IFRS Illustrative)
curl -L -o data/sample_financials.pdf \
  "https://drive.google.com/uc?export=download&id=1OOE1vPBwPP31cB0_MNgooTrhw6KpY6n_"

Step 5: Ingest

python main.py ingest --pdf data/sample_financials.pdf

Expect a progress bar for each page, then a message saying ingestion is complete. To load a different PDF after wiping the old data, add the clear flag:

# Ingest a new PDF and clear the database first
python main.py ingest --pdf data/<your financial data file>.pdf --clear

Step 6: Ask a question

python main.py query --question "What was the total revenue shown in the financial statements?"

Summing up the approach

A local multimodal RAG system on an M1 is achievable. PyMuPDF handles quick text and table extraction, Ollama with qwen2.5vl:3b interprets only the visuals that need it, ChromaDB or another lightweight store keeps a persistent parent-child index, and one command-line script ties it together. The outcome costs nothing, sends no data anywhere and answers from confidential documents on the machine itself.

The finished project can also be cloned from its repository:

git clone https://github.com/froilan-sia/m1_multimodal_rag.git
cd m1_multimodal_rag

It includes a setup.sh helper that creates the .gitignore, README and license and initializes Git, which helps when forking or customizing. After cloning, either follow the manual steps above or run the helper:

./setup.sh