From Scanned Lab Reports to Structured Data: OCR Architecture Trade-offs

Turning a photo of a lab printout into a row that says "Hemoglobin, 13.2, g/dL, reference 12.0–15.5" sounds like an OCR problem, but plain text recognition is only the first step. The real work is getting meaning out of a messy scan cheaply, reliably and without leaking patient data. This walkthrough follows the architectural decisions behind one such extraction pipeline for a medical-reports MVP, so you can reuse the reasoning when you face a similar document-understanding problem on a tight budget.

The requirement that drives everything

Picture an early-stage health product. Patients upload a scanned medical report, either a PDF or a phone photo of a printed lab sheet. The system must convert it into structured health records that both the patient and their doctor can browse, and that later AI-driven analysis can consume. Each record needs four things: the parameter name, its measured value, the unit, and the reference range.

That last sentence is the crux. A tool that returns a block of recognized characters does not satisfy the requirement on its own. Something still has to decide which number belongs to which test, which string is a unit, and which pair of numbers is a normal range. Every decision below is about where that "understanding" step lives and what it costs.

The constraints were typical for an MVP: a small team, a small infrastructure budget, and an expected starting volume of about 500 uploaded reports per day, which works out to roughly 30,000 pages per month.

Decision 1: Choosing the worker language

Python's ecosystem advantage is real

If you start from "which language has the best OCR and ML tooling", Python wins without much debate. Mature open-source engines such as PaddleOCR and EasyOCR are Python-first, OpenCV has first-class bindings there, and most computer-vision examples you will find assume it. A from-scratch OCR pipeline would naturally start in Python.

It is also worth being honest about what is not a differentiator. Google's Vision API was one of the paid options under consideration, and Google publishes client libraries for Python as well as Go. SDK availability therefore did not favor either language.

Why Go won anyway

The deciding factors were about the surrounding system rather than OCR itself. The rest of the asynchronous pipeline was already planned as a Go worker: it consumes jobs from Redis Streams, downloads uploaded files from Cloudflare R2 object storage, and writes results into Postgres. Writing the extraction step in Python would have meant running two worker runtimes, with separate deployment, dependency management and observability, for a single feature. The engineer building it was also most productive in Go for this kind of infrastructure-style service.

The general lesson: the language with the deepest library catalog is not automatically the right one. If the extraction step can be reduced to calling a model over HTTP (which is where this design ends up), keeping a single runtime for the whole worker layer is often worth more than library depth you will barely touch.

Decision 2: Cloud APIs versus self-hosting, priced out

The team compared four options against the expected 30,000 pages per month. The figures below are the estimates used at the time of the evaluation; cloud pricing changes, so check the current price lists before relying on them.

• Google Cloud Vision with DOCUMENT_TEXT_DETECTION. Returns raw text with coordinates and no structured fields. At about $1.50 per 1,000 pages, that is roughly $45 per month.
• Google Document AI (Form or Custom Parser). Returns structured key-value pairs and tables. At $30–$65 per 1,000 pages, that is roughly $900–$1,950 per month.
• Gemini Flash on Vertex AI. Multimodal and prompt-driven, returning structured JSON. Priced per token, the estimate came to roughly $20–$50 per month.
• A self-hosted vision LLM (MiniCPM-V served by Ollama). Produces structured JSON directly. The cost is a fixed server bill of roughly $40–$80 per month for a VPS, independent of volume.

Reading the numbers correctly

The Vision API looks cheap, but it is solving a different problem. It tells you that the characters "13.2" appear at certain coordinates; it has no idea that they are a hemoglobin reading. You would still need to build the entire parsing layer yourself, so its price is not comparable with the options that return structured data.

Document AI does return structure, but close to $1,000–$2,000 a month is a heavy variable cost for a product that has not yet generated revenue.

Gemini Flash is the genuinely interesting contender. Token-based pricing on scanned pages lands in the same range as self-hosting, and you avoid running any GPU infrastructure. So why not pick it?

Why self-hosting still came out ahead

Three arguments tipped the decision:

• A cost ceiling. A dedicated node costs the same whether it processes 5,000 or 50,000 pages. A metered API has no ceiling: a traffic spike, a retry storm or a bulk import shows up directly on the invoice.
• Data isolation. Medical reports contain protected health information. Running extraction inside the team's own private network means patient documents are never sent to a third-party API on every upload, which avoids a whole class of compliance and vendor-review discussions that an MVP team has little capacity for.
• No external quotas or latency. There are no rate limits, concurrency tiers or extra network hops on the critical path.

The result was to drop both the Google Cloud and Vertex AI options and commit to a fully self-hosted stack.

It helps to see the flip side too. A flat server bill is only predictable up to the throughput that one box can handle; beyond that, you are buying more hardware and taking on capacity planning and uptime yourself. Self-hosting does not remove compliance obligations either, since you still own storage, access control and audit of that data. For a small, steady workload with sensitive inputs, the trade usually favors self-hosting; for spiky, fast-growing volume with a team that has no ops capacity, a managed API may be the better bet.

Decision 3: Skipping traditional OCR libraries

Open-source engines like PaddleOCR and Docling are capable, but their output is designed for display. You typically get content wrapped in layout tags or formatted as Markdown so it renders well in a UI or HTML view. That is ideal when a person will read the result and awkward when code needs to pull out "Hemoglobin: 13.2 g/dL" as a clean field.

Bridging that gap is a project in its own right. You would need to extract values from the tag structure, iterate through a lot of trial and error to cope with different labs and report layouts, and probably add OpenCV preprocessing or post-processing for tables and regions the engine did not segment cleanly. All of that sits on top of the work of integrating and hosting the OCR engine in the first place. For an MVP, that is a large amount of engineering surface for something that is infrastructure rather than the product.

One caveat deserves emphasis: this assessment reflects the team's evaluation, not a definitive verdict on these tools. Engineers with deeper PaddleOCR or Docling experience may know a faster route from their structured output to clean fields, and both projects evolve quickly, so it is worth re-checking their current output formats.

Decision 4: A self-hosted vision LLM, sized to the budget

Extracting meaning instead of characters

The alternative to "OCR, then write a parser for the OCR output" is to let a vision-capable language model do both steps at once. Classic OCR recognizes characters. A vision LLM, given a well-structured prompt, can look at the scanned page and return parameter, value and unit fields directly, skipping the intermediate step of mining tagged Markdown.

The vision capability is essential here, not an optional upgrade. A text-only model cannot help with a scanned image because there is no text to give it yet. The model has to read the pixels itself.

This approach has its own risks, and it is worth planning for them from day one. Language models can misread digits or produce plausible-looking values that are not on the page, and smaller models in particular do not always return exactly the JSON shape you asked for. For health data, that argues for defensive parsing of the model's response, validation of values against expected types and ranges, and a way for users or clinicians to see and correct the source document.

Choosing a model you can actually run

Model selection followed the same budget logic. Qwen's larger vision-capable models are strong, but self-hosting a model in that weight class needs more VRAM and compute than made sense for infrastructure the team was paying for out of an MVP budget. The choice landed on MiniCPM-V: a far smaller model that Ollama can serve on inexpensive hardware without strain, yet one still good enough for structured extraction from lab-report images.

If you repeat this exercise, test candidate models on a representative sample of your real documents, including blurry phone photos and unusual lab layouts, rather than relying on leaderboard scores. The best model on paper is irrelevant if you cannot afford to serve it.

What the implementation needs next

The architecture above settles what runs; the production pipeline still has to handle how. The follow-up work covers converting PDFs into images the model can read, structuring the Ollama API request and prompt, parsing JSON defensively when a small model drifts from the requested format, and using Redis Streams in the Go worker so reports are processed reliably as volume grows.

Key takeaways

• Define the output you need (here: name, value, unit, reference range) before comparing tools; raw OCR text and structured data are different products with different prices.
• Choose the worker language for the whole system, not for the richest library of one component.
• Compare cloud options on equivalent output, then weigh cost ceilings, data isolation and quotas alongside the headline price.
• Rendering-oriented OCR output can hide a large parsing project; a vision LLM can collapse OCR and parsing into one step.
• Size the model to the hardware you can pay for, validate its output rigorously, and recheck pricing and tool capabilities before you commit.