Self-Hosted Vision LLM OCR in Go: Rasterizing, Prompting, Parsing

Picking a self-hosted vision model over a cloud OCR API is only half the decision; the other half is a pipeline that survives what the model actually returns. This walkthrough follows a production Go worker that reads medical lab reports and stores the results in Postgres. You will see how pages are rasterized, how the prompt is shaped, how to parse output that ignores the requested structure, and how the worker scales without double-processing jobs.

Step 1: Turn PDFs into page images

Ollama's vision models accept images, not PDFs, so each upload is rendered into one image per page first. Instead of linking a heavy rendering library, the worker shells out to pdftoppm from the Poppler suite, installed on the worker host. The function writes the bytes to a temp file, renders every page to PNG at 300 dpi, then reads and deletes the generated files. exec.CommandContext ties the child process to the job's context, so cancellation also stops the render.

func extractImagesFromPDF(ctx context.Context, pdfBytes []byte) ([][]byte, error) {
    tmpFile, _ := os.CreateTemp("", "report-*.pdf")
    defer os.Remove(tmpFile.Name())
    tmpFile.Write(pdfBytes)
    tmpFile.Close()

outPrefix := strings.TrimSuffix(tmpFile.Name(), ".pdf")
    // Render each PDF page to a 300dpi PNG
    cmd := exec.CommandContext(ctx, "pdftoppm", "-r", "300", "-png", tmpFile.Name(), outPrefix)
    if err := cmd.Run(); err != nil {
        return nil, fmt.Errorf("pdftoppm failed: %v", err)
    }
    matches, _ := filepath.Glob(outPrefix + "*.png")
    var images [][]byte
    for _, match := range matches {
        imgBytes, _ := os.ReadFile(match)
        images = append(images, imgBytes)
        os.Remove(match)
    }
    return images, nil
}

300 dpi proved to be the right balance: small print such as units and reference ranges stays readable for the model without producing oversized page images. The snippet discards several errors for brevity; handle them in real code. Also note that filepath.Glob sorts lexically, so for ten or more pages you may want to order by the page number pdftoppm appends.

Step 2: Ask the model for a specific JSON shape

The extraction target is a list of test results, and the struct spells out the contract: name, value, unit, reference range, abnormal flag and a self-reported confidence score.

type ExtractedParameter struct {
    ParameterName  string `json:"parameter_name"`
    Value          string `json:"value"`
    Unit           string `json:"unit"`
    ReferenceRange string `json:"reference_range"`
    AbnormalFlag   bool   `json:"abnormal_flag"`
    OcrConfidence  int    `json:"ocr_confidence"`
}

The call is a plain POST to Ollama's /api/generate. The image is base64-encoded into Images, the prompt is filled in with the report category, streaming is off, and Format: "json" is set. The envelope's Response field is then unmarshalled into the parameter slice. Some statements in this snippet are run together on single lines, so read it as an outline of the flow, and check resp.StatusCode plus the ignored errors in production.

func (c *OllamaClient) ExtractParameters(ctx context.Context, imageBytes []byte, category string) ([]ExtractedParameter, error) {
    base64Image := base64.StdEncoding.EncodeToString(imageBytes)
    prompt := fmt.Sprintf(promptTemplate, category)    reqData := ollamaRequest{
        Model:  c.Model,
        Prompt: prompt,
        Images: []string{base64Image},
        Stream: false,
        Format: "json", // nudges Ollama toward JSON output — doesn't guarantee it
    }    reqBytes, _ := json.Marshal(reqData)
    req, _ := http.NewRequestWithContext(ctx, "POST", c.URL+"/api/generate", bytes.NewReader(reqBytes))
    req.Header.Set("Content-Type", "application/json")    resp, err := (&http.Client{}).Do(req)
    if err != nil {
        return nil, fmt.Errorf("failed to send request to ollama: %v", err)
    }
    defer resp.Body.Close()    var oResp ollamaResponse
    json.NewDecoder(resp.Body).Decode(&oResp)    // parsing continues below — see "Handling model output reliability"
    var parameters []ExtractedParameter
    if err := json.Unmarshal([]byte(oResp.Response), &parameters); err == nil {
        return parameters, nil
    }
    // ...
    return parameters, nil
}

The prompt, paraphrased rather than the production wording, lists hard rules and ends with an example of the exact array to return:

You are a medical data extraction assistant. You are given an image of a
medical report.
Rules:
1. Only extract tests that are explicitly visible in the image - never
   invent, guess, or infer a result that isn't printed.
2. The report may contain several distinct tests. Create one JSON object
   per test - do not merge multiple tests into a single object.
3. Capture the test name, result, unit, and reference range exactly as
   printed. Ignore narrative "interpretation" paragraphs.
4. If a result is flagged abnormal (marked with *, H, L, or similar),
   set abnormal_flag to true.
5. Rate your own confidence in reading each value, 0-100.
6. Respond with ONLY a flat JSON array - no wrapping object, no markdown.
[
  {
    "parameter_name": "Hemoglobin",
    "value": "13.2",
    "unit": "g/dL",
    "reference_range": "13.0 - 17.0",
    "abnormal_flag": false,
    "ocr_confidence": 98
  }
]

Each rule targets a failure mode. Banning invented values addresses hallucination, the most dangerous error with medical data. One object per test stops the model from merging a panel into a blob. Copying values as printed preserves units and ranges, and skipping interpretation paragraphs keeps narrative out of structured fields. The confidence score is useful for routing doubtful values to a human, though a model's self-rating is a heuristic rather than a calibrated probability. The last rule, respond with nothing but a flat array, matters more than it looks, because JSON mode alone does not enforce structure.

Parsing output that ignores the requested shape

Format: "json" biases generation toward valid JSON, but it is not a schema. In practice a smaller local model may nest the list inside an object like {"tests": [...]}, return a single object, or emit something that is not one clean parseable document.

Rather than failing the extraction on the first mismatch, the parser tries progressively looser strategies and stops at the first success:

// 1. Try the shape we actually asked for: a flat array
if err := json.Unmarshal([]byte(raw), &parameters); err == nil {
    return parameters, nil
}
// 2. Try an object wrapping the array under an unknown key, e.g. {"tests": [...]}
var wrapper map[string][]ExtractedParameter
if err := json.Unmarshal([]byte(raw), &wrapper); err == nil {
    for _, v := range wrapper {
        if len(v) > 0 {
            return v, nil
        }
    }
}
// 3. Try a single object instead of an array
// 4. Recursively walk arbitrarily nested JSON, matching field names loosely
//    (e.g. "test_name", "name", "parameter_name" all resolve to the same field)
// 5. As a last resort, regex out individual {...} blocks and parse each one
//    independently - catches cases where duplicate keys break a clean unmarshal

The strict parse runs first, so well-behaved responses take the cheapest path. Next, any wrapper key holding a non-empty array is accepted, then a lone object is promoted to a list. A recursive walk handles deep nesting and maps loosely matching field names (test_name, name, parameter_name) to one field. As a last resort, a regex extracts individual {...} blocks and parses each alone, which rescues responses broken by duplicate keys.

It is not elegant, and ideally the model would comply every time. It is the price of running a compact model yourself rather than paying for a polished commercial API: a subscription bill traded for defensive parsing, which was worth it for an MVP. Two suggestions if you copy the idea: log which strategy succeeded so you can measure whether prompt or model changes help, and validate results from the looser strategies more strictly. Recent Ollama releases can also take a JSON schema in format; check the current docs for your version.

Step 3: Extract multi-page reports page by page

Each page of a multi-page report goes to Ollama as its own request, and the per-page results are concatenated into one list before anything is written to Postgres. If one page fails, the error is logged and the remaining pages still get processed instead of failing the whole report. The trade-off is that a report can be saved with some results missing, so recording failed pages keeps that partial state visible.

Step 4: Scale the worker with Redis Streams consumer groups

The worker reads jobs from a Redis Stream as part of a consumer group, so several instances can run in parallel while each message is delivered to only one of them. A message is acknowledged with XAck only after extraction and the database write have both succeeded.

streams, _ := rdb.XReadGroup(ctx, &redis.XReadGroupArgs{
    Group:    groupName,
    Consumer: consumerName,
    Streams:  []string{streamName, ">"},
    Count:    1,
    Block:    0,
}).Result()

// ... process message, extract parameters, save to DB ...
rdb.XAck(ctx, streamName, groupName, message.ID)

The > ID requests messages never delivered to the group, Count: 1 takes one job at a time, and Block: 0 waits indefinitely. Because the ack comes last, a crash leaves the message in the pending entries list instead of losing it. Pending messages are not redelivered on their own, though; plan for XAUTOCLAIM or XPENDING with XCLAIM to hand stale work to a healthy consumer.

Transient failures are retried up to three times with a short backoff before the report is marked FAILED. Errors that retrying cannot fix, such as the source file being missing from storage, are treated as fatal and skip retries altogether.

Wrapping up

None of these choices is the theoretical best in isolation. Shelling out to pdftoppm, stacking fallback parsers and accepting partial reports are compromises that fit real constraints: a small MVP budget, sensitive patient data kept in-house, and a small team that had to ship.

• Tune rasterization resolution against the smallest text that matters.
• Treat JSON mode as a hint and parse for the shapes the model really produces, strict path first.
• Write prompts as explicit rules, each aimed at an observed failure.
• Isolate failures per page and per job, and make partial results visible.
• Acknowledge queue messages only after every side effect succeeds.