Can Small Local Models Do Receipt OCR?
The Question
We needed to evaluate the feasibility of deploying a lightweight, open-source OCR pipeline that can extract structured data from receipt images — vendor, date, totals, tax, line items. The million-dollar question was simple: can a small open-source model running on consumer hardware meet production-quality thresholds?
We tested three pipeline architectures across 200 receipt images from the SROIE dataset (ICDAR 2019), all running on an RTX 3080 10GB under WSL2 via Ollama. We also evaluated two additional candidates that didn't make it to the benchmark. The short answer: not yet.
What We Tested
We evaluated three distinct pipeline architectures, each representing a different approach to the problem:
1. GLM-OCR native — two-pass VLM
A 0.9B vision-language model purpose-built for OCR tasks. In two-pass mode, it first reads the receipt image to extract raw text, then a second pass structures that text into JSON. This was our best-performing prompt strategy out of three variants tested on a 50-image sample (strict_schema, two_pass, table_first).
2. Granite3.2-vision — single-pass VLM
IBM's 2B document-focused vision model. Sends the receipt image directly with an extraction prompt and gets structured JSON back in one shot. We also tested a two-pass variant (read image, then extract from text), but single-pass outperformed it on every metric while running 2.3x faster — so single-pass is the configuration reported here.
3. PaddleOCR + GLM-OCR — OCR + LLM pipeline
A traditional two-stage approach: PaddleX PP-OCRv5 extracts raw text from the image, then GLM-OCR (the same 0.9B model, but text-only) parses that text into structured JSON. This tests whether a dedicated OCR engine feeding a language model can outperform end-to-end vision models.
Two additional models were considered but didn't reach the benchmark:
- Tesseract + GLM-OCR — Early baseline, scored worse than GLM-OCR native across the board. Results archived.
- dots.ocr (1.7B) — RedNote's SOTA model on OmniDocBench. Rejected due to tooling immaturity: no GGUF/Ollama support, missing HuggingFace processor files, flash_attn build failures on WSL2, and garbage output from community quantizations. The only viable path was vLLM Docker, which was too heavyweight for a validation comparison. Worth revisiting if a GGUF appears on Ollama.
How We Measured
Four metrics, each with a kill threshold — fail any one and the pipeline gets a NO-GO verdict:
| Metric | What It Measures | Threshold |
|---|---|---|
| Numeric Accuracy | Does the extracted total match ground truth exactly? | ≥ 90% |
| Date Accuracy | Does the extracted date match after normalization? | ≥ 85% |
| Hallucination Rate | Did the model invent values not present on the receipt? | ≤ 5% |
| Structural Integrity | Is the output valid JSON with all required keys? | ≥ 95% |
These thresholds are deliberately strict. Receipt OCR is a financial data problem — an 81% accuracy rate means roughly 1 in 5 receipts has a wrong total. That's not viable for any real application.
Results
All three pipelines received a NO-GO verdict.
| Pipeline | Numeric | Date | Halluc. | Struct. | Errors | Avg Time | Verdict |
|---|---|---|---|---|---|---|---|
| GLM-OCR native two-pass VLM | 81.5% | 90.0% | 38.0% | 94.5% | 4 | 3.1s | NO-GO |
| Granite3.2-vision single-pass VLM | 51.5% | 86.0% | 64.5% | 100% | 0 | 3.8s | NO-GO |
| PaddleOCR + GLM-OCR OCR + LLM | 64.5% | 79.5% | 46.0% | 89.0% | 19 | 16.7s | NO-GO |
Best Pipeline Detail — GLM-OCR native
Performance
GLM-OCR and Granite are both fast enough for interactive use (~3–4s per receipt). PaddleOCR's 16.7s average is dominated by the OCR preprocessing step, and its 19 errors (9.5% failure rate) make it the least reliable pipeline by a wide margin.
All evaluations ran on a single RTX 3080 10GB. GPU contention between Ollama models is a real constraint — concurrent model loading caused severe stalls, so all pipelines were run sequentially.
GLM-OCR Prompt Strategy
Before the full 200-image evaluation, we tested three prompt strategies for GLM-OCR on a 50-image sample:
| Variant | Numeric | Date | Halluc. | Struct. | Composite | Speed |
|---|---|---|---|---|---|---|
| strict_schema 1-pass | 86.0% | 86.0% | 36.0% | 100% | 0.844 | 1.4s |
| two_pass 2-pass | 88.0% | 90.0% | 34.0% | 92.0% | 0.848 | 2.8s |
| table_first 2-pass | 70.0% | 30.0% | 20.0% | 4.0% | 0.508 | 2.9s |
table_first variant's 4% structural integrity indicates near-total output format failure — the model returned raw tabular text instead of the requested JSON schema, producing parseable output on only 2 of 50 images.All Models Evaluated
| Model / Pipeline | Type | Size | Status | Result |
|---|---|---|---|---|
| GLM-OCR native | VLM (OCR-specific) | 0.9B q8_0 | Tested | NO-GO (best) |
| Granite3.2-vision | VLM (document-focused) | 2B fp16 | Tested | NO-GO |
| PaddleOCR + GLM-OCR | OCR + LLM | PPOCRv5 + 0.9B | Tested | NO-GO |
| Tesseract + GLM-OCR | OCR + LLM | Tess 5 + 0.9B | Archived | NO-GO |
| dots.ocr | VLM | 1.7B | Rejected | Tooling immaturity |
What We Learned
End-to-end VLMs beat OCR+LLM pipelines
This was the biggest takeaway. We expected PaddleOCR (a modern, well-regarded OCR engine) feeding GLM-OCR to outperform GLM-OCR looking at the image directly. The opposite happened — on every single metric.
The intuition is that raw OCR text is lossy. It strips layout, spatial relationships, and visual context that vision models can use. A receipt's total is usually the largest number at the bottom — that's a visual signal. When you flatten the receipt into a text stream, that signal disappears and the model has to guess which number is the total based on surrounding keywords alone. PaddleOCR's 19 HTTP 500 errors (likely from oversized text payloads overflowing GLM-OCR's context window) didn't help either.
OCR-specific training matters more than model size
Granite3.2-vision is over twice the size of GLM-OCR (2B vs 0.9B) and is specifically marketed for document understanding. Yet GLM-OCR's OCR-focused training data gave it a 30-point lead on numeric accuracy (81.5% vs 51.5%) and far lower hallucination rates (38% vs 64.5%).
Granite does excel at one thing: structural output. It produced valid JSON with all required keys on 100% of images — perfect structural integrity. It understands what to output, it just gets the values wrong. This is an interesting signal: a model that reliably produces the right schema but fills it with hallucinated values might be fixable with better prompting or fine-tuning, whereas a model that produces broken output is harder to recover from.
Hallucination is the universal blocker
Every pipeline hallucinates at rates 7–13x above the 5% threshold. This isn't surprising for sub-2B models — they simply don't have enough capacity to reliably distinguish "I see this on the receipt" from "this seems like a reasonable value." The hallucination metric flags any image where at least one extracted field contains a value not present in the source, and at these model sizes, creative gap-filling is the norm rather than the exception.
Two-pass hurts Granite, helps GLM-OCR
An interesting divergence. For GLM-OCR, two-pass mode (read the image, then extract from text) was the best prompt strategy — it slightly outperformed single-pass structured extraction on a 50-image sample. For Granite3.2-vision, two-pass was strictly worse: -9% numeric, -5.5% date, +11% hallucination, and 2.3x slower. The likely explanation is that GLM-OCR's first pass produces high-fidelity OCR text (it's trained for exactly this), while Granite's first pass produces a rougher transcription that loses more information than it preserves.
Phase 1 Verdict: NO-GO
No locally-hosted small model pipeline meets our production thresholds on the SROIE benchmark. The best performer (GLM-OCR native at 0.9B) gets closest, but an 81.5% numeric accuracy and 38% hallucination rate are not production-viable for financial data extraction.
What's Next
Phase 1 establishes a clear baseline and eliminates the hypothesis that sub-2B local models can do receipt OCR at production quality today. The path forward likely involves one or more of:
- Larger local models (7B+ class) that may have enough capacity to reduce hallucination rates, at the cost of higher memory requirements and slower inference
- Cloud API comparison to establish an upper bound on what's achievable and quantify the accuracy gap between local and hosted models
- Hybrid architectures that use a small local model for initial extraction with a cloud model for verification/correction of low-confidence results
- Fine-tuning GLM-OCR or a similar model on receipt-specific data, since its OCR-focused pretraining already gives it a meaningful edge
The open-source OCR landscape is evolving rapidly — GLM-OCR and dots.ocr are very recent additions to the OCR ecosystem. We plan to revisit this evaluation when the next generation of models ships, particularly if dots.ocr releases Ollama-compatible weights. The scoring pipeline, evaluation harness, and dashboard are all reusable for Phase 2 testing.