Fine-tuning Gemma 4 31B for On-Prem Document Analysis
End-to-end fine-tuning pipeline for structured document analysis, trained on 100% synthetic data — QLoRA on a rented A100, GPTQ quantization, vLLM serving, and formal evaluation against gold-standard labels.
What it does
A personal R&D project answering the question: when cloud models are off the table entirely, how far can an on-prem open model go on structured document understanding? The full pipeline — synthetic data → fine-tune → quantize → serve → evaluate — built deliberately on 100% synthetic data (no real documents; privacy by design), on nothing more than a rented GPU.
The pipeline
- Generated 200 synthetic documents across 10 types (financial statements, invoices, filings and more) and 600 gold-standard training pairs (extraction / summary / Q&A) with Claude
- QLoRA fine-tune of Gemma 4 31B on a rented A100: 4-bit NF4, 0.78% trainable parameters, 467MB adapter — working around Gemma's multimodal quirks (vision-tower LoRA exclusion, text-only architecture swap for vLLM compatibility)
- GPTQ W4A16 quantization: 62GB → 16GB, ~2.4x faster than BF16 — chosen over AWQ after diagnosing AWQ's documented accuracy bug on Gemma
- Served on vLLM at 256K context with tool-call parsing
Evaluation
Formal evals against gold-standard labels across all three tasks — extraction, summarization, and Q&A — covering JSON validity, exact-field and null-field accuracy, and serving latency. Full results, and the complete cost breakdown, in the pipeline write-up publishing shortly.
The tooling and serving lessons are already up: see "Lessons from Fine-tuning and Serving a 31B Model on a Single GPU" on the blog.