Lessons from Fine-tuning and Serving a 31B Model on a Single GPU

Earlier this year I set out to learn the full lifecycle of a self-hosted LLM — fine-tuning, quantization, and production serving — by building a pipeline around Gemma 4 31B for long-form document analysis. Not a toy model: 31 billion parameters, 62 GB of weights, the kind of thing you actually have to engineer around.

This post is the field notes: the gotchas, the one genuine plot twist, and the mental models that changed how I think about inference. A full write-up of the pipeline — synthetic data generation, training setup, and evaluation results — is coming separately.

A note on freshness: the hands-on work dates to April 2026, and this ecosystem moves fast. I've re-checked the version-sensitive claims as of publication and tagged what's been fixed since — the fixed ones are still worth reading, because the same failure patterns recur with every new model launch.

QLoRA makes 31B a one-GPU problem

Full fine-tuning of a 31B model is a multi-GPU, multi-thousand-dollar affair. QLoRA collapses it to one rented A100 80GB: the base model's weights are frozen and quantized to 4-bit during training, and only small low-rank adapter matrices (r=32 in my case) are trained in full precision.

Training VRAM: about 27 GB. Output: a 467 MB adapter — against a 62 GB base model. That asymmetry is the whole point: the adapter is the only thing you trained, versioned, and can throw away and redo cheaply.

One tradeoff worth knowing: I trained at a 16K max sequence length even though the documents run much longer, because full-length sequences would have needed gradient checkpointing and far slower training. The bet is that long-context attention is pre-trained in — fine-tuning teaches the task, not the attention mechanics.

The merge trap

Plan was: serve the base model + LoRA adapter directly in vLLM. Reality: vLLM doesn't support LoRA inference on Gemma 4's multimodal architecture (Gemma4ForConditionalGeneration). The adapter has to be merged into the base weights first — ~20 minutes, and it needs both the adapter and base model resident simultaneously, which an A100 80GB handles barely.

Then the sneaky part: after merging, config.json still declares the multimodal architecture, so vLLM tries to load a vision processor and dies with preprocessor_config.json not found. The fix is one line — declare it text-only:

config["architectures"] = ["Gemma4ForCausalLM"]

Nothing in any error message points you at this. It cost me an evening.

The quantization plot twist: AWQ silently breaks Gemma

Why quantize at all? On an A100 80GB, the 62 GB BF16 model leaves ~9 GB for KV cache after vLLM overhead — capping usable context around 65K tokens, far below the model's native 256K. At 4-bit (~16 GB), the full context window becomes usable. Quantization isn't just a speed optimization; it's what makes long context fit.

The standard 4-bit method for vLLM serving is AWQ. Here's the trap: AWQ has a documented accuracy bug on Gemma models — still open as of this post's publication. Gemma uses an offset RMSNorm — output * (1 + weight) instead of the standard output * weight — and AWQ's activation-aware weight smoothing doesn't account for the offset, corrupting the calibrated weights. In the llm-compressor issue tracker, a comparable Gemma-family model drops from 85.6% to 69.2% accuracy under AWQ — about 16 points — while GPTQ on the same model retains 83.6%.

The fix is simply to use GPTQ, which does no activation-aware smoothing, so the offset never enters the picture. But if you'd only ever heard "AWQ is the standard," you'd ship a silently broken model and blame your fine-tune.

The tooling is its own adventure

• AutoAWQ is deprecated as of early 2026 and never supported Gemma 4. The living tool is llm-compressor from the vLLM project.
• In April, llm-compressor on PyPI pinned transformers<=4.57 while Gemma 4 needs >=5.5.0 — the only path was installing from git main, then force-reinstalling transformers after (and again after any other pip install that touched it). Released versions now ship Gemma 4 + Transformers v5 support, so this dance is over — but the pattern of release pins lagging brand-new models recurs with every launch.
• The merged model still carries the base model's vision and audio tower weights (Gemma 4 is multimodal; I use it text-only). GPTQ fails on those layers — their dimensions aren't divisible by the quantization group size — so they need explicit ignore patterns (re:.*vision_tower.* and friends).
• model_type: gemma4 triggers multimodal processor auto-detection during calibration; pass the tokenizer directly to bypass it.

None of this is hard once you know it. All of it is undocumented in the places you'd look first.

Why W4A16 and not W4A8

Weights are quantized to 4-bit; activations stay 16-bit. The reasoning: weight quantization error is static — calibration accounts for it once. Activation errors are dynamic and compound layer over layer; in a 62-layer model, a small error at layer 5 has amplified by layer 30. W4A8 exists and is faster, but for a fine-tuned model you can't casually re-calibrate, W4A16 is the safe default — and as it turns out, the speed win from 4-bit weights alone is the one that matters. Which brings me to the biggest mental-model shift of the project.

Inference speed is memory bandwidth, not compute

I went in thinking "bigger GPU = faster." The actual first-order model for single-request inference is:

tokens/sec ≈ memory_bandwidth / model_size

Every generated token requires streaming the entire model's weights from VRAM through the compute cores. The GPU's TFLOPS mostly idle, waiting on memory. That single equation explains the landscape (theoretical maxima; real-world lands at 60–80%):

Two consequences that changed how I budget for inference:

1. Quantization beats hardware upgrades. Shrinking 62 GB → 16 GB is ~4x speed on the same GPU. An H100 over an A100 is only ~1.67x — at many times the cost.
2. The serving stack multiplies. In my runs, moving from naive model.generate() to vLLM (PagedAttention, fused kernels) was ~4x by itself; quantization stacked another ~2.4x on top — roughly 9x end to end on identical hardware.

vLLM gotchas for Gemma 4

Collected here because each one cost real time (status re-checked at publication):

1. Nightly build required (fixed since) — in the first days of April, Gemma 4 needed a vLLM nightly; stable support landed in v0.19.0 on April 2, 2026, so any current stable release is fine.
2. Exactly transformers==5.5.0 (April-era; relaxed since) — at the time it was silently downgraded by other installs, requiring a reinstall after any pip operation. Current vLLM releases support Transformers v5 properly.
3. --max-model-len 262144 (still applies) — Gemma 4 31B's native 256K context. The 128K figure floating around applies to the smaller E2B/E4B variants; without the flag vLLM may default lower.
4. No --quantization flag needed (still applies) — GPTQ in compressed-tensors format is auto-detected from config.json.
5. HF_HUB_ENABLE_HF_TRANSFER=0 (environment-specific) — in my April runs on RunPod, the default fast-download path crashed the HF client.

The result serves an OpenAI-compatible API, so everything written against the OpenAI client works unmodified — which matters more than it sounds, because it makes the self-hosted model a drop-in behind any existing application code.

And if you only have a Mac

The GPTQ/compressed-tensors artifact is vLLM-specific. For Apple Silicon, convert from the BF16 merged model (not the quantized one) to GGUF via llama.cpp or to MLX — both then serve the same OpenAI-compatible API locally. A 31B model at 4-bit runs comfortably on 128 GB unified memory, tightly on 48 GB, and not usefully below that: the weights may fit, but the KV cache won't.

What I actually learned

The skills I set out to acquire — QLoRA training, quantization, production serving — turned out to be the easy half. The valuable half was the systems intuition: quantization is a context-window feature as much as a speed feature; bandwidth math predicts throughput better than spec sheets; the ecosystem's sharp edges live in version pins and architecture metadata, not in the ML.

The full pipeline write-up — how the synthetic training data was built, the training configuration, and evaluation results against gold-standard labels — is next.