Why LoRA r8 beats r32

When we started training Lor-1 we assumed more rank was better: more parameters, more capacity, more ability to absorb the SA clinical corpus. We were wrong. This post is the short version of what happened and why we now train every Lorraine model at LoRA r8.

The experiment

Same base (Qwen3.5-27B), same training data (high-eland, 9,960 examples), same 2 epochs at LR 5e-5. The only variable: LoRA rank. We trained at r8, r16, and r32 and evaluated each checkpoint against LorBench vnext — 203 clinical questions — plus a held-out set of parametric medical-knowledge probes to check for catastrophic forgetting.

What we found

r32 hit the highest LorBench score mid-training but collapsed on the parametric probes — losing up to 12 points on general medical knowledge the base already knew. r16 held parametric ability better but never matched r8 on the target benchmark. r8 was the sweet spot: it absorbed the SA corpus without evicting knowledge the base already had.

Why this makes sense

LoRA adapts a base by projecting updates through a low-rank subspace. Higher ranks effectively let the optimiser rewrite more of the base — which is fine when you are training on generic data, but costly when the base already has correct knowledge you want to preserve. For clinical adaptation, you want the minimum rank that still fits the new distribution.

Operational takeaway

We now treat r8 as the default for clinical adaptation of any base in the 27B–35B range. We only consider higher rank when the target distribution is a genuine domain the base has not seen, which is rarely the case in medicine.