Three Tests to Run Before You Switch from LoRA to FullFT

Three Tests to Run Before You Switch from LoRA to FullFT Kimi K3 on Fireworks: Frontier Intelligence You Can Own Blog Three Tests To Run Before You Switch From Lora To Fullft Three Tests to Run Before You Switch from LoRA to FullFT PUBLISHED 7/30/2026 Table of Contents Experimenting with LoRA The test bed: Qwen3.5-9B How this relates to prior work Placement: rank test → data coverage → tuned learning rate Register allocation: where rank moved the result Nexa VM: fixed recipe → tuned learning rate → rank sweep → broader curriculum First, recipe tuning erased a fixed-recipe FullFT advantage On supported behavior, the rank ladder hit a ceiling Beyond the horizon, curriculum coverage moved the result Three-task mixture: matched exposure → fixed budget → recipe sweep What did the fixed-budget runs cost? What moved the results Before you switch from LoRA to FullFT Fireworks: one platform for inference and training If training with LoRA isn't producing the results you were expecting, the adapter itself may not be the problem. We did controlled experiments on Qwen3.5-9B showing how data coverage, optimization, and rank can create or close the quality gap to Full Parameter Fine-Tuning, or FullFT. Which lever moved? Within-experiment changes; gray cells were not tested In our earlier Kimi K3 work, LoRA was fast, inexpensive, and effective for bounded objectives, and we drew a line: broad or genuinely novel targets are stronger reasons to test FullFT. Before you reach for that more expensive FullFT, you should know that in our experiments when a FullFT beat a simple LoRA run, it was hard to tell whether the adapter simply needed better data, a better training recipe, more rank to close the gap, or whether FullFT was genuinely the better method all along. Experimenting with LoRA LoRA freezes the base model and trains a small adapter instead. FullFT updates every model weight. When FullFT performs better, it is tempting to conclude that the adapter is simply too small. Our experiments explored three possible explanations: •Coverage: The training data did not show enough of the task's variety. The test: broaden the data and re-measure. •Optimization: The training recipe, especially the learning rate, the size of each training adjustment, was poorly tuned. The test: sweep the learning rate and schedule. •Adapter capacity: The LoRA adapter needed a higher rank — a larger adapter. The test: increase rank while holding update scaling and the rest of the recipe fixed, and look for validity and quality to improve together. An intervention shows which change helped under the tested conditions. It does not prove that the lever was the only cause or rule out other contributing factors. The test bed: Qwen3.5-9B We used Supervised Fine-Tuning (SFT) on Qwen3.5-9B and compared LoRA with FullFT. Within each comparison, both methods received the same prompts, targets, and evaluation code. We tested three synthetic tasks with automatic checkers: each task comes with the equivalent of a unit-test suite, so every answer could be scored without a human judge: •Placement, borrowed from chip design: Arrange components on a line to minimize wiring while obeying hard constraints. One violated constraint invalidates the answer. •Register allocation, borrowed from compilers: Assign processor registers so values that are in use at the same time never share one. •Nexa VM: Execute a program written in a synthetic instruction language and report every intermediate state. Program "depth" is its number of instructions. We picked tasks that have programmatic checks for validity, solution quality, and exact execution. That matters because a model can reach nearly zero training loss and still fail when one wrong decision invalidates the whole answer. How this relates to prior work Biderman et al. (LoRA Learns Less and Forgets Less, 2024) reported that LoRA trails FullFT on demanding domains such as code and math while preserving more of the base model. Thinking Machines (LoRA Without Regret, 2025) reported that a well-configured LoRA can match FullFT across much of post-training, with an optimal learning rate roughly an order of magnitude above FullFT's. Our results point the same direction: the best LoRA rate here, 8e-5, sat far above the best FullFT rate, 1e-6 on Nexa. What we add is narrower: a per-gap protocol for testing whether coverage, optimization, or adapter capacity explains a specific FullFT advantage. Placement: rank test → data coverage → tuned learning rate Placement asks the model to arrange cells on a line while minimizing total wire length. In the hard version, a single forbidden position or separation violation invalidates the answer, and that all-or-nothing check revealed a clear LoRA–FullFT gap. We first tested rank, then data coverage: Placement coverage testLoRA-r32LoRA-r128FullFT 2k rows, original seed34.33%31.67%57.00% 2k rows, new data and training seed37.33%36.67%61.67% 20k rows, default recipe68.33%69.33%80.33% Each held-out set contains 300 examples, and a separately generated dataset and training seed reproduced both the flat rank-32 versus rank-128 result and the direction of the FullFT gap. Both adapters reached 100% validity on sampled training prompts, yet more rank did not improve held-out generalization. We define validity as the final output passing the task's checker. An exact sequence or exact trace matched the reference answer step for step. Acing the practice questions did not mean passing the new exam. On the same 6,000-step budget, ten times more unique data roughly doubled LoRA validity and cut the FullFT gap in half; quadrupling rank moved the result by one point. Coverage was only part of the story. On a common selection suite, default rank-32 LoRA reached 76.33% validity and FullFT reached 83.00%. Holding the data and training budget fixed, we raised LoRA's peak learning rate: 20k placement recipe, common suiteValidity LoRA-r32, 2e-576.33% LoRA-r128, 2e-574.00% LoRA-r32, 8e-582.67% LoRA-r128, 8e-582.00% FullFT83.00% At this sample size, the evaluation could not distinguish tuned rank-32 LoRA from FullFT: the 0.33-point difference had a 95% confidence interval of −4.33 to +5.00 points, which is not evidence that the two methods are equivalent. The selected rank-32 recipe then reached 81.67% on an untouched final suite, compared with 80.33% for FullFT. Verdict: The adapter was not too small. The gap came from the training data and the learning rate. More varied data and a higher learning rate closed it; making the adapter four times larger moved the result by one point. Register allocation: where rank moved the result Register allocation maps interfering virtual registers onto different physical registers. One conflict makes the allocation invalid. The main evaluation contains 400 held-out graphs. Placement shows why rank is not always the answer. Here, rank moved the observed result. MethodValid allocationMatches reference register countExact decision sequence LoRA-r880.50%65.75%3.50% LoRA-r3282.50%69.00%6.00% LoRA-r12890.50%81.00%25.50% FullFT93.00%83.75%35.50% Moving from rank 32 to 128 improved validity by eight points. It also improved solution quality and exact policy imitation in the same direction. Rank 128 recovered most of the operational gap to FullFT. FullFT retained an advantage in exact trace reproduction. Note what that last metric measures. Two valid allocations are two routes reaching the same destination. Exact sequence requires taking the reference route. A lead on exact sequence is a lead in imitation, and does not translate one-for-one into practical quality. A separate 60-graph control reproduced the direction: r32/r128/FullFT reached 83.33%/90.00%/90.00% validity. The sample is too small to estimate the effect precisely, and we have not completed placement-style coverage and LR controls here. Rank clearly moved validity and quality under the tested recipe, and we have not excluded coverage or optimization as we did for placement. All register-allocation LoRAs adapted the attention and feed-forward weight matrices. Alpha increased proportionally with rank, keeping alpha/rank = 2, which held the nominal update scaling constant while increasing adapter capacity. Verdict: Here, adapter size was the lever. A larger adapter improved validity and solution quality together, which is the pattern that points to size. The data and learning-rate checks that would confirm it are still open. Nexa VM: fixed recipe → tuned learning rate → rank sweep → broader curriculum Nexa VM asks the model to execute short programs and reproduce the exact state trace. It gives us all three tests (recipe, rank, and coverage) inside one task. A minimal held-out program shows the exact-execution format: 12345678910 Execute the Nexa VM program using the learned opcode semantics. Return each bound register in order, then the final output. program { input r0 := -0027. input r1 := -0025. input r2 := +0012. bind r3 := baf. yield r3. } trace The exact trajectory is: 12 r3 = -0027. output -0027. The longer depth-24 suite applies the same format across 24 dependent instructions, where one post-horizon error can compound through the remainder. First, recipe tuning erased a fixed-recipe FullFT advantage An early 10,000-row comparison made FullFT look much more sample efficient. Under the original recipes, it led LoRA by 20 points on depth-eight final answers and 31 points on exact traces. A same-suite learning-rate sweep reversed that interpretation: Tuned 10k recipe, depth-8 subset (n=64)Peak LRFinal answerExact trace LoRA-r648e-598.44%96.88% FullFT1e-696.88%95.31% The table reports the 64-example depth-8 subset. Across the full 480-example suite, LoRA and FullFT reached 478 versus 476 correct final answers and 475 versus 474 exact traces. Learning-rate tuning removed the fixed-recipe FullFT advantage. Verdict: A 31-point gap that looked like FullFT learning faster was really an untuned learning rate. Tuning it is the cheapest step in the ladder, and it removed the largest apparent method difference in the study. On supported behavior, the rank ladder hit a ceiling We then trained on 100,000 depth-1–8 programs and swept LoRA ranks {64, 32, 16, 8, 4, 2, 1}. Exact-trace accuracy stayed between 99.17% and 100%. Supported behavior did not separate the tested ranks. A separate control trained on the same token count landed near the same ceiling: Token-matched Nexa controlFinal-answer accuracyExact-trace accuracyDepth-extrapolation exactSampled train fit LoRA-r3299.17%97.92%92.97%100% LoRA-r12899.17%98.33%94.53%100% FullFT96.88%95.42%82.81%100% In the control, r128 improved exactness over r32 by two of 480 examples. All ranks were already near ceiling, so supported Nexa behavior did not expose an active adapter-capacity bottleneck. The lower FullFT result describes its tested recipe; it is not evidence that LoRA is intrinsically better. Beyond the horizon, curriculum coverage moved the result The failure mode changed on longer programs. Models trained on a fixed range of register positions became less reliable when depth-16 programs introduced new positions. Increasing rank did not produce a monotonic recovery. Randomizing register starts during training did: Depth-16 exact traceFixed register rangeRandom register starts LoRA-r6496.09%100.00% FullFT64.84%99.22% The coverage horizon: The fixed curriculum taught transitions only at familiar register positions. Increasing rank could not supply missing examples; random register starts expanded the support and repaired the depth-16 trace. Broader register-position coverage moved FullFT by 34.38 points and LoRA by 3.91. Rank had not produced a monotonic recovery. Depth-24 exactness remained lower (89.84% for LoRA and 80.47% for FullFT), so coverage extended rather than eliminated the reliability boundary. Th [truncated for AI cost control]