Local validation gain
+20.3pp
bit_manipulation 33.3% → 53.6% · teacher-forced probe → derivation-first trace fix
A technical retrospective on fine-tuning a rank-32 LoRA adapter for Nemotron-3-Nano. The project separates three bottlenecks that are easy to conflate: solver coverage, token-by-token trace learnability, and inference-runtime parity.
Under greedy decoding, a correct solver trace still fails if the model cannot reproduce its consequential tokens in sequence. Solver coverage, trace learnability, and runtime parity must be measured separately.
The competition scored six reasoning categories under greedy decoding on a frozen backbone with a LoRA adapter capped at rank 32. In that setting, the training target must be both correct and locally predictable. Because all categories share one adapter, a trace redesign can improve one task while shifting behavior in another.
Low-rank delta. Format regularity is what fits in the budget, not solver breadth.
Token-by-token reproducibility is the pass/fail line. No sampling to hide divergence.
Six categories, one LoRA. Divergent trace formats interfere through shared weights.
A single forward pass over the ground-truth trace, checking at every position whether the gold token is still the model's argmax. The first position where it drops to rank > 1 is where greedy decoding is guaranteed to diverge. It diagnoses format-level learnability, not answer correctness.
The diagnostic move is applying teacher-forcing not to measure loss, but to localize where a trace becomes un-reproducible (a cousin of Tong Hui Kang's min-logprob confidence signal — provenance in §08). Run one forward pass with the reference trace forced as input. At each position, compare the model's argmax against the forced token: the first position where the ground-truth token is no longer the argmax (rank > 1) is where greedy decoding will not reproduce the trace. At the divergence points we observed, the gold token's logprob sat between −0.7 and −2.3 — near-ties the model resolves in favor of a different token. Aggregating first-divergence points across a category tells you whether the failure is a solver problem or a format problem — and format problems are fixable within the rank budget.
This was a one-off post-hoc diagnostic, not a continuous pre-submission loop — an honest scoping note, since the difference matters for how much the method actually drove iteration.
Ground-truth trace, teacher-forced. One forward pass, no generation.
Per-token rank of the forced token — divergence is where it stops being the argmax.
First-divergence position — where greedy is guaranteed to break.
Solver gap vs format gap. Only the latter is cheap to fix.
On bit_manipulation, the probe localized a recurring format divergence early in the trace: 44 of 47 failing traces first diverged at the rule-statement line. Rewriting the trace to derive that line — not touching the solver logic — moved local accuracy from 33.3% to 53.6% (+20.3pp). That's the proof the diagnostic points at real, addressable failures.
The failures clustered at the rule declaration, where the operator and operand indices appeared before the trace had derived them. Reordering the trace so those values were computed first improved the category without changing the solver. The result shows that trace structure itself can be the binding constraint.
28/84 on the frozen dev split. Greedy divergence at a fixed structural position (rule statement). Solver was fine; format wasn't reproducible.
45/84 (+20.3pp). Derivation-first trace format, same solver, only the bit_manipulation traces changed — the diagnostic pointed true.
Scope of the gain. The 33.3% → 53.6% result is a canonical local measurement on dev_frozen with vLLM 0.20.1. The submitted adapter scored 0.56 public-LB versus the 0.58 baseline, so the local improvement did not transfer directly.
Shared-adapter tradeoff. The longer bit-manipulation traces also reduced text_encryption accuracy through cross-task interference. The net overall local gain was about 1 point. Full version-by-version evidence is in RESULTS.md and the canonical writeup.
Byte-identical weights produced a 25.3-point text_encryption swing across two local inference environments. The result establishes that runtime differences were material, although the exact Spark 2 attribution and the runtime share of the Kaggle gap remain inferred.
Kaggle's row is honest by omission: the leaderboard exposes only an overall score, never per-task accuracy — so text_encryption under vLLM 0.17.1 (a version user-reported from the competition, not locally verifiable) is unmeasurable. What is measurable: the fully-packaged adapter scores 0.58 on the Kaggle public LB against 0.694 locally, an ≈11pp residual attributed — inferred, not confirmed — to the same runtime-version divergence the two local bars demonstrate directly.
For scale: local greedy evaluation carries a measured ±0.5pp run-to-run noise floor, so a ~25pp swing is more than 50× ordinary non-determinism.
The practical conclusion reframes the scoreboard: a runtime gap this size swamps the adapter-quality signal you think you're measuring, so ordinary submission-level deltas cannot be read cleanly until the inference stack is pinned. Local gains should be treated as environment-specific until tested under deployment parity.
Each case below began with a plausible explanation for a score movement. A later single-variable comparison narrowed or overturned that explanation.
The C2 converter fix regressed the score, so the bug was "load-bearing" — the scrambled weights were somehow helping.
The regression was a packaging confound, not evidence that scrambled weights were beneficial. The reshape fix is mathematically correct; its isolated Kaggle effect remains untested.
"This was never a runtime divergence problem" — an earlier draft's framing.
The later same-weights comparison showed that runtime divergence was material. Its exact contribution to the Kaggle gap remains inferred.
Transposed lora_B reshape — (out,r,E) where it should be (out,E,r) — scrambled 23 MoE layers × 128 experts. Fix is math-correct; clean Kaggle effect still confounded by C11.
v13's naive-packaged submission scored 0.55 (public LB) — a packaging confound, not trace quality; the A-packaged v13 resubmission scored 0.56. Score movement ≠ adapter movement.
text_encryption phantom signal and backbone-prefix rename noise — both ruled out as measurement artifacts, not real levers.
Parity claims require a verification table: confirmed, inferred, or assumed — never blurred. The durable Kaggle levers were mundane and real: packaging, not traces.
| Claim | Evidence state | Effect |
|---|---|---|
target_modules must be a list, not regex, on Kaggle vLLM 0.17.1 (C9/C10) — single-variable A/B on byte-identical weights, regex 0.55 → list 0.57 | confirmed | +2pp public LB |
Mixed-rank experts must be padded to r32 or declared via rank_pattern, or Kaggle's loader mis-applies them | confirmed | 0.56 → 0.57–0.58 public LB |
Mechanism of the list lever — 0.17.1 silently under-applies a regex target_modules | inferred | not run on 0.17.1 locally |
C2 lora_B reshape fix is mathematically correct | confirmed | round-trip verified |
| C7 runtime divergence is a dominant contributor to the local↔Kaggle gap — not the sole cause | inferred | 25.3pp local swing · ≈11pp Kaggle residual |
| Clean isolated Kaggle effect of the C2 fix | assumed | untested — confounded by C11 |
| Backbone-prefix rename is noise, not signal | confirmed | ≤1pp, sign-inconsistent across A/Bs |
Sources: RESULTS.md packaging ladder (Kaggle public-LB submission matrix 2026-06-13: A backbone+list 0.58 · model.model+list 0.57 · model.model+regex 0.55, weights byte-identical) · BACKBONE_DIAGNOSTIC.md §2 (prefix-only A/B) · docs/SESSION_LOG.md C9/C10/C11 entries.
Final standing: public LB 0.58 · private LB 0.604 · rank 3488/4182. Public and private are different metrics over different held-out splits; every per-submission score on this page — the ladder above included — is a public-LB value, and a public-LB delta does not necessarily predict private standing. The private split held slightly above the public one.
The transferable result is a workflow for locating format-level divergence, redesigning traces, and separating adapter changes from packaging and runtime effects.
Huikang used min-logprob probing continuously as a pre-submission gate. This project used it post hoc on one failing category. The higher-leverage workflow is to run the probe after every training iteration, before spending a leaderboard submission.
Teacher-forced logprob probing localizes format-level greedy divergence in one forward pass — a cheap, reusable diagnostic.
Single-variable ablations caught C7, C8, and C11. Explicit on-page reversals signal rigor, not weakness.
Runtime version parity can dominate apparent adapter quality. Pin it before trusting any local→leaderboard extrapolation.
Trace learnability > solver coverage under greedy decoding at low rank. Format is the binding constraint.
Public repo, Apache-2.0, with a clean provenance trail for third-party contributions.
Code & docs: github.com/kpeterson1/nemotron-reasoning (Apache-2.0) · canonical writeup · RESULTS.md · OPEN_QUESTIONS.md (corrections C1–C11).
Attribution: the min-logprob (teacher-forced) diagnostic and the character-by-character cipher decode are tonghuikang's contributions — his public writeup and midpoint Open Prize–winning repository were the structural benchmark for this work. Our application of them — post-hoc format-learnability diagnosis, and naming our own confounds — is original. Provenance for the reference copies we worked from: references/README.md.