An information bottleneck turns memorized, spurious features into room for real, generalizable code knowledge.
At a glance
DeepSeek‑Coder‑7B‑Instruct‑v1.5 fine‑tuned on OriGen. IB‑FT beats standard fine‑tuning (FT) on top‑1 accuracy and, crucially, on the stricter multi‑sample metric Pass@k(m) — where a problem counts as solved only if at least m of k samples pass the unit tests.
Problem
Practitioners adapt pretrained LLMs to code by supervised fine‑tuning. We find this is often brittle — and trace the cause to something the base model already did before you ever started.
Fine‑tuned code models can show a large gap between greedy decoding (Pass@1) and sampling‑based metrics: a correct program often appears somewhere among k samples, while the one‑time generation stays wrong. That exposes how little standard fine‑tuning actually acquires, and motivates our central question:
What causes the ineffectiveness of LLM code fine‑tuning, and how can fine‑tuning be improved to achieve stronger generalization?
Examining the interaction between fine‑tuning data and the pretrained model through the lens of memorization, we find the base model already strongly memorizes the downstream code data — before any adaptation. This traps optimization in a region the standard objective cannot escape.
Pruning works, but it is impractical: it needs example‑wise memorization attribution (expensive) and a search over how much to prune (multiple fine‑tuning runs). We want a remedy that needs no attribution and no pruning.
Method
Instead of deleting memorized examples, compress their representations. An IB penalty on the hidden states discards input‑specific, spurious detail while keeping what predicts the target — so heavily and lightly memorized points get treated more equally.
Given input X, target Y, and a hidden representation Z, the information bottleneck seeks a Z that preserves task‑relevant information about Y while discarding redundant details of X:
Direct computation is intractable in LLMs, so we follow the variational IB framework. On a designated hidden layer hθ(x) (e.g. layer 20), a variational encoder qφ produces Z, and the objective splits into two terms. The compression loss upper‑bounds I(X; Z) with a KL to a simple prior p(z) — discouraging Z from encoding exact input patterns:
The prediction term keeps Z informative for the target, preventing over‑compression — the log‑likelihood of generating Y from Z:
Together these give the IB regularizer, with β balancing compression against prediction:
Finally, adding the IB regularizer to the standard cross‑entropy fine‑tuning loss ℓFT yields IB‑regularized fine‑tuning (IB‑FT), where α controls the IB strength:
No data attribution. No pruning. No prior knowledge of which examples form the barrier — just one penalty on the hidden representations.
Partition the data into most‑memorized and least‑memorized groups and measure their hidden representations at layer 20. Standard FT exaggerates the separation between the two groups — building the very barrier that forces the model to treat them differently. IB‑FT compresses that gap, cutting both distance and angular disparity by more than half.
Experiments
Two benchmarks (OriGen, Evol‑CodeAlpaca‑V1), multiple base models, and comparisons against regularization baselines (Dropout, Gaussian noise, label smoothing) and data‑selection baselines (Min‑K%, GRAND). Below: the headline OriGen table.
| Method | Human · Pass@1 | Human · Pass@10(10) | Machine · Pass@1 | Machine · Pass@10(10) |
|---|---|---|---|---|
| Base (no FT) | 34.62 | — | 44.76 | — |
| Standard FT | 42.63 | 27.56 | 67.83 | 49.65 |
| FT w/ Dropout | 43.64 | 33.55 | 72.97 | 51.41 |
| FT w/ GN | 45.91 | 34.36 | 72.21 | 51.66 |
| FT w/ Min‑K% | 50.89 | 42.95 | 73.78 | 68.52 |
| IB‑FT (ours) | 51.99 | 48.07 | 75.67 | 69.23 |
The pattern is sharpest under the strictest setting (m = 10, all ten samples must pass): IB‑FT improves over FT by roughly 20 points on both Eval‑Human and Eval‑Machine. FT’s successes are often isolated single samples; IB‑FT’s are consistently reproducible.
Varying T ∈ {0.2, 0.6, 1.0} stresses sampling stochasticity. IB‑FT sustains superior accuracy across all temperatures, while FT collapses as T grows — e.g. on Eval‑Machine at T = 1.0, IB‑FT reaches 50.3% (Pass@10(10)) while FT falls to 2.1%.
The memorization barrier is not code‑specific. Extending both the probing protocol and IB‑FT to three non‑code tasks — commonsense reasoning (HellaSwag), science QA (OpenBookQA), and natural‑language inference (MNLI) — reproduces the same three‑way memorization separation, and IB‑FT again beats FT. Together with the code results, IB‑FT is now validated across five task categories and four model families (DeepSeek‑Coder, CodeLlama, Llama‑2, Llama‑3).
| Method | HellaSwag (Llama‑2‑7B) | OpenBookQA (Llama‑3‑8B) | MNLI (Llama‑3‑8B) |
|---|---|---|---|
| Base | 0.57 | 0.36 | 0.48 |
| + FT | 0.81 | 0.67 | 0.84 |
| + IB‑FT | 0.84 | 0.71 | 0.87 |
What is left open
IB‑FT is a practical fix; the memorization barrier is a deeper question. Directions we think matter:
Cite this work
Accepted at the Conference on Language Modeling (COLM) 2026. If you find this work useful, please cite:
@article{wang2025breaking, title = {Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization}, author = {Wang, Changsheng and Chen, Xin and Liu, Sijia and Ding, Ke}, journal = {arXiv preprint arXiv:2510.16022}, year = {2025} }