Beyond Temperature:
Hyperfitting as a Late-Stage Geometric Expansion

It isn't temperature scaling — it's a context-dependent rank reordering driven by a "Terminal Expansion" in the final transformer block.

Meimingwei Li^*,1  ·  Yuanhao Ding^*,2  ·  Esteban Garces Arias^1,3  ·  Christian Heumann¹

¹Department of Statistics, LMU Munich    ²School of Computer and Information Engineering, Henan University    ³Munich Center for Machine Learning (MCML)

_{* Equal contribution}

---

Official PyTorch implementation of the ICML 2026 paper "Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion".

Hyperfitting fine-tunes a pretrained LLM on a tiny dataset to near-zero loss and, counterintuitively, improves greedy open-ended generation. This repository shows that the effect is not temperature scaling: it is a context-dependent rank reordering mechanism driven by a "Terminal Expansion" in the final transformer block (effective-dimensionality increase of about +80), and it introduces Late-Stage LoRA, which reproduces the gains by adapting only the final 5 layers.

Key results

• Entropy-Quality Paradox (Sec. 3.2-3.3): at matched entropy, the temperature-scaled base model stays repetitive (TTR ~0.40) while the hyperfitted model is diverse (TTR ~0.68).
• Falsification of static bias (Sec. 3.5): injecting a fixed per-token logit bias only hurts, so the reordering must be dynamic/context-dependent.
• Terminal Expansion (Sec. 4): early layers are preserved; the final layer undergoes a large geometric expansion (Delta-Dim ~ +80.8).
• Late-Stage LoRA (Sec. 5): adapting only the last 5 layers reproduces the rank-reordering dynamics with ~80% fewer trainable parameters.

Repository structure

beyond_temperature/
  config.py          # canonical default hyperparameters (paper Section 2.1)
  data.py            # dataset registry, training datasets, eval-data loader
  metrics.py         # TTR, n-gram rep, Distinct-n, Self-BLEU, entropy,
                     #   effective dimensionality, perplexity, analyzers
  modeling.py        # model/tokenizer loading (PEFT + quantization), generation helpers
  mauve_utils.py     # MAUVE corpora generation + scoring (Section 6)
  cli.py             # unified entrypoint: train / mechanism / section6 / viz
  training/
    hyperfitting.py        # full SFT + LoRA + Late-Stage LoRA (Sections 2, 5)
    normal_finetuning.py   # early-stopping control (Section 4 comparison)
  experiments/
    temperature_matching.py  # Experiment 1  (Sec. 3.2, Fig. 3)
    rank_analysis.py         # Experiment 2  (Sec. 3.3-3.4, Table 1, Fig. 4) + teacher-forcing
    static_injection.py      # Experiment 3  (Sec. 3.5, Table 2)
    representation.py        # Experiment 4  (Sec. 4, Fig. 5) + layer replacement + FT comparison
    decoding_baselines.py    # Sec. 6: decoding baselines, temp sweep, complementarity, diversity
    cross_domain.py          # Sec. 6: readable samples + MAUVE corpora (Tables 6-7)
    agnews_low_entropy.py    # Sec. 6: AG News low-entropy Terminal-Expansion check
    checkpoint_sweep.py      # emergence of the effect across epochs (Fig. 2 / Table 8)
    qualitative.py           # token-category + contextual promotion + qualitative examples
    llm_judge.py             # Sec. 5: pairwise A/B generation for LLM-as-Judge (Table 5)
  viz/visualize.py     # figures from saved result JSONs
scripts/               # convenience shell wrappers

Installation

conda create -n beyond-temperature python=3.11 -y
conda activate beyond-temperature
pip install -r requirements.txt
# or, as a package:  pip install -e ".[eval]"

All commands below are run from this directory (the one containing beyond_temperature/).

Quick start

# 1) Hyperfit the primary model (TinyLlama-1.1B), paper Section 2.1
python -m beyond_temperature.cli train \
    --model TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T \
    --save_dir ./checkpoints/hyperfitted_tinyllama \
    --num_samples 2000 --num_epochs 260 --learning_rate 1e-6 --batch_size 8

# 2) Run the mechanism experiments (Sections 3-4) and render figures
python -m beyond_temperature.cli mechanism \
    --original_model TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T \
    --hyperfitted_model ./checkpoints/hyperfitted_tinyllama/final \
    --output_dir ./results/tinyllama
python -m beyond_temperature.cli viz --results_dir ./results/tinyllama

The end-to-end pipeline is also wrapped as bash scripts/run_full_pipeline.sh.

Reproducing the paper

Section 2 - Hyperfitting and the Loss-Rank-Quality triad

scripts/train_hyperfitting.sh (single model) or scripts/train_multi_model.sh (model suite). Checkpoints are saved every 20 epochs; python -m beyond_temperature.experiments.checkpoint_sweep evaluates the training trajectory (emergence; Fig. 2 / Table 8).

Sections 3-4 - Mechanism and localization

python -m beyond_temperature.cli mechanism \
    --original_model <BASE> --hyperfitted_model <HYPERFITTED>/final \
    --output_dir ./results --experiments all

Runs Experiment 1 (temperature matching), 2 (rank analysis), 3 (static injection), 4 (representation / Terminal Expansion).

Section 5 - Late-Stage LoRA

# Full LoRA
python -m beyond_temperature.cli train --model <BASE> --use_lora \
    --save_dir ./checkpoints/full_lora
# Late-Stage LoRA (final 5 layers)
bash scripts/train_late_stage_lora.sh <BASE> ./checkpoints/late_stage_lora

Pairwise A/B generation for the LLM-as-Judge evaluation (Table 5) is produced by beyond_temperature.experiments.llm_judge.PairwiseEvalFramework.

Section 6 - Additional validation

# Decoding baselines, temperature sweep, complementarity, teacher-forcing, etc.
python -m beyond_temperature.cli section6 \
    --original_model <BASE> --hyperfitted_model <HYPERFITTED>/final \
    --output_dir ./results/section6

# Cross-domain readable samples + MAUVE corpora, then score
python -m beyond_temperature.experiments.cross_domain \
    --original_model <BASE> --hyperfitted_model <HYPERFITTED>/final \
    --output_dir ./results/section6 --dataset_names writingprompts
python -m beyond_temperature.mauve_utils score-domains --results_root ./results

# AG News low-entropy Terminal-Expansion check
python -m beyond_temperature.experiments.agnews_low_entropy --model llama --gpu 0

Hardware

Model	Approx. VRAM	Hyperfitting time (260 epochs, single GPU)
TinyLlama-1.1B	24 GB	~2.5 h (full SFT)
Qwen2.5-1.5B / Gemma-2-2B / Llama-3.2-3B	24-96 GB	~10 h (full SFT); use --use_lora to reduce

Late-Stage LoRA reaches a useful checkpoint considerably faster (see Table 8).

Citation

@article{li2026beyond,
  title={Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion},
  author={Li, Meimingwei and Ding, Yuanhao and Arias, Esteban Garces and Heumann, Christian},
  journal={arXiv preprint arXiv:2605.22579},
  year={2026}
}

This work analyzes the hyperfitting phenomenon introduced by Carlsson et al. (ICLR 2025):

@inproceedings{carlsson2025hyperfitting,
  title={The Hyperfitting Phenomenon: Sharpening and Stabilizing {LLM}s for Open-Ended Text Generation},
  author={Carlsson, Fredrik and Liu, Fangyu and Ward, Daniel and Kurfali, Murathan and Nivre, Joakim},
  booktitle={ICLR},
  year={2025}
}

License

MIT License.