The compression hypothesis

Next-token prediction is mathematically equivalent to lossless compression (Shannon, 1948). A model that perfectly predicts the next token achieves optimal compression of the training distribution.

Implications: every "capability" is a compression artifact. The model doesn't "know" facts — it encodes statistical regularities that happen to correlate with factual knowledge. It doesn't "reason" — it compresses patterns that look like reasoning in training data.

The optimizer shapes the compression

AdamW with warmup + cosine decay creates a specific learning dynamic: early training captures broad distributional patterns (high LR, fast learning); late training refines fine-grained distinctions (low LR, slow consolidation). This is compression at different granularities, not a transition from "pattern matching" to "understanding."

Critical batch size

Below critical batch size: gradient noise provides useful regularization. Above it: diminishing returns. The existence of a critical batch size suggests the optimization landscape has a characteristic scale — consistent with compression, not with models discovering abstract principles.

Why mid-training matters for the thesis

The annealing phase reveals what the model has actually learned vs. what it can be steered toward. If annealing on math data improves math performance disproportionately, the base model had latent math capability that was merely being surfaced — not created.

WSD schedule evidence: Llama 3 uses WSD, enabling multiple anneal branches from the same stable-phase checkpoint. Different anneal mixes produce different downstream capabilities. This is consistent with the model as a compressed representation that can be selectively "unzipped" for different tasks.

Domain adaptation evidence: continued pre-training on 50-100B domain tokens can dramatically shift performance on domain tasks. The efficiency of this (50B tokens vs. 15T original) suggests the base model already encodes the relevant patterns in a compressed form — domain data merely amplifies them.

Long-context extension: RoPE rescaling works because positional encoding is a continuous function. The model generalizes to longer contexts because it learned local patterns that compose — not because it "understands" document structure.

The wrapper hypothesis

Does post-training change cognition (internal representations) or only surface policy (output distribution)?

Mechanistic evidence: core world models are largely preserved through SFT and DPO. Alignment layers act as policy heads. LoRA's effectiveness at r=8-16 implies the behavioral update is low-rank — a thin veneer, not deep surgery.

Where the wrapper breaks

RLVR on reasoning appears to alter internal computational pathways. R1-Zero develops emergent self-verification and extended CoT that are not obvious pre-training artifacts. The model allocates more compute to harder problems (longer CoT) — this adaptive resource allocation is not trivially explained by surface pattern matching.

Counter-argument: RLVR may simply be selecting for pre-existing computational pathways that produce correct answers. The "emergent" behaviors were always latent in the base model — RLVR provides the selection pressure to surface them. This is optimization over existing repertoire, not creation of new capabilities.

The alignment tax question

Does preference optimization degrade reasoning? Evidence is mixed and confounded by benchmark saturation. DPO can collapse output diversity. RLHF can induce sycophancy. Both suggest post-training constrains rather than expands the model's effective capability space.

The dissolving boundary

Post-training is no longer a discrete phase — it extends into inference-time optimization. o1/o3, R1, M1 all use extended inference-time computation to improve reasoning. The boundary between "what the model knows" and "what the model computes at inference" is blurring.

MiniMax M1: Lightning Attention makes test-time compute scaling economically viable — ~25% FLOPs of DeepSeek-R1 at 100K generation length. First linear-attention model competitive at frontier scale.

Thesis implications

If models can "think harder" at inference time and produce better answers, does this constitute reasoning? Or is it search over the model's existing pattern repertoire with more compute budget?

The chess analogy: AlphaGo's MCTS (Monte Carlo Tree Search) is clearly "search," not "understanding." LLM test-time compute may be the same — systematic exploration of the model's compressed representation space, not genuine deliberation.

The whole pipeline in pure PyTorch

Every stage on this page — pre-training, then Base → SFT → Reward Model → {PPO, DPO} → GRPO/RLVR — is implemented by hand in plain PyTorch, with no trl, peft, or transformers. Reading the actual ~200-line training loop is the fastest way to see that "reasoning" alignment is a thin policy layer bolted onto a next-token predictor, not a new cognitive engine.

The base model is small and entirely next-token prediction

A 13M-parameter model trains to coherent grammar in about a day on a single consumer GPU (the RTX 5090 run peaked at ~0.6 GiB VRAM). The architecture is the canonical stack: token + learned position embeddings → N transformer blocks (pre-LayerNorm, causal multi-head attention, 4× ReLU MLP, residual connections) → final LayerNorm → lm_head. The training objective is one line: cross-entropy on the shifted sequence. There is no "understanding" module — world knowledge is whatever the Pile compresses into the weights.

Post-training maps 1:1 onto the corpus thesis

Because the reward model and the GRPO loop are right there in the source, this repo also makes the reward-hacking story concrete: the model optimizes the learned proxy reward, not the intended goal — the same mechanism behind OpenAI's "goblins" reward leak and the production RL misalignment papers in the corpus.