Fresh BPB: pretraining quality, measured on data models can't have seen

Ten frontier open models scored on how well they compress human-written text and code published after their training cutoffs.

Trelis Research · July 2026 · ~$10 of inference for the full grid

Benchmarks saturate, leak into training data, and get tuned for. Compression doesn't: a model's bits-per-byte (BPB) on genuinely novel text is a direct readout of how well its pretraining captured the world, and prior work (Huang et al. 2024) shows it tracks downstream capability almost linearly. The catch in 2026: "novel text" is hard to find — roughly half of new web articles are AI-generated, and most new GitHub code is AI-assisted. Scoring models on AI-written data would measure self-similarity, not quality.

We solve this with provenance, not detectors: transcribed parliamentary speech (words spoken aloud by humans), and commits from open-source projects with explicit AI-contribution bans — all from dated windows (June–July 2026) that post-date every scored model's cutoff. Two additional slices are undisclosed canaries. AI-text detectors are deliberately avoided: they select for text language models find surprising, which would bias the very metric being measured.

Leaderboard

Lower is better. Compare within columns — domains differ in intrinsic predictability. Bold = best per column.

The GPT-OSS anomaly

The headline finding. OpenAI's open-weight models are near the pack on fresh code but ~3× worse at compressing fresh human speech — and the 120B barely improves on the 20B (it's worse on Hansard). That pattern is a training-data fingerprint: a heavily filtered, synthetic-skewed mix that covers code and essayistic web prose but lacks the long tail of real-world civic text. Standard benchmarks, which GPT-OSS does respectably on, don't show this at all. (One caveat: GPT-OSS expects its Harmony chat format, so raw-text completion is further off-distribution for it than for the others.)

Domains disagree — that's the point

Kimi K2.6 takes the overall crown on code strength; DeepSeek-V4-Pro wins every prose/speech slice. A single-corpus number would have hidden this. The top four models are within 5% of each other — pretraining quality at the open frontier is genuinely close.

Version deltas — including a regression

Fresh-data BPB is sensitive enough to resolve version-to-version changes: GLM-5.2 improves on 5.1 across every slice; DeepSeek's distilled V4-Flash pays a visible ~10% tax vs V4-Pro. And a surprise: Kimi K2.7-Code compresses fresh code worse than its general-purpose sibling K2.6 (0.294 vs 0.261 on public code) — heavy code-focused post-training didn't improve raw code modeling; it appears to have hurt it.

Closed frontier models — where they land

Closed APIs return no logprobs (and OpenAI's echo is long gone), so exact BPB is impossible there. The escape hatch is behavioral: ask a model to continue fresh text verbatim with thinking disabled, score the continuation, and calibrate against exact BPB on the open models (Spearman −0.77, n=10, protocol v2 — which credits both verbatim-echo and direct-continuation response styles; every model's scoreable rate is ≥92%, eliminating the compliance bias of v1). Seven closed frontier models, ~290 prose positions each:

• Claude Fable 5 exceeds every open model on fresh-prose continuation — clearly, on both slices. One condition caveat: ~30% of its samples contained residual reasoning despite thinking being requested off, which can only have helped it; the CoT share is printed on the chart for every model.
• Gemini-3.1-Pro sits above the open frontier's edge but reasons on 100% of samples — it cannot be run zero-CoT at all, so its score is inflated by an unknown amount. GPT-5 ran near-clean (7% CoT) and lands at the open frontier's edge.
• GPT-5-chat — OpenAI's non-reasoning chat model and the only closed model fully inside calibration range — lands mid-cluster: estimated prose BPB 0.56–1.13, comparable to the open frontier, not above it.
• Claude Opus 4.8 is the cleanest strong result: 87% echo compliance, zero residual reasoning, and still above every open model — the most defensible "closed beats open" data point in the study.
• Claude Sonnet 5 scores high but echoes only 24–54% of passages, so its similarity is computed on a self-selected subset — least trustworthy of the six.

Fable 5 and Gemini remain above the calibration range, so they get bounds (BPB ≲ 0.53), not point estimates. The residual outlier is MiniMax-M2.7, which the proxy underrates relative to its true BPB — one reason the fit is a tier instrument, not a ruler.

Method notes

• Exact BPB via teacher-forcing: providers whose completions endpoint supports echo=true + logprobs return log-probabilities for every prompt token in a single prefill pass — one request per ~3k-token chunk, input-token pricing only. The full 10-model × 5-slice grid cost ≈$10.
• Data: ~5.7 MB across five slices — US Congressional Record and UK Hansard floor speech (verbatim human speech; written insertions like bill text filtered out), commits from an AI-ban open-source project (checksums, generated files and vendored code stripped), plus one undisclosed text and one undisclosed code slice as domain-gaming canaries. Every document window post-dates every model's cutoff.
• Caveats: instruct checkpoints (base models are no longer served anywhere), single provider, serving precision not publicly disclosed per model — read results as "deployed checkpoint quality." One broken deployment (MiniMax-M3, all-zero logprobs) was caught and excluded.
• Next: Kimi K3 the moment its weights land on a provider; scaled closed-model proxies; monthly slice refreshes.