BPBench: which LLMs best compress fresh human text

An LLM is a compressor — the better it models language, the fewer bits it needs to encode text it has never seen. BPBench measures exactly that: bits-per-byte on fresh, provably human-written text published after each model's training cutoff, so nothing is memorized and nothing is gamed. Lower = better compression = stronger pretraining (Huang et al. 2024: BPB tracks downstream capability near-linearly). The best compressors are open models from Chinese labs; the closed frontier (GPT-5.6, Claude, Gemini) is much harder to benchmark this way — it hides the token probabilities BPBench needs, which turns out to be its own result (below). (We don't score fresh code — cleanly sourcing it is hard, details.)

Trelis Research · July 2026 · @ronankmcgovern · code on GitHub

Computing BPB needs the model's probability for every token of the text (teacher forcing), so BPBench can only rank models that expose those probabilities — open weights (served with echo=true + logprobs) and a few logprob-exposing APIs. Across the 12 models we could measure, the best compressors form a cluster of largely Chinese-lab models — DeepSeek-V4-Pro, Nemotron-3-Ultra, Kimi K2.6, GLM-5.2, Qwen3.7-Plus — within a few percent of each other. Thinking Machines' Inkling lands just behind; Poolside's Laguna — billed as "the West's most capable open-weight model" — sits near the bottom on fresh prose.

Lower = better compression. Compare within columns — domains differ in intrinsic predictability. Bold = best per column. 95% CIs bootstrapped over documents: the top eight (0.615–0.652) are one broad cluster, not a strict ranking. Inkling (Thinking Machines, read via Tinker's logprobs) lands just behind; MiniMax-M2.7, then Poolside's Laguna-S/XS-2.1, are clearly separate tiers. Nemotron-3-Ultra measured at NVFP4 (its only serverless precision), others at served default; Inkling scored on a 50-doc sample per slice. Kimi K3 and Qwen's latest join once their weights release.

Why closed models are hard to benchmark this way

The obvious next question is where closed models — GPT-5.6, Claude, Gemini — land. The honest answer is that it's hard without the model's token probabilities. Exact BPB needs the probability of each token (teacher forcing), and commercial APIs don't return that — Anthropic gives none, OpenAI only for generated tokens, reasoning models block them. So we're left estimating by sampling: draw many continuations and read off how often the model reproduces the true next word. (We also tried asking models to state their own next-word probabilities — it separates weak from strong across a wide range but goes to noise among frontier peers.) The catch with sampling is which channel you get.

The estimate depends on how you sample. For open models we can sample two ways and check both against the true logprob BPB. Raw completions sampling reads above true; chat sampling — mode-collapsed by RLHF, so it emits the single likely continuation far more sharply — reads below it. That's a ~0.1–0.2 bits/byte swing for the same model, purely from the channel, and it's model-dependent. The true value sits between. So "just sample the model to get its BPB" doesn't have one answer — it has the answer for whichever channel you're forced to use.

Closed models are chat-only, so the honest thing is to sample them and the open models the same way — over chat — and compare on that shared, biased channel. Do that and the closed frontier (GPT-5.6-Sol, Gemini-3.6-Flash, Claude-Fable-5) lands right among the open frontier: projecting up the ~0.08 chat under-read puts all of them at a true BPB of roughly 0.57–0.62. So the suggestive read is that the closed frontier is comparable to the open one — not a fine ranking (the three closed CIs overlap each other, and chat's bias is model-dependent), but a tier.

So, honestly: fresh-prose BPB is a clean, exact measure for open weights; for closed models it is hard, uncertain, and channel-dependent — at best suggestive that the closed frontier sits in the same tier as the open one, and nowhere near precise enough to rank within it. That's why the leaderboard here is the open-weight result; the closed estimates are a caveated footnote, not the headline.

Even exact BPB is loose — trust tiers, not rankings

One more honest limit, even for the open weights we measure exactly: BPB scores the deployed checkpoint, not pure pretraining, and post-training shifts it. Llama-3-8B base compresses fresh prose at 0.624 BPB; its instruct version at 0.712 — post-training alone costs +0.09, larger than the entire frontier spread (~0.04). So read the leaderboard as a broad frontier cluster plus clearly separate tiers (Inkling just behind it; MiniMax, then Laguna, below), never as a point ranking of pretraining quality.

Where this came from

The inspiration is Ryan Greenblatt's no-CoT math time horizon — measuring what models can do in a single forward pass, no chain-of-thought. The challenge there is benchmarking task difficulty reliably (he used Claude to estimate it). We went back to information theory instead: don't grade tasks, ask how novel unseen human text appears to the model. Difficulty calibration is replaced by a ground truth every model faces identically — the actual next words.

The data problem: fresh AND human

This requires text that is human-written and post-dates every model's training cutoff — and in 2026 roughly half of new web text is AI-written, so scoring models on it would measure self-similarity, not quality. We use provenance, not detectors (detectors select for text models find surprising — biasing the exact metric being measured):

• Transcribed parliamentary speech (US Congressional Record, UK Hansard floor debate) — words spoken aloud by humans, with written insertions filtered out. Imperfect: some speeches may be drafted with AI assistance, but the spoken channel is the strongest cheap provenance signal available.
• A private held-out prose slice to catch future benchmark gaming: a model whose gap between the public and private prose widens across releases is the flag. It ranks models consistently with the public prose slices today.

Public slices on Hugging Face: Congressional Record · Hansard. The private slice is held out.

Every document window post-dates every scored model's cutoff (windows June–July 2026; the small private-prose slice extends to June 1).

Why we don't report code (yet)

We tried to add a fresh-code axis and set it aside. Proving code is human-written needs provenance (post-hoc AI-code detectors don't work), and the only projects with enforceable AI-contribution bans are Western systems repos (NetBSD, Gentoo; QEMU relaxed its ban mid-2026). That's a selection bias we can't filter away — "clean fresh code" is disproportionately Western systems C, so a cross-model code comparison is confounded with corpus choice rather than coding ability. On top of that, the available fresh diffs skew to trivial edits (version bumps, one-line fixes) that compress formulaically and don't test code modeling. Finding cleaner, less-biased fresh-code sources takes more digging than we've done so far, and this round was time-constrained — so rather than ship a confounded number, we leave code out of this round and intend to revisit it.

Method notes & caveats

• Exact BPB via teacher-forcing: providers whose completions endpoint supports echo=true + logprobs return log-probabilities for every prompt token in one prefill pass — one request per ~3k-token chunk, input-token pricing only. The full open-model grid cost ≈$10.
• The closed-model gap, quantified: against open weights (where we hold both true logprobs and each proxy) the sampling estimate is biased high by +0.02–0.24 bits/byte, model-dependent, non-vanishing with more samples, and with no universal temperature that corrects it; asking-the-model-directly is at noise within the frontier band. Both are documented as a negative result, not a leaderboard.
• Compression, not capability: BPB scores the deployed checkpoint's raw-text distribution, so it is a pretraining-quality proxy confounded by post-training. Lower is better compression, not strictly a better model — and post-training alone moves BPB more than the frontier spans (Llama-3 base 0.624 vs instruct 0.712, above). Read placements as tiers with error bars, never point rankings.
• Instruct checkpoints, single provider per measurement, serving precision not always disclosed — read results as "deployed checkpoint quality." One broken deployment (MiniMax-M3, all-zero logprobs) was caught and excluded.