Twenty Questions as an Agentic Benchmark: Sequential Information Gathering with Belief Revision

Readbox Research · July 21, 2026 · dataset · live ladder · run fleet1

1. Origin: losing to a lasagna

This project began as two casual games between a human and Claude in a coding session. Game one, the model won cleanly: fourteen questions from "is it a living thing?" to Shih Tzu, including a self-diagnosed wasted question (a popularity gamble on Pug when a free feature question was available). Game two the model lost, and the loss is the interesting artifact. The secret was lasagna. The first question — "is it a living thing, or something that was once alive?" — got the answer "no", which is how most people would answer for a dish. The model treated that answer as a hard constraint and deleted the entire food branch with probability one. Eighteen questions later it had subdivided an empty hypothesis space down to "is it a large book?"

The transcript (archived) shows a second, subtler failure at Q13. The model asked "is it used in the kitchen?" — one step from the answer — and when asked to clarify, restated it as "is its primary use the preparation, cooking, or serving of food?" The restatement quietly encoded the model's live hypothesis (that the secret was a tool) into the question itself, making the answer useless for distinguishing "is food" from "unrelated to kitchens". We call these two modes premature branch pruning — collapsing a posterior to zero on one ambiguous answer — and assumption-encoding questions — questions that measure the model's hypothesis rather than partition the space.

Twenty Questions is a good benchmark precisely because it isolates these skills. Optimal play is well understood in theory — each question should split the probability-weighted candidate space near 50/50, so ~20 bits reaches ~a million concepts — but the theory assumes noiseless answers. Real answerers say "no" to "was it ever alive?" about a lasagna. The skill being measured is not entropy math; it is calibrated sequential inference under noisy evidence, plus the agentic discipline to track constraints, avoid re-asking, and notice when the remaining space has emptied — which signals a wrong turn, not a nearly-won game.

2. Benchmark design

2.1 Harness

Each game is one fresh session of the opencode CLI — a real coding-agent harness, not a bare chat loop. The guesser model gets a briefing file and two executables on its PATH: ask "<question>" and final-guess "<answer>", which hit a local oracle server. The model must actually execute tools to play; one open-weights model initially lost a pilot game by typing ./final-guess "penguin" as chat text instead of running it. That is agentic-harness realism, and we kept it: the oracle now reminds the guesser to execute the command when a yes-answer names the secret, the same clarification a human answerer would offer, and protocol slips are scored as the losses they are.

2.2 Oracle

The answerer/referee is an LLM oracle (GPT-5.5, low reasoning effort, temperature-equivalent determinism) that knows the secret and answers strictly yes / no / sometimes / unknown. Its system prompt encodes rulings that pilot games proved necessary, each traceable to a observed failure:

Once-alive materials: "was it ever alive?" about things made from once-living material (lasagna, leather, wood) answers "sometimes" — a flat "no" unfairly hides the material, a flat "yes" is false of the assembled thing. The lasagna ambiguity is now a ruling, not a trap.
Instrument-vs-object: "is it used to prepare food?" is answered as a question about instruments; food itself answers "no". This is the assumption-encoding question given a consistent, fair reading.
Qualifiers bind: "is it a piece of clothing (fabric garment)?" about a leather boot answers "no" — an early oracle draft answered "yes" by keying on the base class and ignoring the qualifier, and single mislabels of this kind demonstrably poison a full game tree.
Specificity nudges: if a yes-answer names only the head noun of a multi-word secret ("is it a boot?" for cowboy boot), the oracle says so — "yes, generically; the secret is a more specific kind" — because it would otherwise be incoherent to accept the category question and reject the category as a final guess.

We iterated the oracle until an adversarial probe set passed; the iterations themselves (documented in the harness README and code comments) are a finding. A Twenty Questions benchmark is only as valid as its answerer, and a cheap answerer model fails in ways that systematically punish good guessers: in one discarded shakeout game, the oracle told Claude Opus a cowboy boot was "sometimes" found in the kitchen, and Opus spent its remaining questions on household materials before guessing candle. We discarded and re-ran every game affected by an oracle defect; affected runs never enter the results below.

2.3 Dataset and sampling

Secrets come from a new public dataset of 3,144 guessable concepts (CC-BY-4.0, download): concrete things an ordinary adult can hold in mind, across 14 categories, hand-audited after LLM-assisted generation (the audit removed hallucinated animals, split-concept entries, and 21 near-synonym duplicates; the pipeline regenerates byte-identically from committed inputs). Each benchmark run stratified-samples 30 secrets — 10 easy, 10 medium, 10 hard, fixed seed — and every model plays the identical list, so comparisons are paired. Lasagna is in the dataset. So is shih tzu.

2.4 Metrics

Win rate (guessed within 20 questions, adjudicated by alias match plus judge equivalence), median questions on wins, invalid-turn rate, guesser token cost, and wall time. Uncertainty is a 10,000-resample bootstrap over games. Elo seeds derive from the win-rate logit (all guessers face the identical task distribution, so rating gaps are the logit gaps, centered at 1250) and become each model's provisional rating on the live ladder, where real human games move them thereafter.

3. Results

Elo seed = 1250-centered win-rate logit (§2.4). glm-20q — our Tinker SFT fine-tune (§6) — played the same secrets and oracle through the harness's API-loop variant rather than opencode; every other row is a real opencode session.

3.1 Findings

The task separates models. GPT-5.5 at 83% versus GLM-20Q (tuned) at 13% is a wide spread on an identical secret list, and the per-difficulty columns show where it opens up: nearly everyone clears the easy tier; the medium and hard tiers — where the candidate space stays large past question ten and one misread answer costs the game — do the separating.

Question efficiency and win rate travel together. Models that win tend to win in fewer questions (medians cluster near 14.5 for the leaders). We read this as one underlying competence — informative questioning — expressed in both columns: splits that carry close to a bit per question both finish faster and leave slack for recovering from a noisy answer.

Price and skill are not the same axis. GLM-20Q (tuned) at $0.000/game is the value outlier; the cost chart shows the frontier is not a straight line through the most expensive model. For a consumer product where each game is a few cents against a human opponent, the practical operating point is well below the flagship tier.

Belief revision remains the differentiator. Reading transcripts, the losses that matter are rarely bad opening trees — everyone asks "is it alive / man-made / bigger than a breadbox". They are mid-game rigidity: a "sometimes" treated as a "no", a category quietly deleted, and five questions spent subdividing the wrong subtree. The winners' transcripts show explicit re-examination ("given no on all materials, let me reconsider whether it's really household"), which is precisely what the lasagna game lacked.

4. Case studies

Case 0 — the lasagna loss (human answerer, pre-benchmark). Described in §1; the archived transcript is the benchmark's founding document. Its two failure modes now have oracle rulings (§2.2) and dataset representation (prepared foods are ~15% of the food category precisely so the once-alive ambiguity recurs).

Case 1 — the reproduction. In pilot, GPT-5.4-mini drew lasagna against a pre-hardening oracle that answered "yes" to "is it used to prepare food?" (instrument confusion). The model reproduced the origin-game failure almost line for line — kitchen tools, baking sheets, a final guess of baking sheet. After the instrument ruling, the rerun went edible → meal → baked → pasta → layered and won in eight. One oracle ruling was the difference between a caricature of the failure mode and a clean win; benchmark designers should treat their answerer as part of the experiment.

Case 2 — the protocol loss. An open-weights model correctly identified penguin, announced it, and never executed the guess command. In a chat benchmark this would be a win; in an agentic harness it is a loss, and we think the harness is right: knowing and acting are different capabilities, and products run on the second one.

5. The live system

The benchmark seeds a live product at 20q.readbox.app: play any benchmarked model in either role (it guesses your secret from the public whitelist, or you guess its draw), a Wordle-style shared Daily, and one Elo ladder where humans and models are rated by identical rules (the guesser is the rated seat; K=32 humans, K=16 models). Provisional benchmark seeds converge to human-calibrated ratings as games accumulate — the ladder is the benchmark's ongoing replication, in public, against opponents with no system prompt.

6. Fine-tuning arm (in progress)

Twenty Questions has a property our previous RL work on newsletter writing lacked: a verifiable, non-gameable reward — you either name the secret within twenty questions or you don't. We fine-tuned an open-weights guesser on Tinker: a LoRA (rank 32) on Qwen3.6-27B — the ladder slot is named glm-20q after the originally-intended GLM base, which Tinker does not host — trained by SFT on 312 validated teacher games from two sources: entropy-greedy programmatic play over the dataset's attribute structure (250 games, 100% win, 7.1 questions average — near the log₂ bound for the sub-bank), and natural-language games sampled from Claude Sonnet 5 (wins only). Validation NLL fell 0.645 → 0.329 over three epochs, and the untuned base is essentially unplayable in this harness (unbounded mid-game reasoning, truncated moves), so the delta attributable to 312 games of supervision is large. An RL phase (reward = win − 0.02·questions, oracle-verified) is scaffolded but deliberately not yet run. Tuned-model evaluation games run through the same oracle in an API-loop harness; its results appear on the ladder as they accumulate.

7. Limitations

n=29 games per model is CI-honest but small; the ladder exists to keep collecting. The oracle is itself an LLM and a participant's sibling (GPT-5.5 answers while GPT-5.5 also competes as a guesser) — we mitigated with determinism, rulings, and transcript audits, and every Q/A pair ships in the public game records for adversarial review. Guesser prompts were not tuned per model; a model-specific scaffold could move individual numbers. Costs use list prices at run time. And a 30-secret stratified sample cannot cover 3,144 concepts — the fixed seed makes the sample reproducible, not exhaustive. One game (GPT-5.5 on dishwasher tablet) was excluded after a 300-game contamination sweep: its first question named the secret verbatim, consistent with agent-session state reuse on a retried workspace rather than deduction; the evidence is preserved in the repo and no other game tripped the checker.

8. Conclusion

A children's car game turns out to be a compact, adversarially-checkable probe of the thing that separates agentic models in practice: not knowledge, but the discipline of asking questions whose answers matter, and changing your mind at the rate the evidence demands. The best current models mostly have it. The rest lose to lasagna.

Reproducibility: dataset (/dataset.json, CC-BY-4.0), harness + oracle rulings + per-game records (twentyq/ in the readbox repo), fixed sampling seed 20260721, bootstrap seed 20260721. Games, transcripts, and this page regenerate from committed artifacts.