NOTES / EVALS AND OBSERVABILITY
Benchmarks and their limits
What public benchmarks measure, how they go stale or leak into training data, which ones matter for agents, and how to use them to shortlist models without letting them decide.
What benchmarks are for
A benchmark is a fixed set of tasks with a scoring rule, shared so that models can be compared. That is useful for a first cut: which models are worth trying. It is not a substitute for testing your own application (Why evals come first).
Diagram: my own.
The main families
Knowledge and reasoning quizzes. MMLU-style tests ask multiple-choice questions across many subjects. The MMLU-Pro authors argue that MMLU performance has plateaued, making it hard to tell strong models apart, so they built a harder version with ten answer options instead of four and more reasoning-heavy questions; they report lower accuracy and less sensitivity to prompt wording than on MMLU4. A third-party explainer adds that independent analysis found genuine errors in MMLU’s answer key, so a perfect score is not possible10. Scores on saturated tests tell you little about the difference between two strong models.
Broad suites. HELM tries to avoid cherry-picking. Its authors list a wide set of scenarios and measure seven metrics, such as accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, on its core scenarios so that trade-offs are visible1. They note that before HELM, models were on average evaluated on only 17.9 percent of the core scenarios1.
Human preference arenas. Chatbot Arena collects anonymous, randomised head-to-head comparisons from crowd users and ranks models with an Elo-style rating3. The LLM-as-a-judge paper observes that conventional benchmarks such as MMLU and HELM cannot tell aligned chat models from base models well, which motivated preference-based evaluation2. The same paper reports that strong judge models agreed with human preferences more than 80 percent of the time, about the level of agreement between humans, while documenting position, verbosity and self-enhancement biases2 (Evaluators and LLM-as-judge). Arena rankings reflect the votes of the users who show up, which may not resemble yours.
Agentic task suites. These run an agent on real tasks and check outcomes.
- SWE-bench gives an agent a GitHub issue and a codebase and asks it to edit the files so that hidden tests pass; each sample comes from a resolved issue in one of 12 open-source Python repositories7. OpenAI found that some tasks were hard or impossible to solve, which could underestimate models, and worked with the SWE-bench authors on SWE-bench Verified, a human-validated 500-task subset78.
- tau-bench simulates a conversation between a user (played by a model) and an agent with domain tools and policy rules, and compares the final database state with the goal state9. It introduces pass^k, which measures reliability across repeated trials. The authors report that even strong function-calling agents of the time (such as GPT-4o) succeeded on under half of tasks, with pass^8 below 25 percent in retail9.
Reliability, not just capability
Most reported agent scores are single-attempt success rates. But a user experiences repeated use. pass^k asks how often the agent succeeds on all k trials of the same task9.
Diagram: my own arithmetic for an illustrative 80 percent task, assuming independent trials; real trials are correlated.
For your own agent, run each important case several times and report both numbers (Trajectories and final outcomes, Experiments and eval-driven development).
Why scores mislead
- Contamination. Benchmark data contamination happens when a model’s training data includes benchmark information, producing unreliable evaluation results5. A later survey divides it by lifecycle phase (pretraining, fine-tuning and others) and by type (text, text-label, augmentation-based and benchmark-level), and lists detection approaches that need different levels of access to the model6. Public benchmarks are copied, discussed and scraped, so assume some leakage for any old, popular one.
- Saturation. When top models cluster near the ceiling, the test stops discriminating10.
- Flawed items. Broken tasks and wrong answer keys, as with the SWE-bench issues above7.
- Scaffold differences. The same model scores differently with different agent harnesses, tool sets and retry budgets. SWE-bench’s own leaderboard lists systems from simple agent loops to multi-rollout review systems side by side8. Compare like with like.
- Self-reported numbers. Vendor-reported results use the vendor’s prompts and settings. Prefer independent runs where you can find them.
- Task mismatch. A coding benchmark says little about your support agent, or a clinical summariser (Validation and safety in medicine).
Diagram: my own.
A way to use them
Diagram: my own.
- Shortlist two or three candidate models using public results that are close to your task and recent.
- Build a small private eval from real cases, kept out of any training or prompt-tuning data (Building eval sets).
- Run candidates under the same harness, several trials each, and compare quality, latency and cost (Latency and cost).
- Re-run when models change. Providers update models, so keep the eval in CI (Online evals and A/B tests).
Treat leaderboards as hints. The score that matters is the one on your cases.
Sources
- [1]Holistic Evaluation of Language Models — Liang et al. · 2023PARAPHRASED
- [2]Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. · 2023PARAPHRASED
- [3]Chatbot Arena, benchmarking LLMs in the wild with Elo ratings — LMSYS · 2023PARAPHRASED
- [4]MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark — Wang et al. · 2024PARAPHRASED
- [5]Benchmark Data Contamination of Large Language Models, a survey — Xu, Guan, Greene, Kechadi · 2024PARAPHRASED
- [6]A Survey on Data Contamination for Large Language Models — Cheng, Chang, Wu · 2025PARAPHRASED
- [7]Introducing SWE-bench Verified — OpenAI · 2024PARAPHRASED
- [8]SWE-bench Verified — SWE-bench projectPARAPHRASED
- [9]tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao, Shinn, Razavi, Narasimhan · 2024PARAPHRASED
- [10]Benchmark Saturation: Why 99% on MMLU Means Almost Nothing — Capital & Compute · 2026EXPLANATION OF SATURATION AND MMLU ERRORS, AS SUMMARISED BY A THIRD PARTY