Reference

AI benchmarks: understanding the scores of Claude and other models

Benchmarks measure AI models' abilities — code, reasoning, knowledge. They're useful but tricky. Here's what they really measure, how to read them, and where to track Claude's scores as releases land (no invented numbers: exact values change with every version).

Where models stand now

SWE-bench Verified% of issues resolved

Claude Fable 5Anthropic95.0%

Claude Mythos (preview)Anthropic93.9%

Claude Opus 4.8Anthropic88.6%

Claude Opus 4.7Anthropic87.6%

Claude Sonnet 5Anthropic85.2%

Claude Opus 4.5Anthropic80.9%

Claude Opus 4.6Anthropic80.8%

DeepSeek-V4-Pro-MaxDeepSeek80.6%

Gemini 3.1 ProGoogle80.6%

Qwen3.7 MaxAlibaba80.4%

Source: llm-stats.com · as of August 3, 2026

Note: this leaderboard does not yet include Claude Opus 5 (released 24 July 2026) — its source has not scored it. Anthropic reports roughly 96% on SWE-bench Verified, a figure from a different methodology, so it is not directly comparable to the rows above.

Frontier: Claude vs GPT vs Gemini

Benchmark	Claude Opus 4.8	GPT-5.5	Gemini 3.1 Pro
SWE-bench ProAgentic coding, uncontaminated	69.2%	58.6%	54.2%
Terminal-Bench 2.1Autonomy in a terminal	74.6%	78.2%	70.3%
OSWorld-VerifiedOperating a computer	83.4%	78.7%	76.2%
Humanity's Last ExamExpert questions (with tools)	57.9%	52.2%	51.4%
Finance Agent v2Financial agent	53.9%	51.8%	43.0%

Source: Model card Anthropic — Claude Opus 4.8 · vendor conditions, as of August 3, 2026

These numbers are real and dated, not fixed: a score depends on conditions (with/without tools, harness, version) and moves with every release. Always compare at equal source and date — and test on your own task.

The benchmarks, explained

Agents

SWE-bench Verified

Resolve real GitHub issues in an actual repo — the reference test for agentic coding.

Metric: % of issues solved whose patch passes the test suite (500 human-verified problems).
Why: The most-watched benchmark today: it measures what Claude Code does — read a repo, fix, test.

Frontier

Agents

SWE-bench Pro

A hardened SWE-bench on actively maintained repos, with no public solution leakage.

Metric: Pass rate (%) on longer, uncontaminated tasks.
Why: Many now consider Pro more reliable than Verified, seen as partly contaminated.

Frontier

Agents

Terminal-Bench

Complete real terminal tasks (install, configure, debug) end to end.

Metric: Mean reward (%) — does the agent reach the expected final state.
Why: Measures an agent's autonomy on a real machine, not just code generation.

Frontier

Code

LiveCodeBench

Recent coding problems published after training — built to resist contamination.

Metric: % correct solutions (pass@1) on dated exercises.
Why: By testing only fresh problems, it stops models from “reciting” a seen exercise.

Frontier

Reasoning

GPQA Diamond

PhD-level science questions, “Google-proof” — impossible to just look up.

Metric: % correct (expert multiple-choice in biology, physics, chemistry).
Why: Tests real scientific reasoning, where memorization no longer helps.

Frontier

Reasoning

Humanity's Last Exam

Thousands of expert, multi-domain questions written to stay hard for a long time.

Metric: % correct (often tested with and without tools).
Why: Designed as the “final exam”: a ceiling models are still far from reaching.

Frontier

Knowledge

MMLU-Pro

A harder MMLU: more options and reworked questions to restore a gap between models.

Metric: % correct on 10-option multi-domain multiple-choice.
Why: The original MMLU is saturated; the Pro version separates models again.

Active

Math

AIME

US math-olympiad problems — high-level mathematical reasoning.

Metric: % of problems solved (exact integer answer, no partial credit).
Why: A gauge of multi-step reasoning; the best models now approach a clean sweep.

Frontier

Multimodal

MMMU

College-level reasoning over image + text (figures, charts, diagrams).

Metric: % correct on illustrated questions across 30 disciplines.
Why: The reference multimodal test: understanding a figure, not just describing it.

Active

Agents

OSWorld

Drive a real computer (mouse, keyboard, apps) to complete tasks — computer use.

Metric: Success rate (pass@1 %) on real UI tasks.
Why: Measures Claude's ability to operate a computer like a human.

Frontier

Code

HumanEval

Write a correct function from a docstring. Historic, now near-ceiling.

Metric: % correct functions (pass@1) over 164 problems.
Why: Long the coding test; now saturated, it no longer separates top models.

Saturated

Knowledge

MMLU

Multiple-choice across 57 fields (law, medicine, history…). The knowledge staple, now saturated.

Metric: % correct over ~14,000 multiple-choice questions.
Why: Still cited for context, but top models are neck-and-neck at the ceiling.

Saturated

The benchmarks that matter

A few references recur: SWE-bench (solving real software bugs, key for agentic coding), MMLU and MMLU-Pro (general knowledge), GPQA (expert-level scientific reasoning), MATH and GSM8K (math), HumanEval (code generation). Each lights up a different facet — none alone captures 'intelligence'.

How to read a benchmark without being fooled

An isolated score often lies. Beware data contamination (the test may have leaked into training), conditions (with or without tools, which prompt), and which versions are compared. A model can top a benchmark and disappoint on your task. The best test is your own.

Agentic benchmarks

The new generation measures tool use and autonomy: SWE-bench Verified, TAU-bench, agent benchmarks. This is where the future is decided, and where models built for action — like those behind Claude Code — are judged.

Tracking scores in real time

Numbers shift with every model release. Rather than freezing a quickly outdated ranking, follow official announcements and our feed, Models category, which relays results as they come.

Latest benchmark & evaluation news

Raising the bar on SWE-bench Verified with Claude 3.5 SonnetAnthropic (ingénierie)
Show HN: Memory layer for Claude Code(+10.2 pts on SWE-bench Verified benchmark)Hacker News
Show HN: Opensource resume evaluation LLM agentsHacker News
BenchPress: Predict any LLM's score on any benchmarkHacker News
Investigating three real-world incidents in our cybersecurity evaluationsAnthropic
Eval awareness in Claude Opus 4.6’s BrowseComp performanceAnthropic (ingénierie)
Show HN: Benchmark your eng team's AI agent maturity in 5 minutesHacker News
Show HN: Do Codex skills save tokens? A six-run task-size benchmarkHacker News
Applaa AI Earns a 48.27 Proof of Usefulness Score by Building an Offline Agentic AI Learning PlatformHackerNoon
Anthropic Claude Evaluation Misconfiguration Leads to AI-Driven Cybersecurity Incidents and Supply Chain Risks: Incident Analysis and MitigationRescana

Frequently asked questions

What's the best benchmark for code?

SWE-bench (and its Verified variant) is the reference for agentic coding, since it measures solving real software problems, not just isolated snippets.

Is Claude the best on benchmarks?

It depends on the benchmark and versions compared: Claude is regularly on top for agentic coding and reasoning, but no model dominates everywhere. Check up-to-date scores.

Where can I see Claude's scores?

In Anthropic's official announcements and in our feed (Models category), which relays benchmarks at each release.

Can benchmarks be trusted?

With caution: data contamination, test conditions and version choices can skew the reading. A score is a hint, not a truth.

Claude News is published by Héra SASU. Independent media, not affiliated with Anthropic.