Qwen2.5

AlibabaReleased 19 Sep 2024OpenDate verified

Compare Announcement

11099–122Index score#159 of 229 models

2/ 8Core tests taken2 of 6 areas · plus 10 anchor tests

−13Versus the frontier at releasebest then: o1-mini

$0.36/ $0.40Price per million tokensinput / output, list price

33KContext windowtokens it reads at once

Where it sitson the frontier

Claude 3.5 Sonnet (Jun 2024) reached this score 3 months earlier.

Test resultsby area · tick = best by any model

Reasoning & knowledgeevidence

GPQA Diamond

49%

Humanity's Last Exam

no result

Mathsevidence

FrontierMath T1-3

no result

Codingevidence

SWE-bench Verified

no result

Terminal-Bench

no result

Agentsevidence

APEX-Agents

no result

Novel problemsno evidence yet

ARC-AGI-2

no result

Long tasksevidence

METR time horizon

5.2 min

Independent results from the Epoch AI benchmarking hub, best reasoning setting. The core tests are shown here; anchor tests also inform the score and can be shown above.

What others sayrank on each leaderboard

The Model Indexindependent tests, our method#159of 219 · 110

LMArena · Creative writingcreative writing#152of 386 · 1353 LMArena · Textoverall chat quality, judged by people#171of 388 · 1374 LMArena · Mathsmaths questions in chat#178of 373 · 1363 LMArena · Codingcoding questions in chat#184of 383 · 1402 Epoch Capabilities Indexoverall capability from many benchmarks#128of 239 · 132.5

Further left is better. Leaderboards measure different things: LMArena is people voting blind between two answers, the others are test scores. Click any row for the source.