The Model Index
Up to four models

Claude 3.5 Sonnet vs Qwen2.5

On the index (GPT-4 = 100), Claude 3.5 Sonnet scores 122 and Qwen2.5 112. Side by side on the same third-party tests, with price and context. The link in your address bar shares this exact comparison.

Next to Claude 3.5 Sonnet
Presets
Claude 3.5 Sonnet
122107–138
WeightsClosed
ReleasedOct 2024
Price in / out–
Context–
Core tests taken3 of 9
Qwen2.5
11296–128
WeightsOpen
ReleasedSep 2024
Price in / out$0.36 / $0.40
Context33K
Core tests taken2 of 9

Claude 3.5 Sonnet leads Qwen2.5 by 10 points, ahead on 2 of 2 shared core tests.

Test by testthird-party results

Claude 3.5 SonnetQwen2.5
0%25%50%75%100%REASONING & KNOWLEDGEGPQA Diamond (Epoch AI)GPQA Diamond (Epoch AI)MMLU (HELM runs)MMLU (HELM runs)GeoBenchGeoBenchMATHSAIME-style maths (OTIS mock)AIME-style maths (OTIS mock)MATH (level 5)MATH (level 5)CODINGWeirdMLWeirdMLAGENTSBALROGBALROGDTBenchDTBench
Every result as a table
TestClaude 3.5 SonnetQwen2.5
Reasoning & knowledge
GPQA Diamond (Epoch AI)55.3%49.1%
MMLU (HELM runs)87.3%83.4%
GeoBench62.0%62.0%
Maths
AIME-style maths (OTIS mock)8.5%8.1%
MATH (level 5)57.0%63.2%
Coding
WeirdML40.0%16.0%
Agents
BALROG32.6%16.2%
DTBench67.8%62.9%
Task length · METR27 sec21 sec

Tests at least two of these models have taken. Bold rows are the core tests.

This page as a chartDownload PNG