Gemma 2

GoogleReleased 27 Jun 2024OpenDate verified

Compare Announcement

9785–109Index score#190 of 229 models

1/ 8Core tests taken1 of 6 areas · plus 5 anchor tests

−14Versus the frontier at releasebest then: Claude 3.5 Sonnet (Jun 2024)Replaced by Gemma 3

$0.65/ $0.65Price per million tokensinput / output, list price

8KContext windowtokens it reads at once

Where it sitson the frontier

GPT-4 reached this score 15 months earlier.

Test resultsby area · tick = best by any model

Reasoning & knowledgeevidence

GPQA Diamond

36%

Humanity's Last Exam

no result

Mathsevidence

FrontierMath T1-3

no result

Codingno evidence yet

SWE-bench Verified

no result

Terminal-Bench

no result

Agentsevidence

APEX-Agents

no result

Novel problemsno evidence yet

ARC-AGI-2

no result

Long tasksno evidence yet

METR time horizon

no result

Independent results from the Epoch AI benchmarking hub, best reasoning setting. The core tests are shown here; anchor tests also inform the score and can be shown above.

What others sayrank on each leaderboard

The Model Indexindependent tests, our method#190of 219 · 97

LMArena · Creative writingcreative writing#223of 386 · 1293 Epoch Capabilities Indexoverall capability from many benchmarks#158of 239 · 122.1 LMArena · Textoverall chat quality, judged by people#264of 388 · 1289 LMArena · Mathsmaths questions in chat#270of 373 · 1248 LMArena · Codingcoding questions in chat#282of 383 · 1304

Further left is better. Leaderboards measure different things: LMArena is people voting blind between two answers, the others are test scores. Click any row for the source.