The Model Index
Models/Google

Gemma 2 9B

GoogleReleased 24 Jun 2024Open
Compare
9481–107Index score#194 · provisional, limited data
1/ 8Core tests taken1 of 6 areas · plus 3 anchor tests
−18Versus the frontier at releasebest then: Claude 3.5 Sonnet (Jun 2024)
–Price per million tokensno public API price found
–Context windowtokens it reads at once

Where it sitson the frontier

GPT-4 reached this score 15 months earlier.

Test resultsby area · tick = best by any model

Reasoning & knowledgeevidence
GPQA Diamond
27%
Humanity's Last Exam
no result
Mathsevidence
FrontierMath T1-3
no result
Codingno evidence yet
SWE-bench Verified
no result
Terminal-Bench
no result
Agentsno evidence yet
APEX-Agents
no result
Novel problemsno evidence yet
ARC-AGI-2
no result
Long tasksno evidence yet
METR time horizon
no result

Independent results from the Epoch AI benchmarking hub, best reasoning setting. The core tests are shown here; anchor tests also inform the score and can be shown above.

What others sayrank on each leaderboard

The Model Indexindependent tests, our method#194of 219 · 94
LMArena · Creative writingcreative writing#264of 386 · 1259Epoch Capabilities Indexoverall capability from many benchmarks#166of 239 · 119.8LMArena · Textoverall chat quality, judged by people#286of 388 · 1267LMArena · Mathsmaths questions in chat#285of 373 · 1220LMArena · Codingcoding questions in chat#304of 383 · 1270

Further left is better. Leaderboards measure different things: LMArena is people voting blind between two answers, the others are test scores. Click any row for the source.

Best attop three among current models

Not in the top three among current models on any category we track.

Where this model places in the top three of today's models, on category leaderboards and on our core tests.

Facts and sources

Released24 Jun 2024
Announcementnot linked yet
Release entry–
Scored asGemma 2 9B
Artificial Analysis index–
Output speed–
Hugging Facenot linked

Google releasesaround this one

PaLM 2Closed
Gemini 1.0 NanoClosed