The Model Index
Up to four models

Claude 3.5 Sonnet (Jun 2024) vs Llama 3.1-70B

On the index (GPT-4 = 100), Claude 3.5 Sonnet (Jun 2024) scores 113 and Llama 3.1-70B 106. Side by side on the same third-party tests, with price and context. The link in your address bar shares this exact comparison.

Next to Claude 3.5 Sonnet (Jun 2024)
Presets
Claude 3.5 Sonnet (Jun 2024)
11397–129
WeightsClosed
ReleasedJun 2024
Price in / out–
Context–
Core tests taken2 of 9
Llama 3.1-70B
10689–122
WeightsOpen
ReleasedJul 2024
Price in / out$0.72 / $0.72
Context131K
Core tests taken1 of 9

Claude 3.5 Sonnet (Jun 2024) leads Llama 3.1-70B by 8 points, ahead on 1 of 1 shared core tests.

Test by testthird-party results

Claude 3.5 Sonnet (Jun 2024)Llama 3.1-70B
0%25%50%75%100%REASONING & KNOWLEDGEGPQA Diamond (Epoch AI)GPQA Diamond (Epoch AI)MMLU (HELM runs)MMLU (HELM runs)MATHSAIME-style maths (OTIS mock)AIME-style maths (OTIS mock)MATH (level 5)MATH (level 5)CODINGWeirdMLWeirdMLAGENTSDTBenchDTBench
Every result as a table
TestClaude 3.5 Sonnet (Jun 2024)Llama 3.1-70B
Reasoning & knowledge
GPQA Diamond (Epoch AI)54.0%44.2%
MMLU (HELM runs)86.5%80.1%
Maths
AIME-style maths (OTIS mock)6.5%3.6%
MATH (level 5)51.7%36.7%
Coding
WeirdML31.0%9.0%
Agents
DTBench67.8%60.0%
Task length · METR24 sec

Tests at least two of these models have taken. Bold rows are the core tests.

This page as a chartDownload PNG