Up to four models
Llama 3.1 vs Claude 3.5 Haiku
On the index (GPT-4 = 100), Llama 3.1 scores 110 and Claude 3.5 Haiku 109. Side by side on the same third-party tests, with price and context. The link in your address bar shares this exact comparison.
Llama 3.1 and Claude 3.5 Haiku are too close to call (2 points apart, within the uncertainty), ahead on 1 of 1 shared core tests.
Test by testthird-party results
Every result as a table
| Test | Llama 3.1 | Claude 3.5 Haiku |
|---|---|---|
| Reasoning & knowledge | ||
| GPQA Diamond (Epoch AI) | 50.9% | 38.1% |
| MMLU (HELM runs) | 84.5% | 74.3% |
| Maths | ||
| AIME-style maths (OTIS mock) | 9.7% | 4.3% |
| MATH (level 5) | 49.8% | 46.4% |
| Coding | ||
| WeirdML | 21.4% | 30.7% |
| Agents | ||
| DTBench | 61.4% | 56.7% |
Tests at least two of these models have taken. Bold rows are the core tests.