WeirdML v3
11 hand-made machine-learning and data-analysis tasks with unfamiliar data, unstated goals and very little feedback, worked inside a coding agent. The score is the mean of each task's final result, normalized and with penalties for hints, not raw accuracy.
Writing, fixing and shipping real code, judged by tests or users. One of 6 comparable tests in coding, a category the Index averages.
Read with care. This is the leaderboard's Final Best Score, the measure Epoch AI's capability index reads; the leaderboard's own headline score also rewards using fewer tokens. Models run inside different coding agents, so this compares model-plus-agent systems.
Publisher: Håvard Tveit IhleWhat a model is asked to do
Work through a hand-made machine-learning problem inside a coding agent: explore unfamiliar data, build a pipeline and deliver a result with almost no feedback.
For exampleA folder of unlabeled sensor recordings and a one-line brief: find out what sets the faulty units apart, then predict which of the held-back ones will fail.
Why it matters. Real data work starts with a vague goal and a messy folder. This rewards models that investigate before they train anything.
The example is original and illustrative, not an item from the dataset.
- Models scored here
- 25
- Who produced the numbers
- Official leaderboard
- Items graded
- n = 11 (9.1% each)
- Best published result
- 54.0%
- Newest release covered
- October 26, 2026 (active)
- License
- CC BY 4.0 (via Epoch AI)
- Direction
- Higher is better
- Scale
- Published as a share of items solved. For the Index, each score is a share of the best published result (54.0%), so the frontier reads 100. Guessing earns nothing on this test, so no chance correction applies.
- Provenance
- Produced by the benchmark's own leaderboard or a third-party evaluator, not by the model's maker.
Snapshot November 10, 2026
Ranking on this test
25 models on WeirdML v3
Best published run per model, and how close each one comes to the best published result on this test (the frontier reads 100). Every number links to the publisher that produced it. Internal runs are marked and carry their run details on the model page. One item on this test is 9.1 points, so read gaps smaller than that as noise.
| # | Model | Score | vs frontier | Run | Produced by |
|---|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAI | 54.0% | 100.0 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 2 | GPT-6.1 SolOpenAI | 50.6% | 93.6 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 3 | Claude Opus 5.5Anthropic | 49.8% | 92.2 | xhigh · agent: claude_code · reasoning effort: xhigh | Leaderboard |
| 4 | Claude Fable 5.1Anthropic | 43.5% | 80.5 | xhigh · agent: claude_code · reasoning effort: xhigh | Leaderboard |
| 5 | Gemini 4 ArgonGoogle | 41.4% | 76.6 | high · agent: gemini_cli · reasoning effort: high | Leaderboard |
| 6 | GPT-6.1 TerraOpenAI | 40.4% | 74.8 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 7 | Claude Sonnet 5.5Anthropic | 37.0% | 68.4 | xhigh · agent: claude_code · reasoning effort: xhigh | Leaderboard |
| 8 | Claude Opus 5Anthropic | 32.5% | 60.2 | xhigh · agent: claude_code · reasoning effort: xhigh | Leaderboard |
| 9 | Claude Fable 5Anthropic | 27.9% | 51.6 | xhigh · agent: claude_code · reasoning effort: xhigh | Leaderboard |
| 10 | GPT-6 SolOpenAI | 27.6% | 51.0 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 11 | Muse WonderMeta | 24.9% | 46.1 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 12 | Claude Haiku 5.5Anthropic | 24.1% | 44.6 | xhigh · agent: claude_code · reasoning effort: xhigh | Leaderboard |
| 13 | GPT-5.6 SolOpenAI | 21.4% | 39.7 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 14 | DeepSeek V4.1 FlashDeepSeek | 19.9% | 36.9 | high · agent: codex_cli · reasoning effort: high | Leaderboard |
| 15 | Kimi K3.1 ProMoonshot AI | 18.9% | 35.0 | xhigh · agent: kimi_code · reasoning effort: xhigh | Leaderboard |
| 16 | Grok 4.7xAI | 18.7% | 34.6 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 17 | Kimi K3Moonshot AI | 15.4% | 28.5 | high · agent: kimi_code · reasoning effort: high | Leaderboard |
| 18 | Gemini 3.8 FlashGoogle | 14.6% | 27.0 | high · agent: gemini_cli · reasoning effort: high | Leaderboard |
| 19 | Mistral Large 4Mistral AI | 12.2% | 22.6 | high · agent: opencode · reasoning effort: high | Leaderboard |
| 20 | Muse Spark 1.3Meta | 12.1% | 22.4 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 21 | GPT-6 LunaOpenAI | 12.0% | 22.2 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 22 | GPT-5.2OpenAI | 9.7% | 18.0 | 2025-12-11 · xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 23 | GPT-5.6 LunaOpenAI | 9.3% | 17.3 | xhigh · agent: codex_cli · reasoning effort: xhigh | Leaderboard |
| 24 | Claude Opus 4.5Anthropic | 7.1% | 13.2 | 20251101 · high · agent: claude_code · reasoning effort: high | Leaderboard |
| 25 | InklingThinking Machines | 4.4% | 8.1 | xhigh · agent: opencode · reasoning effort: xhigh | Leaderboard |
The best models, ranked here, working inside Theo.
29 of the models on this page run inside Theo today. Theo picks the right one for each step and always shows which engine answered.