417 replicated study runs across 5 domains. Mid-range scores and failures included.
Read the paper↗︎// Read this first
Prediction can be right for the wrong reason. We test interventions and compare their estimated effects with confidence intervals.
Each score compares a reconstructed experiment with the original human study.
Rank order, effect direction, parameter distance, and uncertainty measure different parts of the match.
A benchmark is evidence, not a universal guarantee.
// Protocol
Reconstruct the study. Match the respondent profile. Run the experiment. Estimate the original model.
Compare the result with the human data and publish the limitation.
Every score has a source and boundary.
Live rankings
| Model | Score | Coverage | Match | Studies |
|---|---|---|---|---|
| Claude Sonnet (Databricks) | 56.78 | 40.07 | 82.63 | 19 |
| GPT-4 (Azure OpenAI) | 55.62 | 46.02 | 78.21 | 19 |
| Claude Sonnet (Databricks) | 54.63 | 45.79 | 70.95 | 23 |
| Gemini Flash (Google) | 54.63 | 23.84 | 81.59 | 19 |
| GPT-4 (Azure OpenAI) | 53.87 | 35.27 | 73.22 | 2 |
| Claude Haiku (Databricks) | 53.62 | 39.92 | 76.60 | 19 |
| GPT-3 Instruct (Azure OpenAI) | 52.79 | 41.50 | 75.97 | 18 |
| GPT-4 (Azure OpenAI) | 52.67 | 34.92 | 78.74 | 9 |
| Gemini (Google) | 51.09 | 33.46 | 76.83 | 9 |
| Claude Sonnet (Databricks) | 50.82 | 28.54 | 77.55 | 9 |
Each table preserves the original domain-level ranking surface from the legacy leaderboard.
8 models ranked
| Model | Score | Coverage | Match | Studies |
|---|---|---|---|---|
| Claude Sonnet (Databricks) | 54.63 | 45.79 | 70.95 | 23 |
| Claude Haiku (Databricks) | 50.30 | 44.93 | 65.83 | 23 |
| GPT-4 (Azure OpenAI) | 50.23 | 43.12 | 67.59 | 23 |
| Gemini Flash (Google) | 49.48 | 25.37 | 68.93 | 22 |
| o1 (Azure OpenAI) | 48.49 | 25.67 | 68.72 | 20 |
| Gemini (Google) | 39.71 | 40.47 | 55.42 | 23 |
| GPT-3 Instruct (Azure OpenAI) | 38.94 | 41.12 | 52.27 | 22 |
| o3-mini (Azure OpenAI) | 37.51 | 21.05 | 55.21 | 22 |
8 models ranked
| Model | Score | Coverage | Match | Studies |
|---|---|---|---|---|
| Claude Sonnet (Databricks) | 56.78 | 40.07 | 82.63 | 19 |
| GPT-4 (Azure OpenAI) | 55.62 | 46.02 | 78.21 | 19 |
| Gemini Flash (Google) | 54.63 | 23.84 | 81.59 | 19 |
| Claude Haiku (Databricks) | 53.62 | 39.92 | 76.60 | 19 |
| GPT-3 Instruct (Azure OpenAI) | 52.79 | 41.50 | 75.97 | 18 |
| o1 (Azure OpenAI) | 50.65 | 25.33 | 75.20 | 19 |
| Gemini (Google) | 50.01 | 37.12 | 74.81 | 19 |
| o3-mini (Azure OpenAI) | 33.11 | 17.91 | 56.59 | 19 |
8 models ranked
| Model | Score | Coverage | Match | Studies |
|---|---|---|---|---|
| GPT-4 (Azure OpenAI) | 52.67 | 34.92 | 78.74 | 9 |
| Gemini (Google) | 51.09 | 33.46 | 76.83 | 9 |
| Claude Sonnet (Databricks) | 50.82 | 28.54 | 77.55 | 9 |
| Gemini Flash (Google) | 47.47 | 23.81 | 77.87 | 9 |
| GPT-3 Instruct (Azure OpenAI) | 45.03 | 21.58 | 75.79 | 9 |
| Claude Haiku (Databricks) | 43.92 | 31.17 | 73.57 | 9 |
| o1 (Azure OpenAI) | 38.40 | 18.67 | 65.19 | 9 |
| o3-mini (Azure OpenAI) | 33.19 | 4.77 | 63.35 | 9 |
4 models ranked
| Model | Score | Coverage | Match | Studies |
|---|---|---|---|---|
| GPT-4 (Azure OpenAI) | 43.18 | 24.30 | 53.48 | 2 |
| Claude Sonnet (Databricks) | 38.52 | 29.08 | 47.40 | 2 |
| Claude Haiku (Databricks) | 32.56 | 28.30 | 43.32 | 2 |
| Gemini (Google) | 29.32 | 28.98 | 44.80 | 2 |
4 models ranked
| Model | Score | Coverage | Match | Studies |
|---|---|---|---|---|
| GPT-4 (Azure OpenAI) | 53.87 | 35.27 | 73.22 | 2 |
| Claude Sonnet (Databricks) | 44.64 | 15.63 | 68.58 | 2 |
| Claude Haiku (Databricks) | 44.00 | 18.51 | 62.30 | 2 |
| Gemini (Google) | 43.12 | 29.16 | 60.10 | 2 |