Models21
Prompts154
Execution91.22%
Assembly91.03%
Signature52.66%
ES-Gap38.56 pp
Measurement ladder
Scroll horizontally to inspect the complete chart.
Model profiles
Scroll horizontally to inspect the complete chart.
Recovery matrix
Scroll horizontally to inspect the complete chart.
Repeatability
Scroll horizontally to inspect the complete chart.
Provider aggregates
Scroll horizontally to inspect the complete chart.
Model-level data
| Model | Provider route | Execution | Assembly | Signature | Ordered | Parameter | Signature AC1 |
|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | OpenAI | 99.35% | 99.35% | 62.99% | 58.44% | 57.14% | 1.000 |
| Claude Fable 5 | Anthropic | 98.05% | 98.05% | 62.99% | 58.44% | 57.79% | 0.947 |
| Gemini 3.1 Pro Preview | 96.75% | 96.75% | 61.04% | 55.84% | 55.19% | 1.000 | |
| GPT-5.4 mini | OpenAI | 98.05% | 97.40% | 60.39% | 56.49% | 55.19% | 0.945 |
| GPT-5.5 | OpenAI | 97.40% | 97.40% | 60.39% | 55.84% | 55.19% | 0.982 |
| Claude Opus 4.8 | Anthropic | 100.00% | 99.35% | 59.74% | 56.49% | 55.19% | 1.000 |
| Claude Sonnet 4.6 | Anthropic | 99.35% | 99.35% | 59.09% | 54.55% | 53.90% | 0.946 |
| DeepSeek V4 Pro | DeepSeek | 91.56% | 91.56% | 59.09% | 56.49% | 54.55% | 0.816 |
| Codestral 25.01 | GitHub Models | 93.51% | 93.51% | 55.84% | 50.65% | 48.05% | 0.944 |
| Gemini 2.5 Pro | 88.31% | 88.31% | 53.90% | 50.00% | 50.00% | 0.833 | |
| GPT-OSS 120B | Groq | 96.75% | 96.10% | 53.25% | 50.00% | 48.70% | 0.636 |
| DeepSeek V4 Flash | DeepSeek | 88.96% | 88.96% | 52.60% | 50.00% | 49.35% | 0.815 |
| GPT-OSS 20B | Groq | 92.86% | 92.21% | 50.65% | 47.40% | 45.45% | 0.553 |
| Qwen3-Coder-Next | Hugging Face | 85.71% | 85.71% | 50.65% | 44.81% | 44.81% | 0.907 |
| Llama 4 Maverick | GitHub Models | 94.16% | 93.51% | 48.70% | 46.10% | 46.10% | 0.629 |
| Mistral Small 3.2 24B | OpenRouter | 90.26% | 90.26% | 48.70% | 43.51% | 42.21% | 0.889 |
| Llama 3.3 70B | Groq | 93.51% | 93.51% | 46.10% | 40.91% | 40.26% | 0.853 |
| Qiskit Mistral 3.2 24B | Hugging Face | 89.61% | 88.96% | 44.81% | 42.21% | 41.56% | 0.572 |
| Llama 4 Scout | Groq | 82.47% | 82.47% | 44.81% | 39.61% | 38.96% | 0.835 |
| Llama 3.1 8B | Groq | 81.17% | 81.17% | 35.06% | 33.77% | 33.77% | 0.802 |
| Qwen3 32B | Groq | 57.79% | 57.79% | 35.06% | 31.82% | 29.87% | 0.661 |