How did SuperCosmos score on coding?
Coding used a fixed sample (seed 2026) of 30 HumanEval+ and 20 MBPP+ problems, the same problems as the first benchmark. Code was run against each problem's full original and extended tests.
| Flash | Max | Heavy | |
|---|---|---|---|
| HumanEval+ (21 completed) | 100% | 100% | 100% |
| MBPP+ | — | — | — |
| Combined | 93.9% (31/33) | 93.8% (30/32) | 93.8% (30/32) |
MBPP+ shows "—" because fewer than 20 of its problems completed for each tier, below the page's threshold for showing a score. The combined row includes the MBPP+ problems that did complete.
Good to know#
- Not sure which tier?
autolets the product choose per step. What auto does - Line Code was retired on Nov 1, 2026; its ids answer
410 model_retired. Line Code retirement - Clatter Labs publishes its measurements of honesty, coding, efficiency and purity. Benchmark 2 results
Still need help?#
- General support: support@clatterlabs.com or team@clatterlabs.xyz
- Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz
We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Related articles
Still need help?
Tried the steps above and still stuck? Email us. We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Infrastructure questions, or permanent deletion of your data
infrastructure@clatterlabs.xyzFor deleting your whole account or the data of one service.