What were SuperCosmos 1.1's Benchmark 2 results?
Pass rates as published on Benchmark 2:
| Benchmark | Cases | Flash | Max | Heavy |
|---|---|---|---|---|
| Honesty | 22 | 90.9% (20/22) | 95.5% (21/22) | 100% (22/22) |
| HumanEval+ | 30 sampled, 21 completed | 100% | 100% | 100% |
| MBPP+ | 20 sampled | — | — | — |
| Coding, combined | HumanEval+ and MBPP+ | 93.9% (31/33) | 93.8% (30/32) | 93.8% (30/32) |
| Purity | 22 sampled, 20 completed | 95.0% (19/20) | 100% (20/20) | 100% (20/20) |
| Efficiency | Flash | Max | Heavy |
|---|---|---|---|
| Output tokens per task (median) | 57 | 206 | 204 |
| Time per task (median) | 3.5 s | 7.2 s | 8.7 s |
| Cost per 1,000 tasks | $1.64 | $2.57 | $3.64 |
"—" means fewer than 20 cases completed, so no score is shown. These are small samples: differences of a few points are within noise.
Still need help?#
- General support: support@clatterlabs.com or team@clatterlabs.xyz
- Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz
We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Related articles
Still need help?
Tried the steps above and still stuck? Email us. We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Infrastructure questions, or permanent deletion of your data
infrastructure@clatterlabs.xyzFor deleting your whole account or the data of one service.