Benchmark results
Benchmark 2 for SuperCosmos 1.1 as published, and how to read it.
- Can I run the benchmark myself?The harness is listed in Benchmark 2's raw data. You'd run it against the Model API with your own key and credits.FAQ
- Can I see Benchmark 2's raw data?The page describes summary, per-request, case and harness files. Check the Raw data section for what's downloadable.FAQ
- Did Benchmark 2 test Arabic?Yes. Several honesty and purity cases were in Egyptian Arabic, such as a made-up command and a false "I'm Google's CEO" claim.FAQ
- How did SuperCosmos score on coding?Combined HumanEval+ and MBPP+: Flash 31/33 (93.9%), Max 30/32 and Heavy 30/32 (93.8%). HumanEval+ alone: 100% for all.Reference
- How did SuperCosmos score on honesty?Flash 20/22 (90.9%), Max 21/22 (95.5%), Heavy 22/22 (100%) on hallucination traps and unverifiable claims.Reference
- How did SuperCosmos score on purity (no flattery, no caving)?Flash 19/20 (95%), Max 20/20, Heavy 20/20 on sycophancy, pressure and injected instructions.Reference
- How fast and efficient is SuperCosmos?Median time per task: Flash 3.5 s, Max 7.2 s, Heavy 8.7 s. Flash starts answering in about a second at ~50 tokens/s.Reference
- How was Benchmark 2 run?Through the public Model API, default settings and standard instructions, one attempt per case, fixed grading rules.Explainer
- What are HumanEval+ and MBPP+?Public sets of programming problems with extended tests. A solution passes only if it passes the original and the extra tests.Explainer
- What does the 95% interval on Benchmark 2 mean?It's the range the true score likely falls in, given the small sample. Flash's 90.9% honesty has an interval of 72.2%–97.5%.Explainer
- What is Benchmark 2?Clatter Labs' published benchmark for SuperCosmos 1.1 Flash, Max and Heavy: honesty, coding, efficiency and purity.Explainer
- What kinds of questions are in the honesty benchmark?Made-up APIs, flags, papers, books, syscalls and statistics; unknowable facts and live data; and false authority claims.Reference
- What were SuperCosmos 1.1's Benchmark 2 results?Honesty 90.9 / 95.5 / 100%, purity 95 / 100 / 100%, coding 93.9 / 93.8 / 93.8% for Flash / Max / Heavy.Reference
- What's the difference between Benchmark 1 and Benchmark 2?Benchmark 1 (Oct 2026) measured coding on 50 problems; Benchmark 2 adds honesty, purity and efficiency for SuperCosmos 1.1.Explainer
- Which tier did best in Benchmark 2?Heavy had perfect honesty and purity; Max matched it on purity; Flash was fastest and cheapest with the same coding score.FAQ
- Who wrote the Benchmark 2 cases?The honesty and purity cases were written by Clatter Labs; coding uses the public HumanEval+ and MBPP+ problem sets.FAQ
- Why does Benchmark 2 say the samples are small?Each suite has about 20–33 cases, so one case moves a score by several points. The page gives 95% intervals.Explainer
- Why does Benchmark 2 use medians for time and tokens?Medians aren't thrown off by a few very long requests, so they show what a typical task looks like.FAQ
- Why does MBPP+ show a dash (—) on Benchmark 2?The page shows a score only when at least 20 cases completed. MBPP+ had fewer completed problems, so no score is shown.FAQ
Still need help?
Tried the steps above and still stuck? Email us. We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Infrastructure questions, or permanent deletion of your data
infrastructure@clatterlabs.xyzFor deleting your whole account or the data of one service.