What's the difference between Benchmark 1 and Benchmark 2?
| Benchmark 1 | Benchmark 2 | |
|---|---|---|
| Models | SuperCosmos (1.0) and Line Code | SuperCosmos 1.1 Flash, Max, Heavy |
| What | Coding: 50 HumanEval+ and MBPP+ problems, one attempt | Honesty, coding (same problems), efficiency, purity |
| SuperCosmos coding | Flash 90%, Max 88%, Heavy 90% | Combined 93.9% / 93.8% / 93.8% |
Coding numbers aren't directly comparable: Benchmark 2 shows combined scores over the problems that completed, and sample sizes are small either way.
Good to know#
- SuperCosmos reads text only; it doesn't take image input. What it can't do
- Every tier shares the same 1M-token context window (1,048,576 tokens). Context window
- Insulting or cursing at the assistant earns a warning, then a 24-hour pause on a second strike within 30 days. Abuse policy
Still need help?#
- General support: support@clatterlabs.com or team@clatterlabs.xyz
- Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz
We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Related articles
Still need help?
Tried the steps above and still stuck? Email us. We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Infrastructure questions, or permanent deletion of your data
infrastructure@clatterlabs.xyzFor deleting your whole account or the data of one service.