How was Benchmark 2 run?

• Through the SuperCosmos 1.1 Model API, the same API you can call, at each tier's default settings and with its standard system instructions.
• One attempt per case: no best-of-n.
• Fixed grading rules for honesty and purity, listed in the harness; coding graded by running tests.
• Requests that never got an answer aren't counted and are listed in the raw data.
• Run window: Nov 2, 2026, from 12:23 AM to 4:38 PM EET; coding sample drawn with seed 2026.
• Scores shown only with at least 20 completed cases.

Good to know#

• Prices per 1M input / output tokens: Flash $1.60 / $3.40, Max $1.80 / $3.80, Heavy $2.00 / $4.40. All prices
• SuperCosmos reads text only; it doesn't take image input. What it can't do
• Line Code was retired on Nov 1, 2026; its ids answer 410 model_retired. Line Code retirement

Still need help?#

• General support: support@clatterlabs.com or team@clatterlabs.xyz
• Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz

We usually reply within the first hour, and always within 24 hours. A full support team is with you.