Can I see Benchmark 2's raw data?

The Raw data section of Benchmark 2 describes four files: summary.json (every aggregate, with intervals), items.jsonl (one line per model × case: pass, tokens, latency, cost), samples.json (the exact case and problem IDs) and harness.tar.gz (harness, prompts and grading rules). Check that section for what's currently available to download.

The first benchmark at the first benchmark published its summary, items, samples and harness so you can rerun it.

Good to know#

• Not sure which tier? auto lets the product choose per step. What auto does
• Insulting or cursing at the assistant earns a warning, then a 24-hour pause on a second strike within 30 days. Abuse policy
• SuperCosmos reads text only; it doesn't take image input. What it can't do

Still need help?#

• General support: support@clatterlabs.com or team@clatterlabs.xyz
• Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz

We usually reply within the first hour, and always within 24 hours. A full support team is with you.