What are HumanEval+ and MBPP+?
HumanEval+ and MBPP+ are public collections of short programming problems. The "+" versions add many extra tests to the original ones, which catches solutions that only look right. In Benchmark 2, code passed only if it passed each problem's full original and extended tests, with one attempt per problem.
Good to know#
- Prices per 1M input / output tokens: Flash $1.60 / $3.40, Max $1.80 / $3.80, Heavy $2.00 / $4.40. All prices
- SuperCosmos 1.1 is built to say "I don't know" rather than invent facts. How it's built to be honest
- Every tier shares the same 1M-token context window (1,048,576 tokens). Context window
Still need help?#
- General support: support@clatterlabs.com or team@clatterlabs.xyz
- Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz
We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Related articles
Still need help?
Tried the steps above and still stuck? Email us. We usually reply within the first hour, and always within 24 hours. A full support team is with you.
Infrastructure questions, or permanent deletion of your data
infrastructure@clatterlabs.xyzFor deleting your whole account or the data of one service.