How do I stream Model API replies through a FastAPI backend?

import os
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI

app = FastAPI()
client = AsyncOpenAI(
    base_url="https://api.gateway.clatterlabs.xyz/v1",
    api_key=os.environ["CLATTER_API_KEY"],
    default_headers={"X-Clatter-Account": os.environ["CLATTER_ACCOUNT_ID"]},
)

@app.post("/chat")
async def chat(body: dict):
    async def gen():
        stream = await client.chat.completions.create(
            model="supercosmos-flash",
            messages=[{"role": "user", "content": body["message"]}],
            stream=True,
        )
        async for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content
    return StreamingResponse(gen(), media_type="text/plain")

Authenticate your own users before calling the API, and moderate their input: your account is responsible for it.

Good to know#

• If calls fail unexpectedly, check status.clatterlabs.com before debugging further.
• Your key's account is responsible for your end users' text under the Abuse policy. What it means for apps
• Every response has an X-Request-Id; log it so a request can be found later. Why log it

Still need help?#

• General support: support@clatterlabs.com or team@clatterlabs.xyz
• Infrastructure questions, or permanent deletion of your data (your whole account or one service): infrastructure@clatterlabs.xyz

We usually reply within the first hour, and always within 24 hours. A full support team is with you.