Cloudflare AI

Cloudflare runs open models like Llama behind an OpenAI-compatible endpoint. Point your /api/chat route at it, stream the reply back, and <kai-chat> never knows the difference.

Workers AI from any server

Workers AI speaks the OpenAI wire format, so the route is the one you’d write for any other model. Send a Bearer token, set stream: true, and prefix the model id with @cf/.

// app/api/chat/route.ts — proxy Workers AI, keep the token server-side
export async function POST(req: Request) {
  const { messages } = await req.json();

  const upstream = await fetch(
    `https://api.cloudflare.com/client/v4/accounts/${process.env.CF_ACCOUNT_ID}/ai/v1/chat/completions`,
    {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${process.env.CF_API_TOKEN}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: '@cf/meta/llama-3.1-8b-instruct',
        messages,
        stream: true,
      }),
    },
  );

  // Workers AI returns OpenAI-format SSE — pass it straight through.
  return new Response(upstream.body, { headers: { 'Content-Type': 'text/event-stream' } });
}

The browser side is unchanged: <kai-chat> reads the OpenAI-format SSE coming back. The reader loop lives in the Streaming recipe — reference it rather than rewriting it here.

Field	Value
Endpoint	`https://api.cloudflare.com/client/v4/accounts/ACCOUNT_ID/ai/v1/chat/completions`
Auth	`Authorization: Bearer API_TOKEN`
Example model	`@cf/meta/llama-3.1-8b-instruct`

Route through AI Gateway

AI Gateway sits in front of providers and adds caching, rate limits, retries, and per-request logs without touching your route logic. Swap the base URL for the gateway’s compat endpoint and prefix the model id with workers-ai/.

const upstream = await fetch(
  `https://gateway.ai.cloudflare.com/v1/${process.env.CF_ACCOUNT_ID}/${process.env.CF_GATEWAY_ID}/compat/chat/completions`,
  {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.CF_API_TOKEN}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'workers-ai/@cf/meta/llama-3.1-8b-instruct',
      messages,
      stream: true,
    }),
  },
);

The gateway can front other providers too — change the model prefix to route OpenAI or Anthropic traffic through the same observability layer. See the chat completion docs for the full prefix list.

Already on Workers? Use the AI binding

If your /api/chat route runs inside a Worker, skip HTTP entirely and call the native env.AI binding. Ask for stream: true and Workers AI returns a text/event-stream body you return directly.

// Worker handler — env.AI is bound in wrangler.toml
export default {
  async fetch(req: Request, env: Env): Promise<Response> {
    const { messages } = await req.json();

    const stream = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
      messages,
      stream: true,
    });

    return new Response(stream, { headers: { 'Content-Type': 'text/event-stream' } });
  },
};

No account id, no token, no fetch — the binding handles auth.

Next steps

Connect any model — the gateway pattern this fits into, plus a model switcher.
Streaming — the reader loop that pulls the SSE reply into the thread.
Connect any backend — the messages contract every route here builds on.