Real-Time Response Streaming with the Vercel AI SDK in Next.js — So Users Never Wait

In our previous tutorial on integrating OpenAI and Claude APIs into Next.js to launch our first chatbot agent, we built a dual-provider adapter that connects GPT-4o and Claude 3.5 Sonnet to our backend. While that architecture returned accurate, well-formatted technical answers, you probably noticed an annoying UX bottleneck the moment you asked a detailed question: the UI froze on 'Thinking...' for five to eight seconds before dumping the entire three-paragraph response onto the screen at once.

To eliminate that multi-second waiting period in a Next.js AI agent, you use the Vercel AI SDK (ai package) with streamText and createStreamableValue to pipe generated tokens from OpenAI or Anthropic to the browser as they are produced. Instead of blocking the HTTP response until the final token is complete, response streaming lowers your Time to First Token (TTFT) to roughly 250–450 milliseconds while still allowing your server to capture final token counts and latency telemetry via the onFinish callback.

In a standard database-backed web app, users expect responses in under 200 milliseconds. Large Language Models, however, generate text sequentially—one token at a time—at roughly 50 to 90 tokens per second. If your DevPulse agent generates a 400-token debugging guide, waiting for the full completion takes up to six seconds of dead silence. Streaming transforms that dead silence into an immediate, live conversation.

Why Blocking LLM Calls Break User Experience (and Serverless Timeouts)

Waiting six seconds is not merely a visual annoyance; blocking LLM responses create two concrete engineering problems in production Next.js applications:

1. Perceived Application Hang: When a user submits a bug report and sees zero visual progress for five seconds, they often assume their network dropped and click the submit button again, triggering duplicate API bills.
2. Serverless Gateway Timeouts: Many edge proxies and serverless platforms enforce strict time-to-first-byte (TTFB) limits. When you stream the response, the server sends HTTP headers and the first chunk almost immediately, keeping the connection alive even if the total generation takes twenty seconds.

Before the Vercel AI SDK standardized LLM streaming in TypeScript, wiring this up manually meant wrestling with raw ReadableStream controllers, TextEncoder byte buffers, and completely different Server-Sent Event (SSE) chunk formats between OpenAI (choices[0].delta.content) and Anthropic (content_block_delta). The Vercel AI SDK abstracts those vendor-specific chunk formats behind a single unified streamText primitive.

How the Vercel AI SDK Streaming Pipeline Works

Instead of throwing away the clean unified architecture we built in Posts 12, 13, and 14, we are upgrading our LLM layer to use @ai-sdk/openai and @ai-sdk/anthropic alongside the core ai package.

Here is how token chunks and final telemetry flow from the LLM provider through our Next.js backend to the client:

`mermaid sequenceDiagram participant Client as React Client UI participant SA as Next.js Server Action / Route participant SDK as Vercel AI SDK (streamText) participant LLM as OpenAI / Anthropic API

Client->>SA: Submit sanitized messages[] + modelId
SA->>SDK: Invoke streamText(model, system, messages)
SDK->>LLM: Open streaming HTTP connection
LLM-->>SDK: Emit token delta chunk 1 (~300ms TTFT)
SDK-->>SA: Push chunk into StreamableValue / DataStream
SA-->>Client: Render incremental text in real time
LLM-->>SDK: Emit final token + usage stats
SDK->>SA: Trigger onFinish({ text, usage })
SA-->>Client: Close stream & deliver final AgentStepTrace

`

Architecture Flow Summary: The client submits the conversation payload to the Next.js server handler, which invokes streamText from the Vercel AI SDK. As OpenAI or Anthropic emits token deltas (~300ms TTFT), the SDK pipes each chunk immediately to the client via a streamable value while awaiting completion on the server to trigger onFinish, record token usage in AgentStepTrace, and close the stream cleanly.

Step 1: Installing the Vercel AI SDK and Provider Packages

In your Next.js project root, install the core ai library along with the official OpenAI and Anthropic provider modules:

bash npm install ai @ai-sdk/openai @ai-sdk/anthropic zod server-only

Your existing .env.local file from Post 14 (OPENAI_API_KEY and ANTHROPIC_API_KEY) works out of the box—the @ai-sdk/openai and @ai-sdk/anthropic packages automatically read those exact environment variable names on the server.

Step 2: Building the Unified Streaming Engine (src/lib/agent/stream-client.ts)

Let us create src/lib/agent/stream-client.ts. Notice how we reuse our sanitizeConversationHistory() function from Post 14 so Anthropic's role-alternation rules and our 10-message sliding window remain strictly enforced before streaming begins.

`typescript // src/lib/agent/stream-client.ts import 'server-only'; import { streamText, type LanguageModelV1 } from 'ai'; import { createOpenAI } from '@ai-sdk/openai'; import { createAnthropic } from '@ai-sdk/anthropic'; import { type AgentConfig } from './blueprint'; import { sanitizeConversationHistory, type ChatMessage, } from './llm-client';

function resolveStreamingModel( modelId: AgentConfig['modelId'] ): LanguageModelV1 { if (modelId === 'gpt-4o') { const apiKey = process.env.OPENAI_API_KEY; if (!apiKey) { throw new Error('Missing OPENAI_API_KEY in environment variables.'); } const openai = createOpenAI({ apiKey }); return openai('gpt-4o'); }

const apiKey = process.env.ANTHROPIC_API_KEY; if (!apiKey) { throw new Error('Missing ANTHROPIC_API_KEY in environment variables.'); } const anthropic = createAnthropic({ apiKey }); return anthropic('claude-3-5-sonnet-latest'); }

export interface StreamTelemetry { modelUsed: string; promptTokens: number; completionTokens: number; latencyMs: number; }

export function createAgentTextStream( messages: ChatMessage[], config: AgentConfig, onTelemetryReady?: (telemetry: StreamTelemetry) => void ) { const startTime = Date.now(); const safeMessages = sanitizeConversationHistory(messages, 10); const model = resolveStreamingModel(config.modelId);

return streamText({ model, system: config.systemPrompt, messages: safeMessages, temperature: config.temperature, maxTokens: 1024, onFinish: ({ usage }) => { if (onTelemetryReady) { onTelemetryReady({ modelUsed: config.modelId, promptTokens: usage.promptTokens, completionTokens: usage.completionTokens, latencyMs: Date.now() - startTime, }); } }, }); } `

Look at how clean createAgentTextStream is compared to managing two separate vendor SDKs manually:

• Unified Model Interface (LanguageModelV1): Whether resolveStreamingModel returns openai('gpt-4o') or anthropic('claude-3-5-sonnet-latest'), streamText accepts the exact same system, messages, temperature, and maxTokens configuration.
• Non-Blocking Telemetry via onFinish: One of the biggest headaches with raw streaming is figuring out how many tokens a call consumed, since token usage only arrives in the very last packet of the stream. The Vercel AI SDK automatically aggregates the stream and fires onFinish({ usage }) the millisecond generation completes.

Step 3: Streaming from Both Server Actions and Route Handlers

In a Next.js App Router project, there are two ways to expose your Vercel AI SDK stream to the frontend, and knowing both is essential:

1. Server Action Streaming (createStreamableValue): Ideal when you want to stream both incremental text chunks and custom structured objects (like our AgentStepTrace telemetry) over a type-safe RPC call.
2. Route Handler Streaming (toDataStreamResponse): The standard HTTP endpoint (src/app/api/chat/route.ts) that powers React's useChat hook—which we will connect on the frontend in Post 16.

Let us write both right now so our backend is 100% ready for custom RPC streams and the useChat hook.

First, add the streaming Server Action in src/app/actions/stream-agent.ts:

`typescript // src/app/actions/stream-agent.ts 'use server';

import { z } from 'zod'; import { createStreamableValue } from 'ai/rsc'; import { defaultDevPulseConfig, type AgentStepTrace, } from '@/lib/agent/blueprint'; import { type ChatMessage } from '@/lib/agent/llm-client'; import { createAgentTextStream } from '@/lib/agent/stream-client';

const streamRequestSchema = z.object({ sessionId: z.string().min(8), modelId: z .enum(['gpt-4o', 'claude-3-5-sonnet-latest']) .default('claude-3-5-sonnet-latest'), messages: z .array( z.object({ role: z.enum(['user', 'assistant']), content: z.string().trim().min(1).max(4000), }) ) .min(1), });

export async function streamDevPulseAgent(rawInput: unknown) { const textStream = createStreamableValue(''); const traceStream = createStreamableValue<AgentStepTrace | null>(null);

const parsed = streamRequestSchema.safeParse(rawInput); if (!parsed.success) { textStream.error( parsed.error.issues.map((i) => i.message).join('; ') ); traceStream.done(null); return { textStream: textStream.value, traceStream: traceStream.value, }; }

const { messages, modelId } = parsed.data; const activeConfig = { ...defaultDevPulseConfig, modelId };

(async () => { try { const result = createAgentTextStream( messages as ChatMessage[], activeConfig, (telemetry) => { traceStream.done({ stepIndex: 1, thoughtSummary: Streamed via ${telemetry.modelUsed} (${telemetry.promptTokens} in / ${telemetry.completionTokens} out tokens), durationMs: telemetry.latencyMs, }); } );

  for await (const delta of result.textStream) {
    textStream.append(delta);
  }

  textStream.done();
} catch (err) {
  const message =
    err instanceof Error ? err.message : 'Streaming interrupted';
  textStream.error(message);
  traceStream.done(null);
}

})();

return { textStream: textStream.value, traceStream: traceStream.value, }; } `

Notice how the async IIFE (async () => { ... })() runs in the background while streamDevPulseAgent immediately returns { textStream: textStream.value, traceStream: traceStream.value } to the client. As for await (const delta of result.textStream) receives words from OpenAI or Claude, textStream.append(delta) pushes only the new characters over the wire.

Second, let us also create the companion Route Handler in src/app/api/chat/route.ts that exposes the exact same createAgentTextStream engine via toDataStreamResponse():

`typescript // src/app/api/chat/route.ts import { NextRequest, NextResponse } from 'next/server'; import { z } from 'zod'; import { defaultDevPulseConfig } from '@/lib/agent/blueprint'; import { type ChatMessage } from '@/lib/agent/llm-client'; import { createAgentTextStream } from '@/lib/agent/stream-client';

export const maxDuration = 60;

const routePayloadSchema = z.object({ modelId: z .enum(['gpt-4o', 'claude-3-5-sonnet-latest']) .default('claude-3-5-sonnet-latest'), messages: z .array( z.object({ role: z.enum(['user', 'assistant']), content: z.string().min(1), }) ) .min(1), });

export async function POST(req: NextRequest) { try { const body = await req.json(); const parsed = routePayloadSchema.safeParse(body);

if (!parsed.success) {
  return NextResponse.json(
    { error: 'Invalid chat payload', details: parsed.error.flatten() },
    { status: 400 }
  );
}

const { messages, modelId } = parsed.data;
const result = createAgentTextStream(messages as ChatMessage[], {
  ...defaultDevPulseConfig,
  modelId,
});

return result.toDataStreamResponse();

} catch (err) { const message = err instanceof Error ? err.message : 'Internal streaming error'; return NextResponse.json({ error: message }, { status: 500 }); } } `

Step 4: Consuming the Stream in React with readStreamableValue

In Post 16, we will dive deep into React's useChat hook to manage full chat state, stop buttons, and automatic scrolling against /api/chat. Today, let us see how to consume our streamDevPulseAgent Server Action directly using readStreamableValue so you can watch the live token deltas and telemetry trace arrive in your test console:

`tsx // src/components/StreamTestConsole.tsx 'use client';

import { useState } from 'react'; import { readStreamableValue } from 'ai/rsc'; import { streamDevPulseAgent } from '@/app/actions/stream-agent'; import { type AgentStepTrace } from '@/lib/agent/blueprint';

export default function StreamTestConsole() { const [prompt, setPrompt] = useState( 'Explain how our API gateway 429 rate limiting and token bucket burst rules work.' ); const [streamedReply, setStreamedReply] = useState(''); const [trace, setTrace] = useState<AgentStepTrace | null>(null); const [isStreaming, setIsStreaming] = useState(false);

async function handleStartStream(e: React.FormEvent) { e.preventDefault(); if (!prompt.trim() || isStreaming) return;

setIsStreaming(true);
setStreamedReply('');
setTrace(null);

const { textStream, traceStream } = await streamDevPulseAgent({
  sessionId: 'stream-session-01',
  modelId: 'claude-3-5-sonnet-latest',
  messages: [{ role: 'user', content: prompt }],
});

(async () => {
  for await (const latestTrace of readStreamableValue(traceStream)) {
    if (latestTrace) setTrace(latestTrace);
  }
})();

for await (const delta of readStreamableValue(textStream)) {
  if (typeof delta === 'string') {
    setStreamedReply((prev) => prev + delta);
  }
}

setIsStreaming(false);

}

return ( <input value={prompt} onChange={(e) => setPrompt(e.target.value)} className='w-full px-3 py-2 rounded-lg bg-slate-950 border border-slate-700 text-sm' /> {isStreaming ? 'Streaming Tokens...' : 'Stream Response'}

  <div className='p-4 min-h-[140px] rounded-lg bg-slate-950 border border-slate-800 text-sm whitespace-pre-wrap'>
    {streamedReply || 'Stream output will appear here token by token...'}
  </div>

  {trace && (
    <div className='text-xs text-cyan-400 pt-2 border-t border-slate-800 flex justify-between'>
      <span>{trace.thoughtSummary}</span>
      <span>Total Stream Duration: {trace.durationMs}ms</span>
    </div>
  )}
</div>

); } `

A Real Production Lesson on Buffered Streams and Missing .done() Calls

When I was working on real-time file chunking in my LocalDrop Pro project and later setting up my first streaming LLM endpoints in Next.js deployed on Vercel, I ran into two streaming bugs that look completely harmless in local development (npm run dev) but break in production.

First, in one of my early Server Action prototypes using createStreamableValue, I wrapped the for await (const delta of result.textStream) loop in a try/catch block. When I tested an invalid model configuration that threw an API error mid-stream, I logged the error on the server (console.error(err)) inside the catch block—but forgot to call textStream.done() or textStream.error(message). On the React frontend, the for await (const delta of readStreamableValue(textStream)) loop hung indefinitely because the server never closed the stream channel, leaving the UI submit button permanently disabled until the serverless function hit its 60-second hard timeout.

Second, when testing a custom Route Handler behind an Nginx reverse proxy on a Linux VPS, my tokens were not streaming word by word at all; they still arrived in one giant block after five seconds because the proxy was buffering the response chunks! Adding explicit unbuffered stream headers (which result.toDataStreamResponse() handles automatically in the Vercel AI SDK) and ensuring every code path in a Server Action explicitly invokes .done() or .error() solved both issues permanently.

Common Mistakes When Implementing Response Streaming with Vercel AI SDK

Before moving on to wiring the full useChat UI in Post 16, double-check that your code avoids these four frequent streaming anti-patterns:

1. Awaiting the Entire Stream Loop Before Returning from the Server Action

If you write for await (const delta of result.textStream) { textStream.append(delta); } directly in the main body of streamDevPulseAgent before the return { textStream: textStream.value } statement, Next.js will wait for the entire loop to finish before returning the stream handle to the browser! That completely defeats the purpose of streaming. Always run the for await consumption loop inside an unawaited background async IIFE (async () => { ... })() and return textStream.value immediately.

2. Using .update() vs. .append() Inconsistently

createStreamableValue offers both .update(fullString) (which replaces the entire value) and .append(delta) (which sends only the newly generated characters). If you use .append(delta) on the server, your client loop receives incremental deltas; if you accidentally overwrite state with setStreamedReply(delta) instead of setStreamedReply(prev => prev + delta) when reading raw deltas, your UI will flicker showing only one word at a time.

3. Forgetting export const maxDuration = 60; on Streaming Routes

Even though streaming sends the first byte in ~300ms, the server function stays executing for the entire duration of the LLM generation. If your agent writes a detailed 1,000-token response that takes 14 seconds and your route defaults to a 10-second serverless timeout, the stream will cut off mid-sentence. Always export maxDuration = 60 in your Route Handler or page file.

4. Dropping Token Telemetry When Switching to Streaming

Many developers remove token logging when they move from generateText to streamText because streamText returns immediately before tokens are generated. Never fly blind in production—always pass an onFinish: ({ usage }) => { ... } callback to streamText so you can record usage.promptTokens and usage.completionTokens once the stream finishes.

Frequently Asked Questions

Does response streaming make the LLM generate tokens faster, or does it only reduce perceived latency?

Streaming does not change the raw token generation speed (tokens per second) on OpenAI's or Anthropic's GPUs. However, it reduces Time to First Token (TTFT) from 4–8 seconds down to ~300 milliseconds. Because humans read slower than 60 tokens per second, the user starts reading sentence one immediately while the model is still generating sentence four, making the application feel instantaneous.

When should I use createStreamableValue in a Server Action vs. toDataStreamResponse in a Route Handler?

Use toDataStreamResponse() inside an App Router /api/chat/route.ts endpoint when you want plug-and-play compatibility with the Vercel AI SDK's useChat hook (which we will implement in Post 16). Use createStreamableValue() inside a Server Action when you want to multiplex multiple custom TypeScript streams—such as streaming raw text alongside custom AgentStepTrace objects or React Server Components.

Does streamText in the Vercel AI SDK support tool calling and multi-step agent loops?

Yes. streamText natively supports passing Zod-validated tools and a maxSteps parameter, allowing the model to invoke server-side tools mid-stream and immediately continue streaming the synthesized answer—a feature we will put to work as we expand our DevPulse agent in the upcoming posts.