Inside a real-time voice AI pipeline: what orchestration actually does, in sequence

The clinic voice assistant piece is about the business case and what broke in production. This one is the companion for the people who’ll actually be asked to evaluate whether to build something like it: what “orchestration” concretely means in a real-time voice AI system, what a framework like Pipecat is doing for you at each stage, and where the real engineering risk sits once you move from a demo to a phone line.

Orchestration means something different here

“Orchestration” gets used loosely across AI tooling, and it’s worth being precise about which kind you need. A framework like LangChain orchestrates reasoning — chains and agents that call tools, read results, and decide what to do next, one step at a time, with no particular time pressure between steps. A voice pipeline orchestrates frames — audio, text, and control signals streaming continuously and in parallel through multiple services, on a budget measured in a few hundred milliseconds before a pause starts to feel like dead air. These aren’t different flavours of the same problem; a tool built for the first doesn’t have a native concept of a WebSocket media stream, voice activity detection, or barge-in, and retrofitting one onto it is a bigger project than adopting a framework built for the second.

Pipecat is built for the second problem. Concretely, that means: a Pipeline — an ordered chain of processors — that data moves through as discrete frames, with the framework handling buffering, backpressure, interruption, and timing so that a caller experiences something closer to a conversation than a walkie-talkie exchange.

The pipeline, stage by stage

This is the actual sequence, in the order data flows through it on every turn:

1. Transport input. Raw audio arrives from the phone provider and gets turned into audio frames the rest of the pipeline can work with.

2. Voice activity detection. A lightweight, on-device model watches the audio stream for the caller starting and stopping speaking. This is table stakes, but it’s also the first place a naive implementation goes wrong: raw silence detection alone triggers on every pause for breath, not just the end of a thought.

3. Streaming speech-to-text. Audio frames become text — interim results as the caller is still talking, a finalized transcript once they stop. A streaming model matters here; a batch transcription model gets you a transcript only after a chunk of audio finishes, which is fine for a voicemail and unusable for a live back-and-forth.

4. Turn detection and context aggregation. This is the stage that actually separates a natural-feeling assistant from a laggy one. VAD alone tells you the caller stopped making sound; it doesn’t tell you they finished their sentence. Turn detection looks at the transcribed content itself — is this linguistically complete, or mid-thought — before releasing the utterance downstream as a finalized turn. Get this wrong in either direction and you either cut callers off mid-sentence or leave an awkward pause after they’ve clearly finished.

5. The LLM, including tool calls. The model streams a response against the accumulated conversation context. If it decides to call a tool — check availability, book an appointment — the framework parses that out of the provider’s native function-calling format and hands it to your own code as a structured call, not raw text to parse yourself.

6. Tool execution. This is where your business logic lives, and it’s worth treating as a first-class pipeline stage even though it’s your code, not the framework’s. A well-built handler does three things: speaks a short filler line immediately if the underlying call is going to take real time (a scheduling API, a database, anything over a network), executes the actual request, and feeds the result back into the model’s context so it can formulate a natural reply grounded in real data — rather than a plausible-sounding guess.

7. Streaming text-to-speech. The model’s reply (or a filler line your own code injected directly) becomes audio. The latency has to be low enough that the gap between “the model decided what to say” and “the caller starts hearing it” stays imperceptible.

8. Transport output, and recording the turn. Synthesized audio streams back over the same WebSocket in real time, and the assistant’s turn gets appended to the shared conversation context, so the next round trip has full history rather than starting from nothing.

What you get from the framework, not from your own code

The reason to reach for something purpose-built rather than hand-rolling this: a real framework gives you several things that are genuinely hard to get right from scratch, and none of them are optional for a system that has to work on a real phone call:

• Barge-in. If a caller starts talking while the assistant is still speaking, the pipeline needs to stop synthesizing and start listening, immediately, without a stale response finishing and stepping on what the caller just said. This is built in, not something we implemented.
• Normalized function calling. Every LLM provider’s tool-calling format is slightly different. A good framework abstracts that into one schema you define once, with your handler receiving a clean, typed call regardless of which model is underneath — which is also what makes swapping models later a config change, not a rewrite.
• Out-of-band speech. Being able to inject a spoken line outside the model’s own turn — a disclosure greeting, a “let me check that” filler, a farewell before hanging up — independent of what the model is currently generating, is what makes a lot of the production polish in the companion piece possible at all.
• A clean way to end a call from code, and lifecycle hooks for when a call starts, ends, or when an underlying service’s connection fails — the extension point that turned out to matter most in production, covered next.

Where the real engineering risk actually sits

None of the stages above are exotic once you know the shape of the problem. The risk that doesn’t show up in a demo is what happens when a piece of this fails mid-call — and a demo, almost by definition, is a session where nothing fails.

We hit this for real: a speech provider’s account had a billing problem partway through building this and started rejecting every connection. The caller heard the call connect, then nothing, because the only way the system could speak was through the part that had just broken. Queuing an apology through the same broken component obviously doesn’t work.

Handling that well means planning for it in the design: knowing when any one of the AI services has failed, and having a way to tell the caller that doesn’t depend on any of them. The first version of our fallback had a timing bug that no mock would have caught. It only showed up on a real call, and the next real call confirmed the fix.

The takeaway for anyone evaluating this

The framework earns its keep on the stages that are genuinely hard to build correctly from scratch — turn detection, interruption handling, normalized function calling — and none of that changes the fact that a real phone line will find every gap a mock never exercises: a missing filler during a slow lookup, a model narrating its own tool calls, a vendor outage with no fallback. Pick the orchestration tool that matches the problem you actually have (frames and timing, not chains and reasoning), and then budget real test calls, not just code review, for finding the rest.

Get in touch if you’re building something in this space and want a second opinion on the architecture.