Inside a real-time voice AI pipeline: what orchestration actually does, in sequence
The clinic voice assistant piece is about the business case and what broke in production. This one is the companion for the people who’ll actually be asked to evaluate whether to build something like it: what “orchestration” concretely means in a real-time voice AI system, what a framework like Pipecat is doing for you at each stage, and where the real engineering risk sits once you move from a demo to a phone line.
Orchestration means something different here
“Orchestration” gets used loosely across AI tooling, and it’s worth being precise about which kind you need. A framework like LangChain orchestrates reasoning — chains and agents that call tools, read results, and decide what to do next, one step at a time, with no particular time pressure between steps. A voice pipeline orchestrates frames — audio, text, and control signals streaming continuously and in parallel through multiple services, on a budget measured in a few hundred milliseconds before a pause starts to feel like dead air. These aren’t different flavours of the same problem; a tool built for the first doesn’t have a native concept of a WebSocket media stream, voice activity detection, or barge-in, and retrofitting one onto it is a bigger project than adopting a framework built for the second.
Pipecat is built for the second problem. Concretely, that means: a Pipeline — an ordered chain of processors — that data moves through as discrete frames, with the framework handling buffering, backpressure, interruption, and timing so that a caller experiences something closer to a conversation than a walkie-talkie exchange.
The pipeline, stage by stage
This is the actual sequence, in the order data flows through it on every turn:
1. Transport input. Raw audio arrives over a WebSocket (from Telnyx, in our case) and gets turned into audio frames the rest of the pipeline can work with. This layer also handles the provider-specific handshake — parsing out the caller’s number and a call identifier used later for provider-side actions.
2. Voice activity detection. A lightweight, on-device model (we use Silero) watches the audio stream for the caller starting and stopping speaking. This is table stakes, but it’s also the first place a naive implementation goes wrong: raw silence detection alone triggers on every pause for breath, not just the end of a thought.
3. Streaming speech-to-text. Audio frames become text — interim results as the caller is still talking, a finalized transcript once they stop. We use Deepgram here specifically because it’s built for streaming; a batch transcription model gets you a transcript only after a chunk of audio finishes, which is fine for a voicemail and unusable for a live back-and-forth.
4. Turn detection and context aggregation. This is the stage that actually separates a natural-feeling assistant from a laggy one. VAD alone tells you the caller stopped making sound; it doesn’t tell you they finished their sentence. A smart-turn model looks at the transcribed content itself — is this linguistically complete, or mid-thought — before releasing the utterance downstream as a finalized turn. Get this wrong in either direction and you either cut callers off mid-sentence or leave an awkward pause after they’ve clearly finished.
5. The LLM, including tool calls. The model streams a response against the accumulated conversation context. If it decides to call a tool — check availability, book an appointment — the framework parses that out of the provider’s native function-calling format and hands it to your own code as a structured call, not raw text to parse yourself.
6. Tool execution. This is where your business logic lives, and it’s worth treating as a first-class pipeline stage even though it’s your code, not the framework’s. A well-built handler does three things: speaks a short filler line immediately if the underlying call is going to take real time (a scheduling API, a database, anything over a network), executes the actual request, and feeds the result back into the model’s context so it can formulate a natural reply grounded in real data — rather than a plausible-sounding guess.
7. Streaming text-to-speech. The model’s reply (or a filler line your own code injected directly) becomes audio. We use Cartesia here for latency low enough that the gap between “the model decided what to say” and “the caller starts hearing it” stays imperceptible.
8. Transport output, and recording the turn. Synthesized audio streams back over the same WebSocket in real time, and the assistant’s turn gets appended to the shared conversation context, so the next round trip has full history rather than starting from nothing.
What you get from the framework, not from your own code
The reason to reach for something purpose-built rather than hand-rolling this: a real framework gives you several things that are genuinely hard to get right from scratch, and none of them are optional for a system that has to work on a real phone call:
- Barge-in. If a caller starts talking while the assistant is still speaking, the pipeline needs to stop synthesizing and start listening, immediately, without a stale response finishing and stepping on what the caller just said. This is built in, not something we implemented.
- Normalized function calling. Every LLM provider’s tool-calling format is slightly different. A good framework abstracts that into one schema you define once, with your handler receiving a clean, typed call regardless of which model is underneath — which is also what makes swapping models later a config change, not a rewrite.
- Out-of-band speech. Being able to inject a spoken line outside the model’s own turn — a disclosure greeting, a “let me check that” filler, a farewell before hanging up — independent of what the model is currently generating, is what makes a lot of the production polish in the companion piece possible at all.
- A clean way to end a call from code, and lifecycle hooks for when a call starts, ends, or when an underlying service’s connection fails — the extension point that turned out to matter most in production, covered next.
Where the real engineering risk actually sits
None of the stages above are exotic once you know the shape of the problem. The risk that doesn’t show up in a demo is what happens when a piece of this fails mid-call — and a demo, almost by definition, is a session where nothing fails.
We hit this for real: a text-to-speech provider’s account had a billing problem partway through building this and started rejecting every connection. The caller heard the call connect, then nothing — because the only channel the system had for telling anyone anything was speech-to-text, the model, or text-to-speech, and one of those three was exactly what had broken. Queuing an apology through the same broken component obviously doesn’t work.
The fix leaned on a part of the framework that’s easy to miss until you need it: every STT, LLM, and TTS service exposes a connection-error event independent of the main pipeline flow. Hooking into that let us fall back to the telephony provider’s own built-in announcement capability — a completely separate system with no dependency on whichever of the three AI services was actually down — to apologise and hang up cleanly instead of leaving a caller on a silent line indefinitely.
That fallback had its own bug on the first real test, and it’s worth including because it’s a genuinely instructive one: the telephony provider’s “speak this” action is a queued, asynchronous command — it returns as soon as the instruction is accepted, not once the audio has actually finished playing. The first version fired the follow-up “hang up” command immediately afterward, so the call ended before any sound played. Nothing in that code path was wrong in isolation; it was wrong about timing, which is precisely the category of bug that only surfaces against a real system, never against a mock. The fix — wait an estimated duration proportional to the message length before hanging up — is a deliberate approximation, not a fully correct one (the fully correct version listens for the provider’s own “speech finished” event), and we said so plainly rather than pretending otherwise. It was confirmed working on the very next real call.
The takeaway for anyone evaluating this
The framework earns its keep on the stages that are genuinely hard to build correctly from scratch — turn detection, interruption handling, normalized function calling — and none of that changes the fact that a real phone line will find every gap a mock never exercises: a missing filler during a slow lookup, a model narrating its own tool calls, a vendor outage with no fallback. Pick the orchestration tool that matches the problem you actually have (frames and timing, not chains and reasoning), and then budget real test calls, not just code review, for finding the rest.
Get in touch if you’re building something in this space and want a second opinion on the architecture.