Soniox STT v6: Voice Agents Finally Hear Both Sides

Posted on by
Contents show
Soniox STT v6

A voice agent asks its caller whether they would prefer a window or an aisle seat. The caller answers “aisle”. Simple, except the speech recognition model transcribing the call never heard the question. It receives one side of the conversation, the microphone audio, so “aisle” arrives as two syllables with no anchor, and the transcription engine does what it can with them. That one-sided setup is the problem Soniox STT v6 was designed to eliminate. Every developer who has shipped a phone agent knows what happens next: the clarification loop, the repeated question, the caller hanging up.

Soniox STT v6, released this week by the speech AI company of the same name, attacks that failure at its root. It is being billed as the first speech recognition model that knows what the voice agent is saying. Two connected features do the work: agent text input streams the agent's outgoing words into the model as context, and agent voice echo cancellation uses that context to keep the agent's own playback out of the transcript while continuing to listen to the user.

This article explains how Soniox STT v6 works, why the engineering details matter more than the feature names, what the release improves beyond voice agents, and where the honest limits sit, so a team can decide whether it belongs in their stack.

Speech recognition has a half-conversation problem

The architecture of most voice agents explains the gap, and explains why Soniox STT v6 had to be designed rather than patched. A pipeline takes the caller's audio, transcribes it, passes the text to a language model, converts the reply to speech, and plays it. The transcription model sees only the caller's side. The question that shaped the answer exists in the pipeline, but in a different component, and never reaches the model that most needs it.

The cost of that blindness is measurable in transcription errors, and it is the core problem Soniox STT v6 was built to fix. The documentation examples make it concrete: asked to choose between “Maple Cafe and River Roast”, a caller says “River Roast, please”, which a model without the question can render as “River Road”. Asked what help they need for a stranded car, the answer “I need a tow” becomes “a tour”. Asked to choose between tom yum and coconut soup, the dish name alone is a guessing game. Short answers suffer most, and short answers are the currency of voice interfaces.

Human listeners never have this problem, because they heard the question. Soniox STT v6 gives the model the same advantage. The information needed to disambiguate the reply was spoken seconds earlier, by the agent itself. The fix, in hindsight, is obvious: give the recognition model the same transcript the caller is hearing.

How agent text input works

The mechanism behind agent text input is a stream, not a batch. While the agent speaks, the developer sends its words to Soniox over the same WebSocket that carries the user's audio, in small pieces, each just before the corresponding audio reaches the caller. The model aligns that text with the audio stream and gains both sides of the conversation.

The timing requirements of Soniox STT v6 reveal what is really being built. Text must arrive roughly half a second to one second before the user hears it, which means following the playback clock rather than the language model or the text-to-speech engine, both of which generate faster than real time. Pieces can be as small as a single character, and splitting mid-word is fine. Each agent turn ends with a segment marker, and if the user interrupts, the developer sends what was already played, marks the turn ended, and drops the rest, because the user never heard it.

That last rule shows the design thinking behind Soniox STT v6. Text sent ahead of an interruption stays valid, since the agent did say those words. Text never played is discarded, since speaking it would fabricate context the caller never received. The system is modelling a real conversation, not annotating a script.

Billing follows the same logic: agent text is billed per token at the same rate as other input text, and only text that reaches the model is billed.

Why echo cancellation needed a model, not a filter, in Soniox STT v6

The second half of Soniox STT v6 addresses a failure mode every voice agent team knows: the agent interrupting itself.

The failure loop has a well-known anatomy. The agent starts speaking. The speaker output reaches the microphone. The transcriber, unaware, converts agent speech into user text. The language model receives a user turn it never got, answers it, and the agent now talks over itself. On speakerphone devices the loop can sustain for several turns before a human gives up. Every voice team has a story about this, usually from a demo that worked on headphones and failed on a conference room phone.

Acoustic echo cancellation has existed for decades, and every voice product runs some version of it. But it is never perfect, and the documentation is candid that parts of the agent's voice often leak into the microphone, especially at the moment the agent starts talking. A transcription model receiving that leakage has no way to know it is not the user, so it transcribes it, the agent hears what sounds like a user speaking, and the conversation breaks in a loop.

Soniox STT v6 handles this at the recognition layer, where the conversation lives. Because the model knows, from the streamed text, exactly what the agent is saying and when, it can classify leftover echo as agent playback and leave it out of the transcript, while still transcribing a user who talks over the agent. The Soniox STT v6 release materials are explicit that this does not replace acoustic echo cancellation; it catches what the acoustic filter leaves behind.

The distinction matters for architecture, and it is where Soniox STT v6 departs from the filter-and-pray approach. Acoustic cancellation operates on waveforms and knows nothing about conversation state. Recognition-layer filtering knows the full script, the timing, and the interruptions, so it can do something a filter cannot: keep listening to the user during agent playback, which is precisely the moment barge-in happens. On speakerphones, laptops and in-car systems, where playback reliably reaches the microphone, that is the difference between an agent that can be interrupted naturally and one that talks over its user.

Building it into a real pipeline

For a team wiring Soniox STT v6 into an existing agent, the work concentrates in one place: the playback clock. The feature flag is the easy part. Everything else is a flag.

The start request enables agent text input on the real-time model. From there, the agent's outgoing reply, the text that text-to-speech will speak, becomes a synchronized stream. The recommended pattern is to gate each piece of text on its spoken start time, taken from TTS timestamps, and send it when that start time falls within the next 700 milliseconds or so of playback. A timer on the order of 50 milliseconds, or simply sending after each chunk of user audio, keeps the pace steady. Many small messages are fine; batching defeats the alignment.

The interruption path is where implementations go wrong. When the user starts talking and playback stops, the developer must flush any text that was already played but not yet sent, mark the segment ended, and suppress the remainder of the reply. Skipping that order leaves the model holding context for words the caller never heard, which poisons the very disambiguation the feature exists to provide.

Content discipline matters as much as timing. Only what the agent speaks aloud belongs in the stream: SSML, markdown and expression tags must be stripped, because the model expects words in time with audio. Instructions, notes and background knowledge belong in the context channel, which has always existed and remains the right home for stable facts like product names. The rule of thumb the documentation implies is clean: context knows your business, agent text knows the conversation.

One more integration detail pays for itself. Because the model places streamed text at the point in the audio where it arrived, a team whose server never receives playback reports can estimate the clock from the time since it started sending audio, with the explicit constraint never to count audio that has not actually been sent. That single rule, counting only what the user has heard, is the contract.

What the release improves beyond agents

The agent features get the headlines, but Soniox STT v6 is a full generation bump across the model's other workloads.

Voice typing gets the largest share of Soniox STT v6's attention. The Soniox STT v6 team reports heavy investment in real-world dictation data, with the accuracy gains concentrated exactly where dictation fails in practice: numbers, currencies, units, dates, alphanumeric identifiers, names and specialized terminology. A single wrong digit changes an amount, and the release treats that class of error as the benchmark that matters. Context customization, supplying product names or project vocabulary, extends the same idea.

Meeting products get their own Soniox STT v6 improvements to transcription and speaker diarization, trained on data that captures rapid speaker changes, interruptions, overlapping voices and similar-sounding speakers, a data investment specific to Soniox STT v6. Both real-time and asynchronous models carry the improvements, which matters because notetaking products typically run the async model on recordings while live assistants use streaming. The failure being fixed is familiar to anyone who has read an auto-generated meeting summary: a decision attributed to the wrong person, a number off by a digit, a name spelled three ways in one page. Speaker diarization errors are more corrosive than raw word errors, because a transcript that misattributes a statement does not just lose information, it assigns it to someone who never said it. Soniox STT v6 treats who-said-what as a first-class output rather than a post-processing step, and for conversation intelligence products that is the difference between a usable record and an embarrassment.

Translation gets a subtle but interesting Soniox STT v6 upgrade: the model now dynamically adjusts how much speech it gathers before translating. Languages such as German and Korean place meaning-carrying words late in the sentence, and a translator that emits too early produces confident nonsense. Variable look-ahead is the correct engineering answer, and the release applies it while keeping output responsive. The general principle generalizes: a streaming system should decide how much future it needs per language rather than applying one global delay, and the fact that this is now handled inside the recognition model, instead of in every customer's buffering code, is the same consolidation story the agent features tell. Decisions that used to be the integrator's problem are becoming the model's job.

All of it runs through one unified model across 60-plus languages, with agent text input and echo cancellation available in every supported language, and mid-sentence language switching preserved. The unified architecture deserves a moment, because it is an operational decision as much as a technical one. Products that serve a global audience have historically routed different languages to different models, which multiplies testing matrices and produces the experience gap that international users notice immediately. A single model that carries the full feature set everywhere, and follows a caller switching language inside one sentence, removes a whole category of routing code and the latency that came with it. For call centers in multilingual markets, that is often the difference between one deployment and five.

The economics and the practical limits

Pricing is unchanged by Soniox STT v6: $0.10 per hour of audio asynchronous, $0.12 per hour real-time, with agent text billed per token like other input text. For a voice agent running thousands of calls, the agent text stream adds a modest token cost on top of audio, and the developer controls it by sending only what the agent speaks aloud, stripping SSML, markdown and expression tags before streaming.

The limits of Soniox STT v6 deserve equal attention. Agent text helps replies that depend on what the agent said; a caller's opening words, before the agent has spoken, get no benefit. The feature requires careful playback-clock discipline, and a team that streams text ahead of actual playback will corrupt the alignment the whole system depends on. Echo filtering complements rather than replaces acoustic echo cancellation, so the DSP layer stays. And the accuracy claims, as with any vendor release, come from the vendor; the honest test remains your own audio, run side by side against your current provider.

A sizing example makes the token cost concrete. A Soniox STT v6 cost estimate makes the token math concrete. A support call where the agent speaks for six of the twelve minutes produces, at typical speech rates of roughly 150 words per minute, about 1,350 agent words, or near 1,800 tokens, streamed as context. At text-token pricing that is a rounding error next to the $0.12 per hour of call audio, and it buys the removal of an entire class of clarification loops. The trade only turns unfavorable if a team streams content the agent never speaks, which is why the strip-your-markup rule is an economics rule as much as a correctness rule.

There is also a deployment note hiding in the Soniox documentation: model names change with versions, older names are aliased and then retired, so production teams should pin versions deliberately rather than by accident.

What this says about where voice AI is going

The interesting signal in Soniox STT v6 is not the feature list. It is the direction of travel: the boundary between the speech layer and the conversation layer is dissolving.

For a decade, speech recognition was a stateless utility, transcribing audio with no idea what conversation it belonged to. Context customization started the shift by letting developers inject vocabulary. Agent text input completes it by giving the model the live dialogue itself. The transcription model is becoming conversation-aware, and once it is, the natural next steps are predictable: tighter coupling with turn-taking, barge-in handling that understands semantics rather than merely energy, and recognition systems that participate in dialogue state rather than serve it.

Teams building voice agents should read Soniox STT v6 as a benchmark for what to demand from their speech layer. If your recognizer cannot be told what your agent is saying the way Soniox STT v6 allows, you are accumulating clarification loops that a conversation-aware alternative would remove, and the economics of agent latency make every unnecessary round trip expensive. The same consolidation logic appears in the rest of the stack, where unified agent workspaces reduce the seams between tools, and where persistent compute for agents removes the reset cost between sessions. Voice is simply the last seam to close. What remains genuinely open is how far conversation-awareness should go. A recognition model that knows the agent's script is uncontroversial. A recognition model that adapts to conversation outcomes starts making decisions about what was meant, and at that point the transcription layer is no longer a utility but a participant, with everything that implies for debugging, evaluation and trust. Soniox STT v6 stops at the defensible line: it uses the agent's words as evidence for understanding the user, exposes none of its own text in the transcript, and leaves the conversation logic to the agent. That division of responsibility is the right one, and it is worth watching whether the next generation keeps it.

Sources
Subscription Form