Fish Audio Live 3: The Voice Model That Learns When to Talk
Fish Audio has released Live 3, a full-duplex voice model built for real-time voice agents, and its central argument is one that most voice stacks still get wrong: speed and timing are different problems. An agent can reply in milliseconds and still speak at the wrong moment. The pause after “what time do you open?” usually ends a turn. The pause in the middle of a phone number does not.
Live 3 keeps listening while it speaks. Instead of hand-tuned silence thresholds deciding when an agent takes its turn, the timing is learned inside the speech model itself, from real human conversations. The company's co-founder and CEO Rissa Cao framed the shift plainly: the fix so far has been more rules and more thresholds, and Fish Audio believes a model should learn this from real conversations, the same way language models learn grammar.
The release matters because voice agents are moving from demos to phone lines, and on a phone line the failure modes are conversational, not computational. A caller who reads a credit card number in chunks, says “mm-hmm” while a policy is read aloud, or changes their order halfway through will break most cascaded pipelines in ways no amount of raw latency improvement fixes. This article draws on Fish Audio's Live 3 page and independent analyses from OrcaRouter and explainx.
Why turn-taking is the hard part of voice agents
Most production voice stacks today are cascades. A speech-to-text endpoint detects the end of an utterance, a language model plans the reply, a text-to-speech engine streams the audio, and a separate barge-in detector tries to cancel playback when the caller cuts in. Each hop adds its own failure modes, and the two most common are mirror images of each other.
The first failure is cutting off a caller who paused to think. A cascade sees silence, declares the turn ended, and starts answering a question that was not finished. The second failure is the opposite: treating every sound the caller makes as an interruption. A caller saying “yeah” or “mm-hmm” while hours are being read aloud is signalling attention, not requesting the floor, but a barge-in detector tuned to stop on any speech cannot tell the difference. Both failures feel robotic, and neither is a latency problem.
The pipeline shape also makes mid-utterance corrections dangerous. A caller saying “two pizzas… no, three, and make one margarita” gives a threshold-driven system a complete plausible order after the first clause. The order gets logged wrong, then logged wrong again. Support calls, bookings and food orders are full of exactly this pattern, which is why Fish Audio chose it as the demonstration case.
The three behaviours Live 3 is built around
Fish Audio's launch material walks through three conversational patterns, and each one targets a specific cascade failure.
The first is acknowledging without taking over. When a caller reads a number in chunks, saying “415… 555… 0132”, Live 3 may say “okay, 415” between the chunks and then wait. The acknowledgement does two jobs at once: it confirms the digits were received, and it leaves the turn open so the caller can breathe, check the next digits and continue. A cascade would hear three separate end-of-turn events, or one long hesitation it cannot interpret.
The second is telling a backchannel from an interruption. A caller saying “yeah” while the agent reads a menu lets the agent continue. A caller saying “wait, actually, I just need Thursday” makes it stop and yield the floor. The model is expected to learn that distinction from data rather than from a developer enumerating every sound a person might make while listening.
The third is waiting through corrections. The two-pizzas-to-three example sits at the centre of the launch material: Live 3 holds its response through each revision instead of locking in the first complete parse. The point is not that the model is clever about pizza. It is that floor control, knowing whose turn it is, belongs in the speech model rather than in a state machine above it.
Each behaviour ships with an audio excerpt from test calls in the launch post, which is unusually concrete for a voice release. The excerpts are behaviour evidence, not benchmark evidence: there are no transcription-level scores attached, and the launch does not claim any. But being able to listen to the exact scenario the model claims to handle is more informative, for timing behaviour, than most latency tables ever are.
How the model learns timing instead of rules
The architectural claim behind Live 3 is that conversational timing was learned from real human conversations rather than configured by engineers. Fish Audio has not disclosed the dataset size, the hours of audio, or the evaluation set, so the claim cannot be audited from outside. What can be examined is the design decision it implies.
A rule-based turn-taker is a state machine built on thresholds: so many milliseconds of silence ends a turn, so much energy in the caller's channel means an interruption, a keyword list decides which sounds count as backchannels. Every threshold is a guess about the distribution of real speech, and every guess fails somewhere, usually on the tails: the caller who thinks for four seconds mid-sentence, the person whose “yeah” is drawn out over a second, the noisy line that makes every pause ambiguous.
Learning timing from data replaces those fixed guesses with a distribution the model itself estimates. The bet is that the long tail of real conversations, the hesitations and overlapping speech and drawn-out acknowledgements, teaches the model where the genuine boundaries are. It is the same argument that replaced hand-written grammar rules in language models, applied to a layer of conversation that grammar never covered.
The limit of that argument is also visible. Learned behaviour matches the distribution it was trained on, and Fish Audio has said nothing about whose conversations trained Live 3. Accents, noise floors and compliance scripts outside that distribution are untested territory until you test them, which is why the sensible deployment plan starts with your own worst calls rather than the demo audio.
What full duplex actually means here
Full duplex means the model processes incoming audio while generating speech output, rather than alternating between listening and speaking modes. The listening and speaking waveforms run interwoven. That architecture is what makes a mid-number acknowledgement possible at all: the model hears the next chunk arriving while it is still finishing “okay, 415”, and can decide to keep waiting.
The contrast with the common alternative is worth spelling out. Half-duplex systems must finish speaking before they listen, so every exchange has a hard handover. Cascaded systems bolt a voice activity detector onto a speech synthesiser and try to simulate the same effect with cancellation logic. Live 3 folds listening into the speech model itself, so acknowledgement policy is learned rather than orchestrated. A separate reasoning layer still exists above it, holding the agent's instructions, knowledge base and tools; the timing lives below.
Fish Audio says Live 3 can decide to respond in as little as 186 milliseconds. The qualifier matters: that figure covers short reactions such as “mm-hmm”, “okay” and quick confirmations, not time-to-first-audio for a complete answer, which is prepared in parallel and still waits on the reasoning layer. No latency distribution has been published, and no independent score exists yet. Treat the 186 milliseconds as a vendor floor for small interjections, not a promise about your full stack.
How to evaluate Live 3 on your own calls
Because the published evidence is all vendor-side, the evaluation burden falls on buyers, and the tests worth running map directly onto the behaviours Live 3 claims.
The digit dictation test comes first. Read a credit card number with a deliberate three-second pause in the middle of the sequence, and score whether the agent holds the floor without finishing your number early or logging a partial. Then read it in the noisy conditions of a real mobile line, because timing behaviour that works in clean audio may not survive a bad channel.
The backchannel test comes second. Say “yeah” three times while the agent reads a long policy, and score whether it continues. Then cut in with “actually, cancel that” after it starts confirming, and score whether it stops. Those two cases, played back to back, measure exactly the distinction the model claims to have learned, and a system that fails either one has not solved turn-taking regardless of its latency.
The correction test comes third. Change the quantity twice in one order, then change a detail again after confirmation begins. A pipeline that fires on the first complete parse will double-log the order, which is the most expensive failure mode in the set because it looks like success in the transcript and costs money downstream.
Score false stops and talk-overs separately throughout. A model that is fast but rude fails customer experience reviews just as surely as one that is slow, and the two numbers answer different questions. The metric that matters most is none of these, though; it is how often the caller has to start over. That single number captures every timing failure at once, and it is the first thing to instrument after deploying Live 3 or any duplex competitor.
Deployment, pricing and the platform around it
Live 3 is the voice layer of a broader product. The Fish Agent Platform, in public beta, exposes a REST surface under /v1/agent/*, with browser clients through the agent-client and agent-react packages. Agents can answer phone numbers, connect over WebRTC, or sit in a website widget, and sessions authenticate with session tokens rather than long-lived keys in the browser.
The platform ships the production tooling that voice agents need: live monitoring with recordings and turn-by-turn transcripts, latency breakdowns across end-of-utterance, language model and speech, automatic summaries and data extraction, and simulation tests judged by a model before each release. A custom LLM hook lets the reasoning layer point at any OpenAI-compatible endpoint, so the model doing the thinking does not have to be Fish Audio's.
Pricing on the Live 3 page lists 0.035 dollars per minute, half the 0.07-dollar list price, until 31 December 2026. The list rate covers speech recognition, text-to-speech, basic language models, knowledge retrieval and tool calls. Phone calls add 0.01 dollars per minute, a dedicated number costs 1.20 dollars per month, and new accounts get a 5-dollar trial balance. Enterprise deployment is available on-premises with zero data retention for security reviews. The voice library behind it holds more than 2,000,000 voices across 30-plus languages, including clones.
The honest scoreboard: what is claimed and what is measured
Every performance figure attached to Live 3 is Fish Audio's own, and the independent picture is thin one day after launch. OrcaRouter's analysis of the release makes the gap explicit: no benchmark evaluation has been published, no latency distribution, and no row for Live 3 on Artificial Analysis's duplex leaderboards as of 11 November. Full-Duplex-Bench and agentic voice tests would be the direct comparisons, and Live 3 has no score on either yet.
The competitive table from third-party coverage makes the same point from the other side. ElevenLabs' Realtime v1 Max publishes an average time to first audio of 0.57 seconds and a 94.1 percent Full-Duplex-Bench score, figures sourced through Artificial Analysis. OpenAI's GPT-Live-1 self-reports a 30-point Full-Duplex Bench gain over its predecessor and a top agentic score with a backend reasoning model. Gemini 3.8 Live claims 97 languages and runs tools in the background. Live 3's row is built from behaviour demonstrations and a 186-millisecond floor, with published scores marked not available.
Two launch limitations are equally material. Live 3 is English-only at launch, which rules out most international support lines on day one, and no timeline for other languages has been announced. The training data size and evaluation set are not disclosed, so the claim that timing was learned from real conversations cannot be checked from the outside. Neither omission is unusual for a launch, but both belong in any deployment decision.
Where this fits in the voice stack
The voice interface stack was already crowded before Live 3. Cascaded pipelines still dominate production, with speech recognition, reasoning and synthesis bought separately and glued with orchestration code. Full-duplex models are the challenger architecture, and they are arriving from every direction: OpenAI and Google from the platform side, ElevenLabs from the creative-audio side, and research systems like NVIDIA's PersonaPlex exploring the same territory.
Fish Audio's bet is narrower and more specific: that conversational timing belongs in the audio model rather than in orchestration code, and that a model trained on real conversations will out-time a rule system built by hand. The company's product line supports the reading. Drama 3 handles directed text-to-speech with open weights, S2.1 Pro handles streaming narration, and Live 3 takes the real-time conversation lane where timing is the product. For builders, the practical test is not which architecture is better in the abstract but which one fails less on their calls: false stops and talk-overs should be scored separately, on your own scripts, before and after.
The soundness of that bet will be settled by numbers that do not exist yet.
For context on how quickly the agent layer around voice is hardening, our coverage of ChatGPT Enterprise reaching AWS Marketplace and the Gemini Decisions API shows the same pattern one layer up: the interfaces agents use are being rebuilt around what real work actually looks like. Voice is simply the next surface, and it is the one where hesitation, overlap and second thoughts have always lived. Our earlier piece on ZONOS3-Mini running speech in a browser covers the synthesis side of the same stack.
Questions readers are asking
What is Fish Audio Live 3?
A full-duplex voice model for real-time voice agents, released 10 November 2026. It listens while it speaks, learns turn-taking from real conversations instead of silence thresholds, and reacts to short cues in as little as 186 milliseconds, a vendor-reported figure for brief reactions only.
How much does Live 3 cost?
The Live 3 page lists 0.035 dollars per minute until 31 December 2026, then 0.07 dollars, covering speech recognition, basic language models, text-to-speech, retrieval and tool calls. Phone calls add 0.01 dollars per minute and a dedicated number costs 1.20 dollars per month. Enterprise on-premises terms are separate.
Can Live 3 handle interruptions properly?
That is its central design claim. Fish Audio demonstrates it distinguishing backchannels such as “yeah” and “mm-hmm” from real interruptions such as “wait, I just need Thursday”, acknowledging phone numbers mid-dictation, and waiting through order corrections. All demonstrations are Fish Audio's own; no independent turn-taking score exists yet.
What are Live 3's main limitations?
English-only at launch, with no published benchmark scores, no latency distribution beyond the 186-millisecond floor for short reactions, undisclosed training data details, and full answers that still depend on the separate reasoning layer above the speech model.
The bottom line
Live 3 attacks the right problem. The awkward pauses, false stops and broken phone numbers that plague voice agents are timing failures, not latency failures, and moving turn-taking into a model trained on real conversations is a credible answer to them. The behaviour demonstrations are concrete, the platform around the model is production-shaped, and the pricing undercuts the list rates of a market that is still figuring out what duplex voice should cost.
The evidence is also, entirely, the vendor's. One day after launch there is no independent benchmark, no latency distribution, no languages beyond English, and no way to check the 186-millisecond claim against a standard test. Teams evaluating it should run their own worst calls against it: the digit dictation with a pause, the three “yeahs” during a policy read, the order that changes twice. If Live 3 clears those where the current cascade fails, the timing bet will have paid off. The scoreboard, when it arrives, will tell the rest.