Kyutai Taught A Speech Model To Do Math Out Loud, And The Method Matters More Than The Score

Speech-to-speech AI has a persistent weakness: models that converse naturally fall apart when the task requires actual reasoning, because a speech model must emit audio at regular intervals to stay interactive, which leaves little room for the hidden reasoning steps that make text models good at math. The result is a voice assistant that sounds natural and answers confidently and wrongly, and until now the standard fix was to route speech through text, paying latency and losing the paralinguistic qualities that made voice worth using. Kyutai's Voice of Reason release attacks exactly this gap. Two open-weight speech-to-speech models, both fine-tuned from GLM-4-Voice-9B, solve math problems out loud, with no transcription step and no separate text LLM in the loop. On spoken GSM8K, accuracy climbs from 27.3 percent for the base model to 77.1 percent for the released checkpoint.

The score is the release headline: 27.3 to 77.1 percent on spoken GSM8K, a math benchmark of grade-school word problems, is a near-tripling of reasoning accuracy in a speech-native model. The method is the story, because two of the training decisions Kyutai made are the difference between the result and a collapse to near-random performance, and both generalize beyond speech to any reinforcement learning run that samples from a stochastic policy. This article also does something the headline numbers cannot: it carries the trade-offs the release itself discloses, including a measured regression in general knowledge, because a result without its costs is marketing rather than research. This article covers the release from the research team's own materials and the paper summary: why speech models lag on math, how the training works, the two released checkpoints, and the trade-offs the results table honestly shows.

Why speech models lag on math

The dominant architecture for capable voice assistants is cascaded: speech-to-text, a text LLM, then text-to-speech. It works because the reasoning happens in text, where models are strongest. But each stage adds latency, and the pipeline loses paralinguistic cues, tone, hesitation, emphasis, that make speech feel like conversation. Cascaded systems still hold the top of the leaderboard, and the release is honest about that; the claim is not that speech-native beats cascaded on raw accuracy, but that the gap has closed from catastrophic to workable while keeping everything that makes voice valuable.

Speech-native models generate audio directly, which preserves those cues and keeps latency low, but they face a structural constraint: to stay interactive they must emit audio at regular intervals, which limits how many hidden reasoning tokens they can afford. Base GLM-4-Voice scored 27.3 percent on spoken GSM8K. An earlier method called STITCH raised that to 58.7 percent by inserting reasoning chunks between spoken segments. Kyutai's work, which the team describes as the first application of reinforcement learning to math reasoning in speech-native models, pushes past both.

How the training works

GLM-4-Voice interleaves its output: 13 text tokens followed by 26 audio tokens, repeating. That interleaving is what makes the model speech-native, and it is also what makes training it to reason technically interesting, because the reward has to reach the reasoning through a mixed text-and-audio stream.

Stage one, supervised fine-tuning: 150,616 problems from Orca-Math, each rewritten for speech by Qwen3-235B, then voiced in many voices by Kyutai's own DSM TTS system. SFT alone lifts speech reasoning accuracy from 27.3 to 61.7 percent, which is a large gain on its own and still nine points short of the released Stitch model.

Stage two, reinforcement learning: for each spoken question, the model samples four replies at temperature 0.9, and a judge model, Qwen3-235B-A22B-2507, scores the decoded text with a binary reward, never seeing the reference answer. Rewards are centered within each group, forming a group-relative REINFORCE objective, related to GRPO but dropping the PPO clipping and KL regularization. Training ran on 16 H100 GPUs for 1,500 updates. On 100 hand-checked cases, the judge agreed with human assessments 88 percent of the time.

Two design choices carried the result, and both are documented with the failure mode that proves their necessity. Temperature correction: the training loss divides logits by the sampling temperature before the log-softmax, correcting for the mismatch between how the model samples and how the loss reads probabilities. Without it, GSM8K accuracy collapsed from 65.5 to 12.3 percent. Audio-token merging: at each audio position, all audio-vocabulary probabilities are summed into one abstract token, so the loss asks only whether audio came next, not which specific audio token. The paper proves this estimator is unbiased and lower-variance under a value-invariance assumption. In plain terms: the model is graded on deciding to speak, and on the words it speaks, while the specific sound of the voice is left free.

The two checkpoints, and what the numbers say

Both checkpoints are 9B speech models, open on Hugging Face, running on a single H100 in BF16.

glm-4-voice-of-reason-9b answers directly, with no extra reasoning tokens; any working is spoken aloud. glm-4-voice-of-reason-stitch-9b writes silent 100-token reasoning chunks between spoken blocks, and because later chunks generate while earlier speech plays, the thinking adds no extra latency.

The results table places the work honestly, mixing matched 9B speech models with larger text-output systems and labelling the difference. PersonaPlex, a full-duplex model, scores 3.2 percent. Base GLM-4-Voice scores 27.3. STITCH reaches 58.7. Voice of Reason's direct model reaches 65.5 under the paper's top-k 50 decoding protocol, and 70.3 without top-k, which is the released checkpoint. The Stitch variant reaches 74.8 under the paper protocol and 77.1 released. Above them sit larger or text-output systems: Qwen2.5-Omni at 84.7 with text output, Qwen3-Omni at 94.6, and a cascaded ASR-LLM-TTS pipeline at 95.7 with a 31B LLM. The card is explicit that those are larger top lines, not matched comparisons, which is the correct way to present them.

Four secondary findings round out the picture, and each one answers an objection a skeptical reader would raise. The gains survive real speech: transcribed with a different ASR system, Qwen3-ASR-1.7B, the Stitch model scores 72.0 percent. Naturalness holds: UTMOSv2 speech-quality scores barely moved after RL. The gains do not come from talking longer: RL actually shortened the direct model's average reply from 41.9 to 36.4 seconds, and Stitch's reasoning tokens grew only from 167 to 176. And RL helps most when data is scarce: with 10 percent of the SFT data, a longer RL run reached 58.5 percent where SFT alone scored 43.9.

The trade-off the release discloses

General knowledge dipped. Spoken TriviaQA fell from 40.6 to 34.0 percent for the direct model, and the authors attribute the drop mainly to the full-data SFT stage rather than the RL itself. That is a real cost: a model tuned this hard toward math reasoning gives up breadth, and the disclosure is what lets a deployer decide whether the trade fits their workload. A voice assistant for math tutoring or technical support gets a large reasoning gain; a general-purpose companion gets a narrower model.

The contamination check is also worth noting for anyone reading the headline number: the team dropped four benchmark sets, AddSub, MultiArith, SingleEQ and SVAMP, from evaluation after finding that 54 percent of 678 checked questions overlapped with Orca-Math training data at the paraphrase level. Removing contaminated tests before reporting is the difference between a benchmark score and a memorization claim, and the disclosure tells you which one this is. Reported scores are on the remaining, decontaminated evaluation, which is the more defensible number and the reason to trust the gain as reasoning rather than memorization.

Why the method generalizes beyond speech

The two training findings travel well beyond this release. Temperature correction in the RL loss, dividing logits by the sampling temperature before the log-softmax, applies to any group-relative RL setup that samples at non-greedy temperatures, and the collapse without it suggests how many RL runs have been quietly damaged by the mismatch. Audio-token merging, reducing a large action vocabulary to an abstract has-action token in the loss while proving the estimator unbiased, is a template for training models whose outputs mix reasoning tokens with generative tokens of any kind. Both ideas matter for the agent-training infrastructure this site has covered, from EnvHarness to the harness economics work: as agents emit more than text, the RL machinery that reaches through mixed output streams is the binding constraint, and Kyutai has published two working solutions. The safety angle travels too: a speech-native reasoning model is also a model whose chain of thought can be spoken aloud or kept silent by design, which intersects the monitoring questions raised in the Claude security coverage.

What this means for voice agents in production

For teams building voice products, this speech release changes the calculation for one class of workload: conversations where the assistant must reason, not just retrieve. A math tutor, a technical support line that walks through diagnostics, a financial advisory bot that walks through arithmetic on air, these are the workloads where the cascaded pipeline's latency tax is most painful and where speech-native reasoning was previously not good enough to use. A 77-percent model that reasons while it speaks, on one GPU, is the first open-weight option that makes those products buildable without a transcription detour.

The deployment profile matters as much as the score. One H100 serves the BF16 checkpoint, the license is inherited from GLM-4-Voice, and the dependencies are two repositories. For teams that have been running cascaded voice stacks, the migration question is whether dropping the transcription round-trip recovers enough latency and naturalness to justify re-tuning prompts around a model that reasons in its own voice, and the honest answer is that only A-B testing against your callers answers it. For a product team, that is a moderate self-hosting lift, not a research project, and the two-checkpoint design gives a product choice: the direct model for flows where spoken working is acceptable or desirable, the Stitch model for flows where the user should hear the answer, not the derivation. Both share the same speech quality after reinforcement learning, which the UTMOSv2 scores confirm, so the choice is about product behavior rather than model capability. The disclosure that the reasoning does not add latency is the detail that makes the Stitch variant production-viable, because a voice product that pauses to think loses the caller.

The open question for production is the breadth trade. A voice agent tuned to GSM8K-class math will handle arithmetic, unit reasoning and multi-step word problems well, and will be measurably weaker at general trivia than its base model, because reinforcement learning against a narrow reward optimizes the model for that reward and the SFT data centered the whole fine-tune on math. The right product shape is probably a router: a general voice model for most of the conversation, with math-reasoning requests escalated to a Voice of Reason checkpoint, which is the same routing discipline this site applies to text models and which the voice stack now supports too.

Where the numbers stop

• The headline figures come from Kyutai's own paper and release. The judge model agreement rate of 88 percent is from 100 hand-checked cases, a small sample, and the judge never saw reference answers, so scoring quality depends on the judge's own math ability.
• The evaluation judge differed from the training judge (GPT-4o for evaluation, Qwen3-235B-A22B-2507 for training), and evaluation audio came from a different TTS system than training audio. Those are good design choices for generality, but they mean the released numbers embed specific tooling choices that a reimplementation would need to match or replace.
• The comparison table mixes matched 9B models with larger text-output and cascaded systems. The card labels them as unmatched top lines, and this article preserves that distinction; no claim here suggests Voice of Reason beats a 30B text model at math.
• The TriviaQA regression is attributed by the authors to SFT rather than RL. That attribution is their analysis, not an ablation shown in the summary materials, and teams deploying for general-purpose use should treat the regression as a property of the released checkpoints regardless of cause.
• Both checkpoints inherit the GLM-4-Voice license, require the GLM-4-Voice repository for the speech tokenizer and decoder, and were not hosted by any Hugging Face inference provider at release, so self-hosting on an H100 is the deployment path.

FAQ

What is Kyutai's Voice of Reason?

It is a pair of open-weight speech-to-speech models released by the Kyutai research lab that solve math problems out loud. Both are 9B models fine-tuned from GLM-4-Voice using supervised fine-tuning on 150,616 speech-adapted math problems followed by group-relative reinforcement learning with a binary correctness judge. There is no transcription step and no separate text LLM in the loop, and the released Stitch checkpoint reaches 77.1 percent on spoken GSM8K, up from 27.3 for the base model.

How does the Stitch version think without adding latency?

The Stitch model writes silent 100-token reasoning chunks between spoken blocks of the interleaved output. Because the model's later reasoning chunks are generated while the earlier speech is still playing to the listener, the thinking overlaps with speaking rather than delaying it. In the STITCH-R layout used for the released checkpoint, the first reasoning chunk precedes the first spoken block, so the model thinks briefly before speaking and continues reasoning behind the audio.

Can I run Voice of Reason myself?

Yes, by self-hosting. Both BF16 checkpoints run on a single H100 GPU, the weights are open on Hugging Face under the GLM-4-Voice license, and you need the GLM-4-Voice repository for the speech tokenizer and decoder. No hosted inference provider carried the models at release, so there is no one-click API path yet; the deployment is a GPU and the two repositories.

The short version

Kyutai's Voice of Reason is the first reinforcement learning result for math reasoning in speech-native models: two open 9B speech-to-speech checkpoints that lift spoken GSM8K from 27.3 to 77.1 percent, with silent reasoning chunks that overlap speech rather than delaying it. The method carries the story: temperature correction in the RL loss prevented a collapse to 12.3 percent, and audio-token merging lets the reward reach reasoning through a mixed text-and-audio stream with a proven-unbiased estimator. The trade is breadth, with spoken TriviaQA down about six points, disclosed rather than hidden. For voice agents that need to reason, this is the current open-weight state of the art, and for anyone training models on mixed output streams, the two training techniques are the reusable part.

Sources

• Kyutai, Voice of Reason: Reinforcement Learning for Spoken Math, the research announcement and paper from the Kyutai team: https://kyutai.org/papers
• Hugging Face, the released glm-4-voice-of-reason-9b and glm-4-voice-of-reason-stitch-9b checkpoints under the Kyutai organization: https://huggingface.co/kyutai