Together Compiler: When Inference Tuning Never Stops

Posted on by
Contents show
Together Compiler

Inference engines are tuned the way software used to be shipped: carefully, once, and then left alone until something breaks. Together Compiler is an argument that this model of tuning is over. An engineer spends days selecting parallelism layouts and tuning kernels for a model on specific hardware, the configuration ships, and production traffic immediately begins to drift away from the assumptions. Prompts lengthen, batch sizes shift, expert routing patterns change, and the engine that was optimal in the lab gets slowly worse at serving the workload that actually exists.

Together AI has introduced Together Compiler, an agent-driven system built on the opposite premise: tuning should never stop, and Together Compiler makes that economically feasible. TKCC, already running on several of Together's serverless endpoints, is a continual compilation loop in which coding agents rewrite a model's forward pass, making Together Compiler a production system rather than a paper, a persistent engine measures each candidate against real traffic, and the winning code ships into production. When traffic drifts, the loop runs again.

The headline result is up to 2.6 times higher tokens per second than the best published InferenceX configurations, reached in under 24 hours per setting on at most two nodes. This article explains how Together Compiler works, the measurement trick that makes it viable, the genuinely surprising finding about expert knowledge, and the honest limits of self-reported benchmarking.

Why static tuning fails: traffic never sits still

The case Together Compiler makes rests on two observations, one about traffic and one about kernels, and both are measurable rather than rhetorical.

Traffic drift is constant. Together reports that Together Compiler's motivating evidence starts here: prompt lengths in one country ran about 97,000 tokens against 70,000 elsewhere, and that one customer's 90th-percentile input length grew from 18,000 to 33,000 tokens in three days. A parallelism layout chosen for the second shape is quietly wrong for the first. Layout choice matters enormously in memory-bound decode: on MiniMax M3.1 Flash, the cost model picking tensor-parallel-8 over the baseline's tensor-parallel-4 halved each GPU's weight traffic, which is where a large share of the gain comes from.

The kernel observation is subtler and more damning. Kernels are typically tuned on random data, and random data lies. In Together's measurements, random inputs activated 13 to 38 percent more MoE experts than real traffic did. A top-k kernel written by an agent took 20 microseconds on random data; on real traffic at 70,000-token context, it took up to 62 milliseconds. That is a three-thousand-fold miss, invisible in every benchmark that tunes on synthetic inputs.

Together, the observations reframe the problem. The tuning target is not a model on hardware; it is a model on hardware under a specific, shifting distribution of real requests. A target that moves needs a process that moves with it.

The Together Compiler persistent engine: 252 times more experiments

The engineering centerpiece is what Together calls the persistent engine, and its purpose is to remove the tax that makes agentic optimization impractical.

Most agent loops restart vLLM, SGLang or TensorRT-LLM for every experiment. The team measured 761 seconds of cold startup for one model on TensorRT-LLM, which means an agent doing a hundred experiments spends most of its budget and wall clock waiting for servers to boot. Together Compiler starts the engine once and keeps it running. Agents submit candidate forward passes over RPC, the server swaps the forward function pointer in place, and CUDA graphs are re-captured. The engine never restarts, which is the Together Compiler trick in one sentence.

Measurement runs in two tiers that balance speed against fidelity. A fast tier serves only 12 of the model's layers for rapid screening; a slow tier serves all 60 to confirm gains on real distributions. The two-tier setup tested 8.1 times more candidates than a single slow tier, 1,354 against 168, and found a solution 7.2 percent faster, which demonstrates that screening finds candidates the confirmation tier would never have reached in budget. Together reports 252 times higher agent experiment throughput than an engine-restart baseline, which is the Together Compiler number that makes everything else practical.

The 252x is the number that makes the rest of the Together Compiler release possible. Everything else, the agent strategies, the benchmark results, the layout search, depends on being able to afford thousands of experiments per day, and that affordability is an infrastructure achievement rather than a modeling one.

The finding that should change how experts work

Buried in the Together Compiler release is a result that contradicts the standard playbook for agentic optimization, and it deserves more attention than the speedups.

An expert-curated knowledge base, encoding how human engineers think about kernel and layout optimization, produced 25 to 31 percent lower speedup after 12 hours than naive agents given a single instruction: make the forward pass as fast as possible. Ensemble scores reached 6.2 times with naive strategies against 4.9 times with expert strategies after ten trials. The experts' heuristics, encoded as guidance, constrained the agents into the paths a human would take, and the humans' paths were not optimal.

A second finding compounds it. Custom logging structures that agents built to understand engine behavior consumed up to 43 percent of tool calls. Together's fix was to give agents raw engine source code rather than an intermediate representation or curated summaries. The agents, reading the same code a human engineer would read, spent their budget on experiments instead of on building instrumentation.

Both findings point in the same direction. The value of agents in systems work is not their ability to follow expert playbooks faster; it is their willingness to explore paths experts would not, unburdened by assumptions about how the system is supposed to be tuned. For teams building agentic optimization anywhere, the lesson is uncomfortable and cheap to act on: write the goal, give the raw materials, and resist the urge to hand down strategy.

What agents actually rewrite

The phrase agents rewrite a forward pass is doing a lot of work, and it is worth being concrete about the surface they operate on.

A forward pass for a modern mixture-of-experts model is a long composition: embedding lookups, dozens of linear-attention or full-attention blocks, expert routing, expert compute, output projection, and the sampling path. Each stage has layout choices, kernel choices, fusion opportunities and precision decisions. The agents in Together Compiler work at that granularity: replacing a routing kernel, restructuring how experts are grouped for a given expert-imbalance pattern, adjusting how key-value state is staged, or reordering operations to improve overlap. The server swaps the whole function pointer, so a candidate can change any part of the composition.

The evaluation discipline around those rewrites is what keeps the system honest. A candidate that improves the fast tier must survive the slow tier on real distributions before shipping, and bit-exactness was required for the scientific-model runs, where speedups of 1.5 to 6.4 times were achieved without changing outputs. Speed obtained by changing results is not optimization; it is corruption with better latency, and the release draws that line explicitly.

What the agents do not do is equally important. They do not retrain the model, change weights, or alter the model's semantics. The optimization surface is the implementation of a fixed computation, which keeps the risk profile comprehensible: worst case is a slower or crashed candidate, caught in measurement, not a subtly wrong model in production.

What the benchmarks establish

On SemiAnalysis InferenceX's AgentX workload, Together Compiler beat the best open configurations by up to 2.6 times tokens per second, the headline Together Compiler result, with each setting running under 24 hours on at most two nodes.

The per-setting results at batch size 1, P90 TPS per user: MiniMax M3.1 Flash on B200 reached 1,111 against a baseline of 633, a 1.8 times gain and the strongest result. DeepSeek v4.1 Flash on GB300 reached 1,205 against 943. GLM 5.3 on B200 reached 417 against 329. Kimi K3 on GB300 reached 450 against 423, a modest 6 percent. GLM 5.3 on MI355X moved from 463 to 480, about 4 percent. Beyond language models, the system sped up eight scientific models by 1.5 to 6.4 times with bit-exact outputs, and on a 1B model it beat Hazy Research megakernels, Mirage and CUDA-graph baselines by up to 1.7 times. Claude Code with a goal directive reached 1.2 to 4.4 times and failed outright on one workload.

Reading the spread honestly: the same system, on the same day, delivered 1.8 times on one hardware-model pair and 4 percent on another. The gains concentrate where the starting configuration had the most headroom, which is exactly what a tuning system should do, and the MI355X result shows the floor when the baseline is already good. The honest Together Compiler summary is that the system reliably finds whatever improvement exists within 24 hours, and how much that is depends on how unoptimized the starting point was.

It is also all self-reported, run by the company selling the optimization service. The InferenceX baselines are published configurations, which gives the comparison a fixed reference point, but the runs, the hardware and the measurement are Together's own. Independent replication on a customer workload has not happened publicly, and the system is in limited beta for Together customers rather than generally available.

The economics, worked through

The value of a system like Together Compiler is easiest to see in units a buyer can price, so consider what the results imply.

A 1.8 times throughput gain on a serving configuration is functionally equivalent to nearly halving the node count for the same traffic, before even accounting for the tail latency improvements that P90 measurement captures. On the MiniMax result, 1,111 versus 633 P90 tokens per second per user means the same quality of service on roughly 57 percent of the GPUs, or dramatically more users on the same fleet. Against hosted API pricing, those hardware savings compound daily, which is why the system is a product rather than a paper.

The 24-hour budget bounds the cost side of every Together Compiler campaign. Each optimization campaign runs on at most two nodes for under a day, so even a campaign that finds nothing costs a bounded amount of compute. Compare that with a human engineer spending a week on kernel work: the agent campaign is cheaper whether it succeeds or fails, and it runs continuously rather than when someone has bandwidth.

The economics also explain the availability choice. Together Compiler is in limited beta for Together customers because its value is inseparable from serving the traffic on Together's own endpoints, a constraint the TKCC design accepts rather than hides. The traffic profile is the training signal for the optimization loop; a customer with no traffic on the platform gets a system with nothing to measure. The beta is not artificial scarcity. It is the product shape.

How it compares with adjacent systems

The competitive landscape clarifies what is new, because agent-driven kernel optimization already has several entries.

Cursor and NVIDIA's multi-agent system optimized 235 standalone CUDA kernels, achieving a 1.38 times geomean improvement across a three-week budget on B200s. AMD's GEAK targets individual Triton kernels. Apple's RSI-Inference Bench scores agents on 37 tasks editing vLLM or SGLang. All of them optimize kernels or tasks outside a running production server.

Together Compiler's difference is scope and integration: the full forward pass, inside the serving engine, measured against real traffic, with hot reload and a two-tier measurement system that no kernel-level system attempts. Optimizing a kernel in isolation and optimizing a forward pass under production load are different problems, and the traffic-distribution findings at the heart of this release only exist in the second framing. The 24-hour budget against a three-week one is also a claim about maturity, enabled mostly by the persistent engine.

For buyers, the practical takeaway is that agentic optimization is moving from research demos into hosted products, and the differentiator will be measurement fidelity. Any vendor can run an agent against a kernel; the hard part is measuring candidates against the distribution of requests that will actually arrive.

Questions to ask any provider claiming agent optimization

The Together Compiler release is the clearest public statement of a trend, and other providers will follow with their own versions. The questions that separate substance from narrative are specific.

What traffic distribution was the configuration optimized against, and how recently? A configuration tuned on synthetic data, as the random-input kernel findings showed, can be thousands-fold wrong on real requests. How are candidates validated, and does any speedup change outputs, or is bit-exactness required? The scientific-model results set the standard. What happens when a candidate regresses under a traffic shift the optimizer has not seen? A loop that ships improvements should also have a story for reverting them. And who owns the optimized code? A customer whose workload drove a 30 percent improvement should understand whether that improvement is theirs, the platform's, or both.

Each question is answerable in a sentence by a provider doing this honestly, and Together's own release answers most of them unprompted, which is a good sign for the category even before independent numbers arrive.

What it means for inference buyers

The Together Compiler release signals a shift in what inference purchasing looks like, and the shift favors exactly the kind of measurement discipline this site has argued for across serving topics, from the inference caching guide to the model routing cost analysis, and the hardware angle ties into what we covered in the Vera Rubin bandwidth analysis.

First, inference performance becomes a moving target owned by the provider. If an endpoint's throughput improves 30 percent next month because agents re-tuned the forward pass against that month's traffic, comparing providers on a snapshot benchmark is comparing photographs of moving objects. Buyers should ask providers how often their configurations are re-optimized and against what traffic profiles, because questions like those now have real Together Compiler answers that differ across vendors.

Second, the DIY version exists for teams running their own serving. The Together Compiler persistent-engine pattern, hot-swappable forward passes, tiered measurement, real-traffic validation, is described in enough detail to replicate, and the traffic-drift findings alone justify instrumenting production prompt-length distributions even for teams that never build a compilation loop.

Third, the expert-knowledge finding applies far beyond inference. Anywhere agents are being given curated playbooks, the assumption that expert guidance helps should be tested rather than assumed. Together's evidence says the guidance cost them a quarter of their speedup, and the same experiment is cheap to run in most agentic systems.

The limits stay honest: self-reported, provider-run, limited beta, gains from 4 percent to 2.6 times depending on starting headroom. None of that undermines the architecture. It defines the proof burden that comes with a system whose entire pitch is measured improvement, and the burden is one the company has, on this evidence, partially met.

Questions engineers will ask

Does Together Compiler change model outputs?

For the language-model results, the system optimizes the implementation of a fixed computation, and Together Compiler ran the scientific-model campaigns under a bit-exactness requirement, meaning outputs were unchanged. Teams deploying it should verify the same guarantee for their model, since a fast forward pass that alters logits is a different product entirely.

Does it work on my hardware?

The tested list covers NVIDIA B200, B300, GB300 and H100, AMD MI355X and AWS Trainium2, with at most two nodes per model. The MI355X result shows the Together Compiler floor: when the starting configuration is already good, gains shrink to a few percent.

Can I run this loop myself?

Not today as a product. The system is in limited beta for Together customers, and the release describes the architecture in enough detail that a determined team could replicate the persistent-engine pattern on its own serving stack.

Sources
Subscription Form