qwenGPU: A SIMT GPU Compiled Into a Language Model

Posted on by
Contents show
qwenGPU

A language model that runs CUDA inside its own forward pass sounds like a joke someone would tell about compute shortages. It is also real, in a narrow and fascinating sense. An experimental project called qwenGPU compiles a small SIMT GPU directly into the weights of Qwen3.5-2B, and the resulting model executes real CUDA kernels, compiled to NVIDIA PTX, as part of generating its next tokens. Nothing about the base model was retrained, and the qwenGPU result is exact rather than predicted. A dot product of [1, 2, 3] and [4, 5, 6] returns exactly 32, computed by the model itself, in a launch it wrote in 47 tokens.

The project comes from an independent developer, released on GitHub under an MIT license for the code with Apache 2.0 for any built checkpoint, and it belongs to a small family of demonstrations that treat transformers as hardware: an earlier project compiled a 32-bit RISC-V CPU into a 4B model, and another built a WebAssembly interpreter into a custom transformer. qwenGPU is the first of the family to target a GPU, and the first to claim exactness against real silicon.

This article explains how the qwenGPU machine fits inside attention weights, what the host still does, what the numbers actually demonstrate, and where the line between research demo and useful tool sits, which in this case is unmistakably drawn.

The trick: a recurrent state that behaves like memory

The whole qwenGPU construction rests on one property of a specific layer type inside modern models.

Qwen3.5 uses Gated DeltaNet, a linear attention layer that keeps a recurrent state matrix updated as tokens flow through. In normal operation the gate modulates how much of the old state survives each update, and writes blend into what is already there. qwenGPU instead holds the gate open and sets the write strength to one, so each update overwrites exactly one row of the state completely. Keys are one-hot, so a query touches exactly one row and reads it back unchanged.

Under those constraints, each row of the recurrent state behaves as a memory cell: write a row, read a row, with nothing in between blurring it. The qwenGPU machine is built out of those cells. It is a 4-lane SIMT design with 15 usable registers per lane, a 4-stage pipeline, a 32-block program ROM and 126 words of global memory, all living in the same state matrix. The machine supports integer arithmetic, predicates, divergent branches and loops, and even shared memory, atomic additions, synchronization barriers and warp shuffles, For qwenGPU, matching those primitives exactly was the engineering challenge. the primitives that make a GPU a GPU rather than a calculator. Compiled into a 2B model, those primitives are the ambition of the project in miniature.

What makes qwenGPU remarkable is that none of its 45 GPU layers were trained. A compiler computes every weight. The transformer is not being taught to simulate a GPU; it is being used as a substrate whose state update semantics happen to be a perfect fit for one.

What the host still does, and why that matters

The model does not read CUDA source. The qwenGPU division of labor is precise and worth stating, because it defines both the achievement and the boundary.

The host plays the role that the compiler toolchain and driver play on a real machine. It compiles CUDA to PTX with clang, lowers PTX to the machine's block format, and feeds the program and input arrays to the model as tokens. From that point the model takes over: each EXEC(b, w) token it emits runs block b for warp w in a single forward pass, VAL tokens carry register results, memory writes come back as records, and the host reads the final results from the recurrent state. The host compiles, loads and reads. It computes none of the answer.

That boundary is why qwenGPU's exactness claims are meaningful rather than circular. If the host computed the results and the model merely echoed them, the demonstration would be theater. Because the host only handles the program and the model derives every value in its forward passes, the correctness comparison against a real GPU tests something real: whether the circuit compiled into weights implements the same machine as silicon.

A launch, token by token

The clearest way to grasp the design is to trace the demo end to end, because every step is specified.

A user asks the model to compute a dot product of [1, 2, 3] and [4, 5, 6]. In normal mode the model recognizes a pattern with a ready-made kernel and writes an expression for it. The host compiles the corresponding kernel to PTX and lowers the result into the machine's block format. The program, with its input arrays, enters the context as tokens.

Generation then proceeds through the merged model. The compiled GPU layers ignore ordinary language tokens and execute only on EXEC markers: EXEC(0, 0) runs block zero, warp zero, one forward pass each. In that pass, lanes load their operands from the input arrays into registers, perform their multiply, and accumulate. The values surface as VAL tokens in the output stream, and any memory writes return as MEM records that the host reads from the final recurrent state. The answer 32 arrives not as predicted text but as a value the circuit computed, retrieved from memory cells that live inside attention state.

Forty-seven tokens for the launch. A human programmer writes comparable kernels in a few lines of C; the model's contribution is the orchestration, choosing the kernel, sequencing the launches, and formatting the exchange. The arithmetic itself was never generated statistically. That is the line the project walks: probabilistic selection of a deterministic machine, then purely deterministic execution inside it.

The evidence, weighed carefully

Every qwenGPU number here is self-reported, measured on Apple silicon with bf16 precision, and the strongest of them is genuinely strong for this kind of project.

The reference interpreter, a software model of the compiled circuit, matched an NVIDIA Tesla T4 on 49 of 49 test kernels. Every launch in the developer's runs was exact under a replay check. The merged model's next-token distribution stays within a KL divergence of about 7e-4 of the original Qwen3.5-2B, because the circuit is muted on ordinary text tokens and the base weights are untouched. In other words, the GPU rides along without measurably damaging the language model that carries it.

The weaker numbers are equally instructive. In normal qwenGPU mode, where the model writes expressions for ready-made kernels, 20 of 24 sampled questions produced GPU-computed answers. In advanced mode, where the model writes the CUDA kernel itself, it succeeded on 4 of 5 questions, and the repository warns plainly that a 2B model's CUDA is unreliable. And the whole machine runs at 10 to 15 tokens per second on Apple silicon. A real T4 would finish these kernels in microseconds.

The honest summary of qwenGPU: the exactness result validates the qwenGPU compilation technique. The speed and reliability numbers confirm that this is a demonstration of principle, not a runtime anyone should ship.

Why a GPU is harder than a CPU

The jump from Qwenputer's CPU to qwenGPU is not a change of scale but a change of class, and the added machinery explains why.

A CPU executes one instruction stream. Its compiled circuit needs registers, a program counter, and arithmetic, all of which map naturally onto state cells that are written and read in sequence. A GPU executes many warps concurrently, and its defining features are interactions between those warps: barriers that force alignment, atomic operations that must not lose updates, shuffles that exchange values between lanes. Each of those is a coordination primitive, and coordination is exactly what independent state cells do not provide for free.

The compiled machine handles it with its 4-stage pipeline and per-lane register files, keeping each warp's state separated while the synchronization primitives operate across the machine. Whether every edge case of warp divergence and barrier semantics is faithfully compiled is precisely what the 49-kernel comparison against a Tesla T4 was designed to probe, and the reported zero mismatches on that suite is the strongest single data point in the release.

What the comparison does not yet cover is scale. A 4-lane machine with 126 words of memory is a toy dimension, and the interesting open question is whether the compilation technique scales to wider machines without the state cells interfering, since the one-hot discipline that makes each row a clean memory cell depends on careful bookkeeping that larger designs may strain.

The family it belongs to, and why the lineage matters

qwenGPU did not appear from nowhere, and comparing the three known members of this compiled-machine genre clarifies what each proved.

Qwenputer compiled a 32-bit RISC-V CPU into a 4B model, ran C and MicroPython, and validated with zero mismatches across roughly 230,000 replayed positions. Percepta built a WebAssembly interpreter into a custom transformer with two-dimensional attention heads, claiming millions of reliable steps. qwenGPU compiles a 4-lane SIMT GPU into a 2B model, runs CUDA lowered from PTX, and validates against physical silicon rather than only against its own interpreter.

The progression is architectural. Qwenputer proved deterministic computation could live in weights. Percepta proved an interpreter could run in context. qwenGPU proves a parallel machine with synchronization primitives can be compiled into modern hybrid attention layers, and checked against the real hardware it imitates. Each step widened the gap between what the model is doing and what the base model was trained to do, which is the actual research question: how far can the transformer substrate be pushed away from its training distribution before exactness breaks?

It is also worth naming what the host boundary means philosophically. The claim is not that language models learn to compute. It is that transformer architectures contain, as a side effect of their state mechanics, enough structure to compile arbitrary machines into. The learning never happens. The compilation does.

What it takes to run it

The requirements tell you the audience immediately.

Running qwenGPU needs Python 3.11 or newer, the uv package manager, the just task runner, and about 16 GB of free memory. The build command downloads the Qwen3.5-2B base model and compiles the GPU into it; there is no prebuilt checkpoint, so every user compiles their own, and any built checkpoint falls under the base model's Apache 2.0 license. Advanced mode, where the model writes its own CUDA kernels, additionally requires a clang toolchain with the NVPTX backend installed.

The hardware story is flexible by the standard of machine code: Apple silicon, an NVIDIA GPU, or a plain CPU all work, with roughly 16 GB free. That portability is itself a tell. The compiled circuit runs wherever the base model runs, because it is weights rather than a device program, and the same property that makes it slow makes it indifferent to hardware.

The qwenGPU KL number and what riding along costs

The least glamorous number in the release may be the most important for the genre's future.

The merged model's next-token distribution differs from the untouched base model by a KL divergence of about 7e-4, which is small enough that ordinary text generation is unaffected for practical purposes. That is achieved by muting the compiled circuit on ordinary text tokens, so the GPU layers only act when the conversation enters machine-execution mode. Qwen's own weights are never modified.

The design answers the obvious objection, that bolting a machine onto a language model would corrupt its language ability. It does not, because the machine is not overlapping with the language pathway; it is a parallel track switched on by token type. The residual drift comes from the extra layers being present in the network at all, and a number that small means the substrate tolerates the passenger.

The cost side is less rosy. The circuit adds 45 layers to every forward pass, and the model runs at 10 to 15 tokens per second on Apple silicon, so text generation pays a latency tax for a machine it is not using. A production design would need the GPU layers to be skippable or sparse, and nothing in the current release addresses that. It is a research artifact with research-artifact overheads.

Why anyone should care, and who should not

For most developers, the correct takeaway on qwenGPU is curiosity rather than adoption, and the reasons are concrete. But curiosity has a direction here, and it is worth naming the three audiences for whom this release is genuinely relevant rather than merely entertaining.

Nothing here is a runtime. Ten to fifteen tokens per second, a 16 GB memory requirement, no quantized build, and a machine with 126 words of global memory are not deployment characteristics. No prebuilt checkpoint is published; building it downloads the base model and compiles the GPU in. And the useful-compute ceiling is microscopic, since the qwenGPU machine exists in a forward pass that costs billions of multiply-accumulate operations to run a kernel that a real GPU would execute in a fraction of a millisecond.

The audiences who should care are specific. Researchers in mechanistic interpretability get a rare artifact: a set of weights whose function is fully specified by a compiler, rather than learned, which makes them a control group for studying what learned layers do. Interpretability research is haunted by the fact that every layer being studied was shaped by gradient descent, so structure and artifact are inseparable. A 45-layer block whose behavior is known exactly, down to the semantics of every cell, offers something the field rarely has: a baseline where any deviation from specification is an measurement error, not an unexplained mechanism. Whether the research community exploits that is open, but the artifact is real, and it costs nothing to produce once the compiler exists. People thinking about verifiable computation get a proof of concept that a model's computation can be exactly specified and replay-checked, a property that matters as agents take on higher-stakes work. The replay check in qwenGPU is the ancestor of a real need: if an agent will act on computed values, someone will eventually demand proof those values were computed correctly, not plausibly generated. A model whose arithmetic runs in a circuit with replay semantics is a sketch of what that proof could look like, and the distance from here to useful is measured in engineering rather than ideas. And model architects get a data point about hybrid attention layers: the Gated DeltaNet state, designed for sequence memory, turned out to be mechanically capable of hosting a general machine, which says something about how much untapped structure exists in these layers.

There is also a useful discipline lesson in the qwenGPU release itself. The developer published the validation method, the exactness replay, the interpreter comparison against real silicon, and the failure rates, all alongside the demo. That is the standard the rest of this genre should be held to, and it is the same standard we applied when examining harness engineering for coding agents: claims about what a system does are only as good as the replay that checks them.

For readers tracking the agent infrastructure space more broadly, the distance between this project and production systems is a useful calibration. The work making agents actually useful today is unglamorous: persistent compute, grounded data access, and reliable model serving. Projects like qwenGPU matter on a different clock, the one that measures what these architectures fundamentally are, and on that clock this is a genuinely surprising data point. The compiled-machine genre is two years old, has already gone from a RISC-V core to a synchronized parallel machine checked against silicon, and shows no sign of having found its ceiling. What remains unknown is whether any of it becomes useful, or whether it stays a beautiful way of proving that the substrate is stranger than its training.

Sources
Subscription Form