GLM-5.3 Runs Across Two DGX Stations With vLLM

Posted on by
Contents show
GLM-5.3

GLM-5.3 has crossed a line that most developers assumed was still years away: a frontier scale mixture of experts model can now run on your desk, as long as your desk holds two workstations instead of one. NVIDIA published a working recipe that serves the quantized GLM-5.3 checkpoint across two DGX Stations linked by RDMA, exposing one OpenAI compatible API endpoint backed by tensor parallelism over both machines. The checkpoint weighs about 464 GB, which is why a single GB300 GPU with 252 GB of HBM3e memory cannot hold it alone. Two GPUs, one model, one endpoint: that is the whole architecture, and every part of it is now documented down to the NCCL environment variables. This article walks through why the memory wall forced the two machine design, how the distributed service is launched and verified, and what the published concurrency numbers actually say about serving a 753 billion parameter model from hardware that fits in an office.

Why One GPU Cannot Hold GLM-5.3

The core constraint is simple arithmetic. Z.ai built GLM-5.3 as a sparse mixture of experts model with 753 billion total parameters and only 40 billion activated per token. Sparse activation keeps the compute per token modest, but it does nothing for how much storage GLM-5.3 needs, because any token can be routed to any expert at any time. All 753 billion parameters must stay reachable, and on this deployment all of GLM-5.3 stays resident in GPU memory rather than paged from CPU RAM.

NVIDIA's NVFP4 quantization shrinks the checkpoint to roughly 464 GB, down from what an 8 bit version would occupy, because the quantization pass targets the linear operators inside the MoE blocks and stores weights and activations in 4 bit format. The model card for GLM-5.3 NVFP4 records a size reduction of about 1.66 times relative to the FP8 build, achieved with Model Optimizer version 0.47.0. Even after that compression, the checkpoint exceeds the 252 GB of HBM3e that a single GB300 GPU carries. CPU weight offload is explicitly disabled in this setup, because offloading expert weights to host memory would wreck the latency profile that makes a local frontier model useful.

The quantization is mixed precision rather than uniform. Inside GLM-5.3, attention layers, embeddings, and other sensitive tensors stay in higher precision, while the MoE block linears carry the NVFP4 format. The checkpoint config declares the quantization algorithm as MIXED_PRECISION, and the architecture is listed as GlmMoeDsaForCausalLM, reflecting the sparse attention design with an IndexShare indexer that supports long context input. On the published evaluations, the NVFP4 build of GLM-5.3 holds its accuracy against the FP8 baseline: MMLU Pro moves from 87.81 to 87.88, GPQA Diamond from 91.79 to 92.68, LiveCodeBench V6 from 52.95 to 53.55, and AIME 2025 from 99.38 to 99.53. Quantization noise, where it exists, is inside the run to run variance of the benchmarks themselves.

What Two DGX Stations Change For GLM-5.3

Serving GLM-5.3 this way takes two identical DGX Stations, each carrying one GB300 GPU, connected with a direct ConnectX-8 link. The deployment uses tensor parallelism with a world size of two, written as TP=2 in the launch configuration. Under tensor parallelism, individual model tensors are split across both ranks, so each GPU holds roughly half of every weight matrix rather than a whole layer or a whole expert group. Both machines compute on every request, exchanging partial results over the RDMA rail, and only Station A talks to the outside world through one OpenAI compatible endpoint on port 8000. Station B runs headless as a pure worker rank.

The split is verified at load time: the startup logs report roughly 218 GiB of weights resident on each GPU, which is close to half of the 464 GB checkpoint, and nvidia-smi shows memory climbing from near zero to about 231 GiB per card once the KV cache and runtime buffers are allocated. Neither GPU holds a complete copy of anything. If either Station drops off the rail, GLM-5.3 stops answering, because every token requires both ranks to finish the forward pass.

The connection details matter more than a typical two node cluster. The GLM-5.3 recipe pins the rail interface, the RDMA device, and the IPv4 addresses of both Stations in an environment file, then validates the link with jumbo frame pings at 8972 bytes before anything else runs. The expected link speed is 400 Gb/s with MTU 9000. GPU Direct RDMA is confirmed at startup by scanning the NCCL debug logs for the GDRDMA marker, and the launch fails loudly into a troubleshooting checklist if the check finds socket fallback or NCCL warnings instead.

The GLM-5.3 Checkpoint Math In Detail

Getting the 464 GB of GLM-5.3 onto both machines is a logistics problem the recipe treats as seriously as the serving problem. The checkpoint is pinned by repository revision, so both Stations run bit identical weights, and the image itself is pinned by digest: vLLM version 0.30.0 for arm64 with a recorded sha256. Each Station keeps a full local copy of the checkpoint rather than sharing one NFS mount, because vLLM partitions the GLM-5.3 weights at load time and each rank reads its own shards from local disk.

The download and distribution flow has three verification gates. First, the pinned container image is pulled on both machines and inspected to confirm matching architecture and digests. Second, the checkpoint is downloaded on Station A and copied to Station B over rsync, excluding cache directories, and the file lists of both copies are hashed and compared. Third, the layout check asserts that exactly 50 safetensors shard files named model-00001-of-00050 through model-00050-of-00050 are present and that the config JSON parses with the expected architecture and quantization fields. Only after the check prints MODEL_READY does the launch sequence begin.

This is unusually disciplined for a hands on tutorial about a model of GLM-5.3's size, and the reason is practical: a corrupted or mismatched shard on one rank produces failures that look like NCCL timeouts or CUDA errors, which are far more expensive to debug than a hash mismatch caught before launch. Disk sizing matters too. The published requirement is at least 464 GB free on each Station for the checkpoint alone, plus room for the container image and temporary files, which makes a pair of 2 TB drives a sensible floor for anyone reproducing the setup.

Launching The GLM-5.3 Distributed Service

The launch script for GLM-5.3 is one file, run on both Stations with only the rank number differing between them. Station B starts first as rank 1 in headless mode, then Station A starts as rank 0, joins as the master, and both processes load their share of the weights from local storage, prepare kernels, and capture CUDA graphs over several minutes. The service is ready when the log prints Application startup complete.

Several serving flags do the heavy lifting. The context ceiling is set with max model len at 98304 tokens, which covers input plus output, and the memory fraction is 0.93, handing vLLM 93 percent of GPU memory. Multi token prediction is enabled through a speculative configuration that drafts four tokens per step; the MTP layer proposes candidates and the full model verifies them, so each decode step can emit several tokens when the drafts are accepted. Acceptance rates vary with the prompt, which is why the benchmark section measures rather than promises the speedup. Readers who want a deeper treatment of speculative decoding on small models can see how the same verification pattern plays out in LFM2.5 DSpark's speculative decoding setup, which pairs a draft path with a verifier for single GPU serving.

The GLM-5.3 KV cache is stored in FP8, halving cache memory compared with 16 bit storage and leaving room for more concurrent tokens inside the same 93 percent memory budget. Prefix caching is enabled so repeated prompt prefixes are not reprocessed. Tool calling is enabled with the glm47 parser for both tool calls and reasoning output, which means agent frameworks that speak the OpenAI tools API can point at this endpoint without adaptation.

One dependency deserves its own paragraph. The launcher explicitly loads NCCL 2.30.7 from the image, because older NCCL versions can deadlock cross node CUDA graph replays over RoCE, a problem tracked in vLLM issue 52504. The tutorial also disables custom all reduce and FlashInfer allreduce paths, pins the Infiniband host adapter and GID index to the selected rail, and raises the distributed timeout to 1800 seconds. None of these are optional tuning knobs in the reference deployment; they are the difference between a stable two node service and a node that freezes on its first long request.

Verifying The GLM-5.3 Endpoint

Verification of the GLM-5.3 service runs in three layers. The first is a health check against the local endpoint, which confirms Station A answers on port 8000. The second is the RDMA assertion described earlier, which scans NCCL logs on both containers for the GPU Direct RDMA marker and fails if any socket transport or NCCL warning appears. The third is memory inspection, where the expected picture is one vLLM worker process per GPU, named Worker TP0 on Station A and Worker TP1 on Station B, each holding about 231 GiB.

A real request closes the loop against GLM-5.3. The tutorial sends a small quarterly sales question with temperature zero and reasoning effort set to low, and expects a trend summary plus a 50 percent increase calculation with finish reason stop. GLM-5.3 accepts reasoning effort levels of low, high, and max, trading latency for longer chains of thought. Because the endpoint speaks the standard chat completions schema, the same curl that verifies the deployment is the same call an application makes in production. Developers who want the broader serving context, from batching strategies to logging, can consult the vLLM serving documentation, which covers the single node case that most teams will meet first.

Security is called out where it is usually ignored. The reference launch binds the unauthenticated API to every interface of Station A through host networking, and the tutorial explicitly instructs operators to firewall the port, add authentication, and add TLS before any wider exposure. A local frontier model is still an internet facing service the moment the port is open past the host.

What The GLM-5.3 Benchmarks Show

The published GLM-5.3 performance numbers come from AIPerf version 0.12.0 replaying the 80 Coding conversations from SPEED-Bench, a prompt set built to evaluate speculative decoding on realistic text rather than synthetic strings. The sweep covers configured concurrency 1, 8, 16, and 32, with warmup requests excluded and results measured with prefix caching enabled, four MTP draft tokens per step, and the FP8 KV cache. The benchmark uses the model's default reasoning effort of max with an 8192 token output cap, so token counts include reasoning tokens.

ConcurrencyOverall throughput (tok/s)Mean request latency (s)Mean TTFT (ms)Mean inter-token latency (ms)
1107.449.84479.758.78
8386.0114.17433.7919.01
16607.8136.46482.9023.77
32385.0453.54899.3677.90

Three readings stand out. First, a single user on a local GLM-5.3 deployment gets 107.4 tokens per second overall and 115.3 tokens per second of mean per user throughput, with a time to first token just under half a second. That is a genuinely usable interactive speed for a frontier scale model running on a desk. Second, aggregate throughput climbs to a peak of 607.8 tokens per second at concurrency 16, which is the sweet spot for this configuration with max sequences capped at 16. Third, concurrency 32 degrades everything at once: overall throughput falls back to 385 tokens per second, mean request latency stretches to 453.5 seconds, time to first token nearly doubles to 899 milliseconds, and per user throughput collapses to 13 tokens per second. The KV cache and the scheduling ceiling, not the RDMA rail, are the bottleneck for GLM-5.3 at that load, which is why the recipe documents a separate configuration group with max num seqs raised to 32 for teams that actually need it.

The per user curve is the one most teams should internalize. Per user throughput drops from 115.3 tokens per second at concurrency 1 to 53.0 at concurrency 8, 42.4 at 16, and 13.0 at 32. Sharing the endpoint is efficient for the machine and expensive for every individual waiting on it, and the right operating point depends on whether the workload is interactive, batch, or agent driven. The recipe also documents how to rerun the sweep without prefix caching, and warns that numbers will differ, because the reference measurements all assume prefix reuse is active.

Reading The GLM-5.3 Concurrency Curve

The GLM-5.3 benchmark table rewards a second pass, because the four columns move in different directions and each one maps to a different user experience. Time to first token stays essentially flat from concurrency 1 through 16, between 433 and 483 milliseconds, which means queued GLM-5.3 requests are not starving new ones at those levels. It jumps only at concurrency 32. Mean inter-token latency tells the streaming story: 8.78 milliseconds per token solo, 19 milliseconds at concurrency 8, and 77.9 milliseconds at concurrency 32, the last of which starts to feel sluggish in a chat interface even though the server is still making progress.

Mean request latency grows superlinearly from concurrency 8 onward, from 114 seconds to 136 seconds to 453 seconds. The full sweep can take hours, and the tutorial warns operators to budget for that, with a two hour request timeout configured precisely because reasoning heavy generations at max effort are long. Teams replanning this benchmark with their own prompts should keep the same discipline: warmup runs excluded, server metrics collected, raw responses exported, and one variable changed at a time. Anyone weighing this two machine local setup against rented accelerators can compare it with the way Vera Rubin class bandwidth cuts reshape cluster economics, where the same memory bandwidth pressures show up at datacenter scale.

For teams serving FP4 quantized models on other accelerators, the AMD side of the same tradeoff is documented in the ZYFA FP8 FlashAttention work on the MI355X, where kernel level attention changes were needed to make low precision serving practical. The pattern is consistent across vendors: quantization gets the model into memory, and kernel plus transport engineering determines what the deployment actually delivers.

Where A Local GLM-5.3 Deployment Fits

The recipe closes with the honest use cases for a local GLM-5.3 deployment, and they are narrower than the headline suggests. Running inference over sensitive or proprietary data that cannot leave the building is the first: medical records, unreleased source code, legal corpora, and client data all stay on the two GLM-5.3 Stations, with no per token billing and no third party in the request path. Prototyping agents against GLM-5.3 locally is the second, because tool calling agents burn tokens in ways that make hosted pricing painful during development, and a local endpoint turns that cost into electricity. Iterating on prompts against a frontier scale model without rate limits is the third.

The costs of running GLM-5.3 locally are equally concrete. Two DGX Stations are a six figure capital purchase, the checkpoint needs a terabyte of fast local storage across both machines, and the full benchmark sweep occupies the hardware for hours. Power and cooling for two GB300 class boxes is a real line item. What the deployment removes is the queue: the model is available at 3 in the morning, accepts an arbitrary context up to the 98304 token ceiling, and never rate limits the team that owns it.

The GLM-5.3 model card adds one more operational note that matters for anyone building on top of this endpoint. GLM-5.3 supports context lengths up to 1 million tokens in its native form, while this two Station vLLM deployment caps the served context at 98304 tokens to fit the memory budget alongside the KV cache. Teams that need longer contexts on other runtimes can find the SGLang configuration on the model card, which documents an 8 GPU tensor parallel setup with a 400000 token context on Blackwell hardware.

Questions readers are asking

Can GLM-5.3 run on a single DGX Station with CPU offload? The reference deployment disables CPU weight offload entirely, because the checkpoint at about 464 GB exceeds what one GPU can hold and paging expert weights from host memory would destroy latency. A single Station configuration is not part of the published recipe, and the model card's supported serving examples start at 8 GPU tensor parallelism for SGLang and 2 node TP=2 for vLLM.

How much does the NVFP4 quantization cost in accuracy? The published evaluations show the NVFP4 build matching or slightly exceeding the FP8 baseline across the benchmark suite, including MMLU Pro at 87.88, GPQA Diamond at 92.68, and AIME 2025 at 99.53. The differences are small enough that they sit within normal benchmark variance, because quantization targets only the linear operators inside the MoE blocks while sensitive tensors keep higher precision.

Why does throughput drop at concurrency 32? At 32 concurrent requests the deployment hits its scheduling and KV cache ceiling. Overall throughput falls from the 607.8 tokens per second peak at concurrency 16 to 385, mean latency stretches to over seven minutes per request, and per user throughput drops to 13 tokens per second. The recipe treats 16 as the reference sweet spot and documents a separate configuration with a higher sequence cap for teams that need it.

Does the endpoint work with agent frameworks? Yes. The server enables automatic tool choice with the glm47 parser and exposes the standard OpenAI chat completions schema, including the tools API. Agent frameworks that already speak that schema can point at the endpoint without code changes, and the reasoning effort parameter passes through chat template arguments.

What is required to reproduce the deployment? Two GB300 DGX Stations running Linux, connected and validated with the Connect Two DGX Stations playbook, at least 464 GB of free disk on each Station for the checkpoint, acceptance of the NVIDIA Open Model Agreement and the GLM-5.3 license, and a Hugging Face account with access to the gated dataset used for the benchmark prompts. The recipe pins the container image by digest and the checkpoint by revision, so both ranks run identical software and weights.

Sources
[fluentform id="2"]