NVIDIA’s New 120B Vision-Language Model Runs Like A 12B: What Nemotron 3.5 Super VL Is For

Posted on by
Contents show
A flat vector illustration of a stylized computer monitor showing a simple white eye shape with thin teal circuit traces radiating outward and one warm amber node glowing on the longest trace, representing the NVIDIA Nemotron 3.5 Super VL vision-language MoE model

NVIDIA released Nemotron 3.5 Super VL on 15 October 2026, an open vision-language model with 120 billion total parameters of which only 12 billion activate per token, a 1M-token context window, and a hybrid architecture that mixes Mamba-2 layers into a mixture-of-experts transformer. The release is the vision-language extension of the Nemotron 3.5 family, and it lands in a market segment, enterprise agents that read documents, watch video and hold long context, where the competition is mostly text-first models with vision bolted on. The model card is unusually complete: architecture, six-stage training methodology, deployment recipes for specific GPU counts, a benchmark table covering more than thirty evaluations with named competitors, and published pre-training datasets, all under the OpenMDW-1.1 license.

The headline claim writes itself from the specs, 120B parameters running like 12B, but the interesting story is what the numbers say about where the model wins and loses. It posts strong hallucination-control and long-context results, and it trails the strongest competitors on raw coding and several agentic benchmarks. This article works from the model card read directly, explains the hybrid MoE design, walks the benchmark table honestly, and covers what deployment actually requires.

The architecture: why 120B acts like 12B

Nemotron 3.5 Super VL is a Mamba-2-transformer hybrid mixture-of-experts model with 512 routed experts per MoE layer and a C-RADIO vision encoder connected to the language backbone. The design descends from Nemotron 3 Super Base, and the model card traces the lineage through distillation from Nemotron 3 Ultra Base, which matters because it means the vision-language release inherits a tested language backbone rather than starting fresh. The design combines three layer types: sparse expert computation, standard attention, and Mamba-2 state-space layers, which handle long-context processing with bounded memory rather than attention's quadratic cost. Vision inputs, images, document pages and video frames, are turned into representations by the encoder and connector and processed by the same backbone.

The efficiency story has three parts. The MoE sparsity means each token activates 12 billion of the 120 billion parameters, so serving cost tracks the active figure. The hybrid Mamba-2 layers keep long-context memory bounded, so the 1M window does not multiply attention cost. And a shared Multi-Token Prediction head enables native speculative decoding: the model predicts several tokens at once and verifies them, with the vLLM recipes shipping the configuration inline, no separate drafter model to download. Each technique attacks a different cost axis, parameters, context and decoding, which is why the combination matters more than any single figure. Deployment is specified precisely: four H100 GPUs or two GB200s, with tensor-parallel configurations and MTP draft lengths published per setup.

Context runs to 1 million tokens, covering prompt, visual inputs, conversation history, tool results and responses together. Video input is sampled at configurable frame rates, the default recipes take up to 256 frames at 2 FPS with pruning, and documents can be supplied as page images with accompanying text. The card's guidance on video profiles is practical rather than perfunctory: 1080p footage gets 1 FPS and up to 128 frames, while lower-resolution video can take 2 FPS and 256 frames, and teams should start with clips of one to two minutes. Those defaults exist because video tokens are expensive and the pruning rate trades detail for coverage, so a deployment serving surveillance footage needs different settings from one processing meeting recordings.

The Mamba part of the hybrid, explained

The least familiar component in the architecture is the one doing much of the long-context work, so it deserves a paragraph of its own. Traditional transformers use attention, where every token can look at every other token, which is powerful but expensive: the memory and compute for attention grow with the square of the context length, and a million-token window becomes costly to serve. Mamba-2 layers replace that mechanism with a state-space approach, where the model carries a compressed running summary of what it has seen and updates it as each token arrives. The cost grows linearly with context, and memory stays bounded.

Neither approach is strictly better. Attention is unmatched at retrieving a specific detail from anywhere in the context; state-space layers are efficient at maintaining the overall thread of a very long document or agent trajectory. Hybrid architectures interleave the two so that a model gets attention's recall where it matters and state-space efficiency everywhere else, which is how Nemotron 3.5 Super VL offers a 1M-token window on hardware where a pure-attention model of the same size would be impractical to serve. The card's deployment recipes even expose the implementation detail that matters operationally: the Mamba state cache has its own dtype and rounding settings, which is the kind of knob that only exists in a hybrid and only matters if you understand the split.

The MoE component then divides the per-token compute: 512 experts per layer, of which a handful route each token. The combination, hybrid layers for context length, sparse experts for cost, MTP for decoding speed, is a stack of three independent efficiency techniques, and the published benchmark table is the evidence that stacking them did not cost quality where the model's workloads need it.

The training pipeline is published, which is the quiet headline

Nemotron's family pitch is open weights, training data and recipes, and the Nemotron 3.5 Super VL model card documents the full six-stage pipeline: foundation training with the MTP head, continued pre-training including a 35B-token context extension to 1M and a 67B-token software-engineering mid-training stage, vision-language training, two-stage supervised fine-tuning on sequences up to 524,288 tokens, reinforcement learning with verifiable rewards plus multi-teacher distillation, and a final MTP boosting stage for speculative-decoding acceptance.

For teams evaluating open models, that documentation is a competitive feature in itself. A model whose training stages, data sources and evaluation harness are published can be reproduced, adapted and audited in ways a model with only weights cannot. The card notes that evaluation results were collected through NeMo Gym with published recipes specifying containers, prompts, inference parameters and scoring settings, which is the reproducibility infrastructure most releases promise and few ship.

The benchmarks: strong where it is aimed, honest where it trails

The card's table spans agentic, reasoning, coding, multimodal, conversational, long-context and multilingual evaluations against six named competitors. Reading it without cherry-picking, three patterns emerge.

Hallucination control is the standout. This is the claim that separates Nemotron 3.5 Super VL from its comparison set most clearly. On the AA-Omniscience index, which balances accuracy against hallucination rate, Nemotron 3.5 Super VL posts 4.03 against deeply negative scores for several competitors, with a 30.0 hallucination rate against rates as high as 84.5 and 95.9 elsewhere. On CritPt it scores 10.86 against 3.1 for the nearest comparable competitor. For document-heavy and knowledge-heavy agent work, where a confidently wrong answer is worse than no answer, that profile is the model's clearest advantage.

Long context delivers. The Mamba-2 investment shows up here. GraphWalks, a long-context graph traversal benchmark, gives Nemotron 3.5 Super VL a score of 89.71 against 88.4 for the best competitor and the mid-60s for others. Combined with the 1M window, that supports the intended workloads: whole-codebase reasoning, long document pipelines, extended agent trajectories.

Coding and several agentic benchmarks trail the leaders. This is the part of the table the launch coverage softened, and the model card does not. On SWE-bench Verified, Nemotron 3.5 Super VL posts 71.10 against 79.10 for GLM-5.3-Flash and 78.7 for Qwen3.8-Flash-Next. Terminal-Bench 2.1 shows 60.67 against 85.80. LiveCodeBench v6 shows 88.22 against 94.6 for DeepSeek-V4-Flash0731. On tau2-bench telecom it reaches 73.19 against 98.2 and 98.5 for two competitors. The card does not bury these numbers, and the honest conclusion is that this is not the strongest open model for pure coding agents; it is a multimodal, long-context, hallucination-resistant model with good-not-leading coding.

There is also a local sibling worth noting: Nemotron 3.5 Bolt, a 74B vision-language model with native 2-bit quantization designed to run everyday agentic tasks in 24GB of memory, released the same day. The pair makes the family strategy explicit: Bolt handles the always-on local agent that lives on a workstation, Super VL handles the serving-tier work that needs quality headroom, and both share the family's tooling, license approach and published recipes. The pair frames a deployment ladder, Bolt on a workstation, Super VL on a node, sharing the family's tooling.

Deployment reality: four H100s or two GB200s

The 12B active figure makes serving efficient but does not make it small. The full-precision weights need four H100s or two GB200s, per the card's deployment table, with the NVFP4 release lowering the floor for DGX Spark-class hardware. Both vLLM and SGLang recipes are published down to the container image, including video sampling parameters, tool-call parser selection and speculative decoding configuration. For teams without that hardware, the license permits hosted deployments, and the NVFP4 variant is the deployment-optimised release, with the card directing DGX Spark and datacenter operators to it explicitly. The serving stack is standard: vLLM and SGLang are the supported runtimes, the container images are pinned, and the OpenAI-compatible client examples cover text, image, video and tool calling, so integration is a configuration exercise rather than an engineering project.

The OpenMDW-1.1 license is worth a sentence: it is a permissive open-model license permitting commercial and non-commercial use, more permissive than Meta's custom terms for Muse Spark Open, and consistent with the Nemotron family's open-data-and-recipes positioning. Combined with the published pre-training datasets, it makes this one of the most complete open releases of the autumn.

Where this fits in the autumn's open-weights wave

October's open-weight releases have mostly been either very large MoE models or small local ones, and Nemotron 3.5 Super VL occupies the space between them on purpose. Nemotron 3.5 Super VL occupies the middle deliberately: big enough for frontier-adjacent agentic work, sparse enough to serve economically, multimodal in a tier where most competitors are text-first, and documented deeply enough to build on. Against Muse Spark Open's 816B scale, it trades peak capability for operability; against the Bolt tier, it offers the quality headroom for customisation. Teams choosing among them are really choosing workloads: pure coding volume points at the leaders on that benchmark, document and video pipelines with low hallucination tolerance point at Nemotron 3.5 Super VL, and always-on local agents point at Bolt. The honest decision framework is per-traffic, and the published tables give enough detail to run it without buying a benchmark suite first.

That workload-matching discipline is the running theme of this site's model coverage, from model routing to the open-weight releases of the past weeks: the question is never which model is best, but which model's strengths line up with the traffic's shape. Nemotron 3.5 Super VL makes that matching easier than most releases, because its card documents the failures alongside the wins, and a team that knows its traffic is document-and-video-heavy with a hard hallucination budget can choose on the merits. The NVIDIA cluster on this site, including the OpenShell coverage, now extends from hardware through security tooling to models.

Where the numbers stop

  • Every benchmark figure comes from NVIDIA's model card. The card states that all evaluation results were collected via NeMo Gym with published recipes, which makes replication possible, but no third-party verification existed at publication.
  • The competitor set and their configuration labels are the card's choices. Different reasoning settings or harnesses can reorder close results, and several competitor entries carry configuration qualifiers in the original table.
  • Some multimodal benchmarks used scaffolding NVIDIA states it plans to open source in the future, so a subset of the vision results is not yet independently reproducible from published tooling.
  • The 12B active figure describes routing behaviour per token, not total memory needs: the full-precision deployment still requires four H100s or two GB200s, and teams sizing infrastructure from the active figure alone will under-provision.
  • Video understanding figures depend on the sampling configuration, frame rate, frame count and pruning rate, and the card's recipes specify defaults that materially affect results. Workloads with different video profiles should re-evaluate rather than extrapolate.

FAQ

What is Nemotron 3.5 Super VL?

It is NVIDIA's open vision-language model released on 15 October 2026: 120 billion total parameters with 12 billion active per token, a hybrid Mamba-2-transformer mixture-of-experts architecture with 512 experts per layer, a C-RADIO vision encoder, and context up to 1 million tokens. It accepts text, images and video and outputs text, with native speculative decoding through a Multi-Token Prediction head. It is released under the OpenMDW-1.1 license with weights, published pre-training datasets and evaluation recipes.

How does a 120B model run like a 12B?

Two mechanisms. The mixture-of-experts sparsity means each token routes to 8 of 512 experts, activating only 12 billion parameters, so compute cost per token tracks the active figure rather than the total. And the native Multi-Token Prediction head enables speculative decoding, where the model drafts several tokens ahead and verifies them, increasing throughput without a separate drafter model. The memory footprint remains a 120B model's, so deployment needs four H100 GPUs or two GB200s in full precision, with an NVFP4 release for smaller footprints.

What is it best used for?

By its own published benchmarks, its strongest profile is long-context, document-heavy, video-inclusive agent work where hallucination control matters: it posts the best hallucination-rate figures in its comparison table, leads the long-context GraphWalks benchmark, and handles whole-repository and multi-document context in a 1M-token window. Pure coding-agent throughput is not its lead: SWE-bench Verified and Terminal-Bench 2.1 scores trail the strongest competitors, and teams whose traffic is mostly coding volume should weigh that honestly. Its local sibling, Nemotron 3.5 Bolt, covers the 24GB workstation tier for everyday agentic tasks.

The short version

Nemotron 3.5 Super VL is NVIDIA's 120B-total, 12B-active vision-language model with a 1M-token window, a Mamba-2-hybrid MoE architecture, native speculative decoding, and a benchmark table that reads like an engineering team's internal review: leading hallucination control and long context, trailing on coding. The deployment recipes are published to the container, the training pipeline is documented in six stages, and the license is permissive. It is not the open-weights king of coding, and its own card says so. It is the open model built for the workload most enterprise agents actually have, reading documents, watching video, holding long context and not making things up, and it makes that workload affordable at four H100s.

Sources
Subscription Form