26 Trillion Tokens, 2-Bit Weights: Inside Nvidia's Nemotron 3.5 Super VL and Bolt

On October 15, 2026, Nvidia announced two lines at once: Nemotron 3.5 Super VL and the Nemotron 3.5 Bolt models, which the company says cover the full range of agent deployments from enterprise servers down to edge devices and local hardware. Super VL is an open 120-billion-parameter multimodal mixture of experts that activates 12 billion parameters per token, built with the Nemotron Coalition, and it folds reasoning, instruction following, tool use and text, image, document and video understanding into one model. Nvidia's claim is comparable power to its largest reasoning models at one-fifth the size, and roughly 5x the throughput of the next best model in its class. Bolt 74B-A9B uses 2-bit quantization to fit 74 billion parameters into 24GB for a workstation or PC GPU, with a 4GB Bolt E4B for wearables and dashboards following later this month. Two pieces of arithmetic are worth doing before any of that matters to you: how much headroom 24GB actually leaves after the weights, and what a 5x throughput claim has to do with the latency your users feel.

A 12-billion-active tier inside a 120-billion-parameter body is the whole trick.

Three tiers, and what open actually covers

This is not one model, it is a layered set: Super VL at 120B for the data centre, Bolt 74B-A9B at 24GB for the workstation, and Bolt E4B at 4GB for edge devices, arriving later this month. Open is worth decomposing. The weights are public, so an organisation can post-train on its own data, deploy on its own infrastructure and keep the whole thing behind the firewall, which is the sovereignty, privacy and safety argument. The training approach is open too: about 26 trillion tokens across pretraining and continued pretraining, 3.8 million post-training samples, and reinforcement learning across 100 open environments, with datasets released on Hugging Face covering agentic tool use, finance, instruction following, long context, safety, fairness and vision-language understanding. The value is not that it is free. It is that you can hold the weights, the provenance and the evaluation evidence yourself, which is what you need when someone asks what model was running on their data.

# Fitting 74B into 24GB is a budget problem, not a magic trick. Do the arithmetic.
BYTES_PER_PARAM = {"fp16": 2.0, "fp8": 1.0, "int4": 0.5, "int2": 0.25}

def weights_gib(params_b, precision):
    return params_b * 1e9 * BYTES_PER_PARAM[precision] / (1024 ** 3)

CARD_GIB = 24.0
OVERHEAD_GIB = 2.0        # driver, CUDA context, framework

print("74B (Bolt 74B-A9B) on a 24GB card")
for p in ("fp16", "fp8", "int4", "int2"):
    w = weights_gib(74, p)
    room = CARD_GIB - OVERHEAD_GIB - w
    verdict = "fits" if room > 0 else "does not fit"
    print(f"  {p:5} weights {w:6.1f} GiB  headroom {room:+6.1f} GiB  {verdict}")

print()
print("Headroom after 2-bit weights must hold KV cache and activations.")
def kv_budget(room_gib, layers=48, kv_heads=4, head_dim=128, bytes_per_value=2):
    per_token = 2 * layers * kv_heads * head_dim * bytes_per_value / (1024 ** 3)
    return {"tokens_at_batch1": int(room_gib / per_token), "per_token_gib": round(per_token, 6)}

print(kv_budget(CARD_GIB - OVERHEAD_GIB - weights_gib(74, "int2")))

# Nvidia says 2-bit quantization is what makes the 24GB target work. This is why
# the headroom matters more than the headline: the weights are only the first bill.

5x is throughput, not latency

Nvidia's wording is the fastest open agentic model in its class, 5x faster than the next best, based on Artificial Analysis testing of the model against similarly sized competitors, with InclusionAI's Ling 3.0 Flash second and DeepSeek's V4.1 Flash third. There are three layers to reading a number like that. Throughput in tokens per second accumulates under batching, so per-GPU throughput is fundamentally a billing metric. What a user feels is time to first token and inter-token speed, and a model can win on aggregate throughput because batching is efficient while still losing on single-stream latency. And throughput says nothing about quality, because a same-class comparison only establishes that it is quick among models of its size, not that it is right on your task. Separating the three tells you what 5x means for you: for bulk document and log processing, throughput converts directly into cost, while for interactive code completion, first-token latency is the constraint that decides.

# "5x faster" is throughput. Your users experience latency. Those are not the same.
def throughput(tokens_per_sec, batch):
    return tokens_per_sec * batch

def cost_per_million(tokens_per_sec, batch, gpu_hour_usd):
    per_hour = throughput(tokens_per_sec, batch) * 3600
    return round(gpu_hour_usd / per_hour * 1_000_000, 4) if per_hour else None

CANDIDATES = [
    {"name": "very large dense", "tok_s": 260, "batch": 8},
    {"name": "mid dense", "tok_s": 700, "batch": 8},
    {"name": "12B-active MoE", "tok_s": 1300, "batch": 8},
]

print(f"{'model':18} {'tok/s':>7} {'batch':>6} {'agg tok/s':>10} {'$/1M tok @ $3.20/h':>20}")
for c in CANDIDATES:
    agg = throughput(c["tok_s"], c["batch"])
    print(f"{c['name']:18} {c['tok_s']:7} {c['batch']:6} {agg:10} "
          f"{cost_per_million(c['tok_s'], c['batch'], 3.20):>20}")

# Single-stream latency is a different table. A model can win on aggregate
# throughput because batching is efficient while losing on time-to-first-token.
def ttft(prompt_tokens, prefill_tok_s):
    return round(prompt_tokens / prefill_tok_s * 1000, 1)

print()
for p in (2000, 32000, 128000):
    print(f"prompt {p:>7} tokens -> TTFT {ttft(p, 6000):>8} ms (at 6000 prefill tok/s)")

# Nvidia's 5x claim is a same-class comparison, tested by Artificial Analysis
# against similarly sized competitors. Read it as throughput, not as quality.

The same weights are meant to run on a data-centre node and on a workstation. That is a new bar.

74B in 24GB starts with the headroom, not the headline

Two-bit quantization puts 74 billion parameters at roughly 17.2GiB, which on a 24GB card leaves a little over 4GB after about 2GiB of runtime overhead, and that remainder has to hold KV cache, activations and buffers rather than weights. The same arithmetic fails at every other precision: about 68.9GiB in FP8 and 137.7GiB in FP16, both far beyond a single card. That is what makes this release worth attention. It is not another quantized checkpoint, it is a model that takes the small-active-parameter route and the low-bit-weight route at the same time, which is what makes a 24GB target reachable at all. It also changes the first question worth asking when you evaluate it. Not how clever the model is, but how much headroom is left on the target card after the weights, and how much context and concurrency that headroom can carry.

# Open weights plus open recipes only pay off if you gate the post-training run.
from dataclasses import dataclass, field

@dataclass
class EvalCase:
    name: str
    category: str          # tool_use | finance | instruction | long_context | safety | vision
    passed: bool

@dataclass
class PostTrainGate:
    baseline: dict = field(default_factory=dict)
    max_allowed_regression: float = 0.02

    def score(self, cases, category):
        subset = [c for c in cases if c.category == category]
        if not subset:
            return None
        return sum(c.passed for c in subset) / len(subset)

    def check(self, before, after, categories):
        report = []
        for cat in categories:
            b, a = self.score(before, cat), self.score(after, cat)
            if b is None or a is None:
                continue
            delta = a - b
            blocked = delta < -self.max_allowed_regression
            report.append({"category": cat, "before": round(b, 3), "after": round(a, 3),
                           "delta": round(delta, 3), "blocked": blocked})
        return report

BEFORE = [EvalCase(f"b{i}", c, True) for i, c in enumerate(
    ["tool_use"] * 20 + ["finance"] * 10 + ["safety"] * 10 + ["long_context"] * 10)]
AFTER = ([EvalCase(f"a{i}", "tool_use", i < 19) for i in range(20)] +
         [EvalCase(f"f{i}", "finance", i < 10) for i in range(10)] +
         [EvalCase(f"s{i}", "safety", i < 8) for i in range(10)] +
         [EvalCase(f"l{i}", "long_context", i < 10) for i in range(10)])

gate = PostTrainGate()
for row in gate.check(BEFORE, AFTER, ["tool_use", "finance", "safety", "long_context"]):
    flag = "BLOCK" if row["blocked"] else "ok"
    print(f"{row['category']:14} {row['before']:.3f} -> {row['after']:.3f} ({row['delta']:+.3f}) {flag}")

# Nvidia publishes the ingredients (about 26T tokens, 3.8M post-training samples,
# RL across 100 open environments) plus datasets on Hugging Face. The gate above is
# what stops a fine-tune from quietly trading safety for tool-use accuracy.

Post-training needs an eval gate, or you are gambling

The real value of open weights plus an open recipe is that you can post-train on your own data, and Nvidia points at NeMo for using your own data, tools, policies and workflows. That also moves the risk from the vendor to you. A fine-tune aimed at tool-use accuracy can quietly degrade safety behaviour, and nothing in the training run will tell you. So the pipeline needs an eval gate: run the same categorised suite before and after, compare per category, and block the merge when any category drops past its threshold. The snippet above is the skeleton. Set thresholds per category rather than globally, because losing two points of long-context capability and losing two points of safety are not the same event, and the second is usually a zero-tolerance case.

# Local or cloud is a policy question before it is a cost question.
from dataclasses import dataclass

@dataclass
class Job:
    name: str
    data_class: str        # public | internal | regulated
    prompt_tokens: int
    needs_vision: bool
    latency_budget_ms: int

TIERS = {
    "edge_4gb":   {"max_prompt": 8_000,    "vision": False, "cost_per_1k": 0.00002},
    "workstation_24gb": {"max_prompt": 60_000, "vision": True,  "cost_per_1k": 0.00012},
    "datacenter": {"max_prompt": 256_000,  "vision": True,  "cost_per_1k": 0.00040},
}

def eligible(job, tiers):
    out = []
    for name, t in tiers.items():
        if job.prompt_tokens > t["max_prompt"]:
            continue
        if job.needs_vision and not t["vision"]:
            continue
        if job.data_class == "regulated" and name == "datacenter":
            # a regulated payload leaves the building; only allowed with an explicit waiver
            continue
        out.append(name)
    return out

JOBS = [
    Job("inline code completion", "internal", 4_000, False, 300),
    Job("scan 40-page contract", "regulated", 45_000, True, 8_000),
    Job("summarise public docs", "public", 20_000, False, 5_000),
]

for j in JOBS:
    opts = eligible(j, TIERS)
    cheapest = min(opts, key=lambda n: TIERS[n]["cost_per_1k"]) if opts else None
    print(f"{j.name:26} class={j.data_class:10} -> {opts} -> pick {cheapest}")

# Bolt E4B 4GB targets wearables, dashboards and device assistants. Bolt 74B-A9B
# targets a workstation. Super VL targets the data centre. The split is the point:
# pick a tier per job, not per company.

Throughput per GPU is a billing metric. Latency per user is the experience. They diverge.

Local or cloud is a policy filter before it is a price comparison

Three tiers suggest a three-way decision, but which tier a specific job belongs to depends on three things rather than on price ordering. The first is data classification: once a regulated payload reaches a cloud endpoint it has left the building, which makes the local tier the only compliant option regardless of cost. The second is context length: the 4GB tier handles a far smaller prompt than the workstation tier, so a 40-page contract will not fit through it. The third is a latency budget: interactive completion needs a few hundred milliseconds while an overnight batch can afford seconds, and the same tier behaves very differently under those two budgets. Use those three as filters to produce the set of eligible tiers, then pick the cheapest one remaining. That ordering produces a different answer than starting from the cheapest option, which for regulated data is not merely more expensive but out of policy.

# Day-0 vLLM support makes the tuning surface explicit. Start with the knobs that matter.
SERVING_PLAN = {
    "model": "nemotron-3.5-super-vl",
    "tensor_parallel_size": 1,          # 12B active; scale out only when KV cache forces it
    "max_model_len": 32768,             # below the trained maximum unless you need the tail
    "gpu_memory_utilization": 0.90,     # leave headroom for fragmentation
    "max_num_seqs": 64,                 # aggregate throughput lever
    "enable_prefix_caching": True,      # repeated system prompts are the cheapest win
    "quantization": None,               # bf16 weights unless the target card is small
}

def estimate_kv(model_cfg, max_model_len, max_num_seqs):
    layers = model_cfg["layers"]
    kv_heads = model_cfg["kv_heads"]
    head_dim = model_cfg["head_dim"]
    bytes_per_value = 2
    per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
    return round(per_token * max_model_len * max_num_seqs / (1024 ** 3), 1)

MODEL = {"layers": 56, "kv_heads": 8, "head_dim": 128}

for seqs in (8, 32, 64, 128):
    gib = estimate_kv(MODEL, SERVING_PLAN["max_model_len"], seqs)
    print(f"max_num_seqs={seqs:>4} -> KV cache {gib:6.1f} GiB on top of weights")

print()
print("Sanity checks before you tune anything:")
for k, v in SERVING_PLAN.items():
    print(f"  {k:26} = {v}")

# Order of operations that holds up in practice: correctness at max_num_seqs=1,
# then prefix caching, then batching, then context length. Reversing it produces
# numbers you cannot attribute to any single change.

Four numbers to measure before you migrate, in order

Before switching a workload, measure four things: per-GPU throughput on the target hardware under your real input-length distribution; quality on your own task set rather than a public leaderboard; time to first token and inter-token speed; and cost per completed task. The second is the one most often skipped, and it is where open models bite, because a model can look excellent on public benchmarks and still be worse on your domain data. The tuning order on the serving side matters just as much. Establish correctness at concurrency one, then enable prefix caching where system prompts repeat, then raise batch size, and only then extend context length. Do it in the other order and you end up with a set of numbers you cannot attribute to any single change. Nvidia's launch pitch is fastest in class, but for your deployment, the four measurements above are the ones that decide, and none of them comes from a press release.