26 万亿 token 训练、2-bit 权重：拆开英伟达 Nemotron 3.5 Super VL 与 Bolt

2026 年 10 月 15 日，英伟达同时发布了两条线的模型：Nemotron 3.5 Super VL 和 Nemotron 3.5 Bolt，官方说法是覆盖「从企业服务器到边缘设备与本地硬件」的全部 Agent 部署形态。Super VL 是开放的 120B 多模态混合专家模型，每 token 激活 12B 参数，与 Nemotron Coalition 合作开发，把推理、指令遵循、工具调用与文本/图像/文档/视频理解放进同一个模型；英伟达称它的能力接近自家最大的推理模型，但体积只有五分之一，并且「同级别中吞吐约为次优者的 5 倍」。Bolt 74B-A9B 用 2-bit 量化把 74B 权重压进 24GB，跑在工位或 PC 显卡上；本月晚些时候还会有一个 4GB 的 Bolt E4B 面向可穿戴与车载。真正值得算的是两件事：24GB 里到底还剩多少余量给 KV 缓存，以及「5 倍吞吐」和你的用户实际感受到的延迟是什么关系。

120B 的身板里塞一个 12B 的活跃档位，这就是全部诀窍。

这次发布的三条线，和「开放」到底开放到哪

这次发布不是一个模型，是一组分层：数据中心的 Super VL（120B）、工作站的 Bolt 74B-A9B（24GB）、以及即将到来、面向边缘设备的 Bolt E4B（4GB）。把「开放」拆开看更清楚：权重公开，所以组织可以自己后训练、自己部署、放进防火墙内，英伟达的说法是借此维持主权、隐私与安全。训练方法也是开放的——约 26 万亿 token 的预训练与继续预训练、380 万条后训练样本、100 个开放环境里的强化学习；配套在 Hugging Face 释放的数据集覆盖 Agent 工具调用、金融、指令遵循、长上下文、安全、公平性与视觉语言理解。这个组合的意义不在于「免费」，而在于组织可以把评测与合规证据握在自己手里：你能证明用的是哪份权重、跑在什么数据上、评测结果是什么。

# Fitting 74B into 24GB is a budget problem, not a magic trick. Do the arithmetic.
BYTES_PER_PARAM = {"fp16": 2.0, "fp8": 1.0, "int4": 0.5, "int2": 0.25}

def weights_gib(params_b, precision):
    return params_b * 1e9 * BYTES_PER_PARAM[precision] / (1024 ** 3)

CARD_GIB = 24.0
OVERHEAD_GIB = 2.0        # driver, CUDA context, framework

print("74B (Bolt 74B-A9B) on a 24GB card")
for p in ("fp16", "fp8", "int4", "int2"):
    w = weights_gib(74, p)
    room = CARD_GIB - OVERHEAD_GIB - w
    verdict = "fits" if room > 0 else "does not fit"
    print(f"  {p:5} weights {w:6.1f} GiB  headroom {room:+6.1f} GiB  {verdict}")

print()
print("Headroom after 2-bit weights must hold KV cache and activations.")
def kv_budget(room_gib, layers=48, kv_heads=4, head_dim=128, bytes_per_value=2):
    per_token = 2 * layers * kv_heads * head_dim * bytes_per_value / (1024 ** 3)
    return {"tokens_at_batch1": int(room_gib / per_token), "per_token_gib": round(per_token, 6)}

print(kv_budget(CARD_GIB - OVERHEAD_GIB - weights_gib(74, "int2")))

# Nvidia says 2-bit quantization is what makes the 24GB target work. This is why
# the headroom matters more than the headline: the weights are only the first bill.

「5 倍更快」是吞吐，不是延迟

英伟达的表述是「同级别中最快的开放 Agent 模型——比次优者快 5 倍」，依据是 Artificial Analysis 对同尺寸竞品的测试，InclusionAI 的 Ling 3.0 Flash 第二、DeepSeek 的 V4.1 Flash 第三。这类数字读的时候要分三层。第一层，吞吐（每秒 token 数）在批处理下会累加，所以「每卡吞吐」本质是账单指标；第二层，用户感受到的是首 token 延迟和出字速度，一个模型可以因为批量效率高而赢下总吞吐，却在单路延迟上输；第三层，吞吐与质量无关——同级别比较只保证「同尺寸里跑得快」，不保证「在你这个任务上答得对」。把这三层分开，你才能判断这条 5 倍对你的场景意味着什么：如果任务是大批量的文档与日志处理，吞吐直接换算成成本；如果是交互式编码补全，首 token 延迟才是命门。

# "5x faster" is throughput. Your users experience latency. Those are not the same.
def throughput(tokens_per_sec, batch):
    return tokens_per_sec * batch

def cost_per_million(tokens_per_sec, batch, gpu_hour_usd):
    per_hour = throughput(tokens_per_sec, batch) * 3600
    return round(gpu_hour_usd / per_hour * 1_000_000, 4) if per_hour else None

CANDIDATES = [
    {"name": "very large dense", "tok_s": 260, "batch": 8},
    {"name": "mid dense", "tok_s": 700, "batch": 8},
    {"name": "12B-active MoE", "tok_s": 1300, "batch": 8},
]

print(f"{'model':18} {'tok/s':>7} {'batch':>6} {'agg tok/s':>10} {'$/1M tok @ $3.20/h':>20}")
for c in CANDIDATES:
    agg = throughput(c["tok_s"], c["batch"])
    print(f"{c['name']:18} {c['tok_s']:7} {c['batch']:6} {agg:10} "
          f"{cost_per_million(c['tok_s'], c['batch'], 3.20):>20}")

# Single-stream latency is a different table. A model can win on aggregate
# throughput because batching is efficient while losing on time-to-first-token.
def ttft(prompt_tokens, prefill_tok_s):
    return round(prompt_tokens / prefill_tok_s * 1000, 1)

print()
for p in (2000, 32000, 128000):
    print(f"prompt {p:>7} tokens -> TTFT {ttft(p, 6000):>8} ms (at 6000 prefill tok/s)")

# Nvidia's 5x claim is a same-class comparison, tested by Artificial Analysis
# against similarly sized competitors. Read it as throughput, not as quality.

同一份权重，既要跑在数据中心节点，也要跑在工位上——这是一条新的分数线。

24GB 里装 74B，先算余量再谈别的

2-bit 量化把 74B 参数压到约 17.2GiB（74B × 0.25 字节），在一张 24GB 卡上扣掉约 2GiB 的运行时开销后，还能剩 4GB 出头——这些余量要装 KV 缓存、激活值与运行时缓冲，不是留给权重的。同一条账放在别的精度上完全不成立：FP8 约 68.9GiB、FP16 约 137.7GiB，都远超单卡。这也解释了为什么这个发布值得关注——它不是「又一个量化版本」，而是把「激活参数小 + 权重低位宽」两条路一起走，才让 24GB 这个门槛成立。选型时该问的第一个问题因此不是「模型多聪明」，而是「在目标卡上，权重之外还剩多少余量，够撑多长的上下文和多大的并发」。

# Open weights plus open recipes only pay off if you gate the post-training run.
from dataclasses import dataclass, field

@dataclass
class EvalCase:
    name: str
    category: str          # tool_use | finance | instruction | long_context | safety | vision
    passed: bool

@dataclass
class PostTrainGate:
    baseline: dict = field(default_factory=dict)
    max_allowed_regression: float = 0.02

    def score(self, cases, category):
        subset = [c for c in cases if c.category == category]
        if not subset:
            return None
        return sum(c.passed for c in subset) / len(subset)

    def check(self, before, after, categories):
        report = []
        for cat in categories:
            b, a = self.score(before, cat), self.score(after, cat)
            if b is None or a is None:
                continue
            delta = a - b
            blocked = delta < -self.max_allowed_regression
            report.append({"category": cat, "before": round(b, 3), "after": round(a, 3),
                           "delta": round(delta, 3), "blocked": blocked})
        return report

BEFORE = [EvalCase(f"b{i}", c, True) for i, c in enumerate(
    ["tool_use"] * 20 + ["finance"] * 10 + ["safety"] * 10 + ["long_context"] * 10)]
AFTER = ([EvalCase(f"a{i}", "tool_use", i < 19) for i in range(20)] +
         [EvalCase(f"f{i}", "finance", i < 10) for i in range(10)] +
         [EvalCase(f"s{i}", "safety", i < 8) for i in range(10)] +
         [EvalCase(f"l{i}", "long_context", i < 10) for i in range(10)])

gate = PostTrainGate()
for row in gate.check(BEFORE, AFTER, ["tool_use", "finance", "safety", "long_context"]):
    flag = "BLOCK" if row["blocked"] else "ok"
    print(f"{row['category']:14} {row['before']:.3f} -> {row['after']:.3f} ({row['delta']:+.3f}) {flag}")

# Nvidia publishes the ingredients (about 26T tokens, 3.8M post-training samples,
# RL across 100 open environments) plus datasets on Hugging Face. The gate above is
# what stops a fine-tune from quietly trading safety for tool-use accuracy.

后训练要用评测门禁，不然是在赌

开放权重加开放配方最大的价值是能拿自己的数据后训练（官方提到可用 NVIDIA NeMo 用自己的数据、工具、策略与工作流做后训练），但这也把风险从供应商转移到了你身上：一次针对工具调用准确率的微调，很可能让安全行为静默退化。所以流程里必须有一道评测门禁：改动前后跑同一套分类评测（工具调用、金融、安全、长上下文），逐类比较，任何一类跌幅超过阈值就阻断合入。上面那段代码就是这道门禁的骨架。注意阈值要按类别设置——长上下文掉 2 个点和安全掉 2 个点不是同一件事，后者通常应该是零容忍。

# Local or cloud is a policy question before it is a cost question.
from dataclasses import dataclass

@dataclass
class Job:
    name: str
    data_class: str        # public | internal | regulated
    prompt_tokens: int
    needs_vision: bool
    latency_budget_ms: int

TIERS = {
    "edge_4gb":   {"max_prompt": 8_000,    "vision": False, "cost_per_1k": 0.00002},
    "workstation_24gb": {"max_prompt": 60_000, "vision": True,  "cost_per_1k": 0.00012},
    "datacenter": {"max_prompt": 256_000,  "vision": True,  "cost_per_1k": 0.00040},
}

def eligible(job, tiers):
    out = []
    for name, t in tiers.items():
        if job.prompt_tokens > t["max_prompt"]:
            continue
        if job.needs_vision and not t["vision"]:
            continue
        if job.data_class == "regulated" and name == "datacenter":
            # a regulated payload leaves the building; only allowed with an explicit waiver
            continue
        out.append(name)
    return out

JOBS = [
    Job("inline code completion", "internal", 4_000, False, 300),
    Job("scan 40-page contract", "regulated", 45_000, True, 8_000),
    Job("summarise public docs", "public", 20_000, False, 5_000),
]

for j in JOBS:
    opts = eligible(j, TIERS)
    cheapest = min(opts, key=lambda n: TIERS[n]["cost_per_1k"]) if opts else None
    print(f"{j.name:26} class={j.data_class:10} -> {opts} -> pick {cheapest}")

# Bolt E4B 4GB targets wearables, dashboards and device assistants. Bolt 74B-A9B
# targets a workstation. Super VL targets the data centre. The split is the point:
# pick a tier per job, not per company.

每卡吞吐是账单指标，每用户延迟是体验指标，两者并不一致。

本地还是云端：先过策略，再比价格

三档模型对应三分法，但对某个具体任务该选哪一档，取决于三件事而不是价格排序。第一是数据分级：受监管的数据一旦发到云端就已经出界，此时唯一合规的选项是本地档，哪怕贵一点。第二是上下文长度：4GB 档能处理的提示词上限远低于工作站档，一份 40 页的合同扫不进小模型。第三是延迟预算：交互式补全要求几百毫秒，而离线批处理可以等几秒甚至几分钟，同样一档模型在两个预算下表现完全不同。把这三点当成过滤器先筛出可选档位，再在剩下的里挑最便宜的，得到的结论通常和「直接挑最便宜的」不一样——后者在受监管数据上直接违规。

# Day-0 vLLM support makes the tuning surface explicit. Start with the knobs that matter.
SERVING_PLAN = {
    "model": "nemotron-3.5-super-vl",
    "tensor_parallel_size": 1,          # 12B active; scale out only when KV cache forces it
    "max_model_len": 32768,             # below the trained maximum unless you need the tail
    "gpu_memory_utilization": 0.90,     # leave headroom for fragmentation
    "max_num_seqs": 64,                 # aggregate throughput lever
    "enable_prefix_caching": True,      # repeated system prompts are the cheapest win
    "quantization": None,               # bf16 weights unless the target card is small
}

def estimate_kv(model_cfg, max_model_len, max_num_seqs):
    layers = model_cfg["layers"]
    kv_heads = model_cfg["kv_heads"]
    head_dim = model_cfg["head_dim"]
    bytes_per_value = 2
    per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
    return round(per_token * max_model_len * max_num_seqs / (1024 ** 3), 1)

MODEL = {"layers": 56, "kv_heads": 8, "head_dim": 128}

for seqs in (8, 32, 64, 128):
    gib = estimate_kv(MODEL, SERVING_PLAN["max_model_len"], seqs)
    print(f"max_num_seqs={seqs:>4} -> KV cache {gib:6.1f} GiB on top of weights")

print()
print("Sanity checks before you tune anything:")
for k, v in SERVING_PLAN.items():
    print(f"  {k:26} = {v}")

# Order of operations that holds up in practice: correctness at max_num_seqs=1,
# then prefix caching, then batching, then context length. Reversing it produces
# numbers you cannot attribute to any single change.

迁移前要量四个数，而且有先后顺序

决定切换模型之前，先量四个数：在目标硬件上、你的真实输入长度分布下的每卡吞吐；你实际任务集上的质量（不要用公开基准代替自己的评测集）；首 token 延迟与出字速度；以及单任务成本。四个数里最容易被忽略的是第二个——开放模型的一个常见陷阱是公开基准漂亮、在你的领域数据上反而更差。部署侧的调参顺序同样重要：先在并发为 1 的情况下确认正确性，再打开前缀缓存（系统提示词重复率高的场景收益最大），然后才调批大小，最后才延长上下文。反过来做，你得到的是几个无法归因的数字。英伟达这次的发布把「同级别最快」当作卖点，但对你来说，真正决定成败的是这四个数里没有一个来自新闻稿。