1.1B 的体量跑出 70B 的质量：Booking.com 那篇最佳论文到底做了什么

一、这套流水线先解决了什么问题

旅行平台每天要回答几千万次查询：AI Trip Planner 要在不到一秒内从对话里抽出偏好，Smart Filters 要把一句自然语言实时变成结构化的筛选条件，住宿描述生成要按百万级房源批量产出。这些场景的共同点是：模型质量够用，但每次调用的钱和延迟都不够用。业界的做法通常是二选一——上便宜的小模型、接受质量下降，或者上大模型、把账全付了。这篇论文找的是第三条路：先把体量压下来，再把每次前向计算能吐出的 token 数提上去。

二、为什么两个技巧是乘的，不是加的

标准自回归解码的瓶颈不在算力，在显存带宽：每生成一个 token 都要把整个模型的权重过一遍。知识蒸馏动的是「每一遍有多贵」——模型小了，单位步长的成本就低；Medusa 动的是「需要多少遍」——一次前向并行吐出多个候选 token，再统一校验，接受的保留、拒绝的丢弃。两个维度被同时压缩，收益就近似相乘。这也是这篇论文最有价值的工程洞见：不要问「哪个技巧更好」，先问「我在压哪个维度」。如果你的瓶颈是排队，压模型大小没用；如果你的瓶颈是权重读取，堆解码头也没用。

# Step 1: build the distillation corpus from unlabelled production text.
# The teacher's cost is a one-off; the student's cost is every request forever.
import json

def label_with_teacher(prompts, teacher, out_path="distill.jsonl"):
    kept = 0
    with open(out_path, "w") as fh:
        for p in prompts:
            r = teacher.generate(p, temperature=0.7, max_tokens=512)
            if not r.text.strip():
                continue  # an empty teacher answer teaches nothing
            fh.write(json.dumps({"prompt": p, "completion": r.text}) + "
")
            kept += 1
    print(f"teacher labelled {kept} examples")

# Keep this step boring and reproducible: same prompt set, same sampling
# settings, versioned output file. Your student inherits every mistake here.

三、四步实验，每一步的账都公开

第一轮，只在任务数据上微调 TinyLlama，结果已经明显强于 few-shot 的 GPT-4o——但仍然追不上 70B 老师。第二轮加知识蒸馏，用人工标注和老师标注混合训练，把和老师之间的质量差距补上了 90% 以上，推理同时快了约 6 倍。第三轮挂 Medusa 头，在 6 倍的基础上再快 2 到 3.6 倍。第四轮把两者合在一个流水线里训练，得到的最终模型质量追平老师，速度随用例不同落在 10 到 20 倍之间。注意这个 6 倍：它不是「小模型自动快 6 倍」，而是这个特定任务上、这个特定体量比例下的实测值。

# Step 2: Medusa heads. Base weights frozen; each head predicts a token
# further ahead. Medusa-2 trains base and heads in one stage, which is why the
# authors preferred it for teams already juggling several production models.
import torch.nn as nn

class MedusaHead(nn.Module):
    def __init__(self, hidden: int = 2048, vocab: int = 32000):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(hidden, hidden),
            nn.SiLU(),
            nn.Linear(hidden, vocab),
        )

def build_heads(base, k: int = 4):
    for p in base.parameters():
        p.requires_grad_(False)      # freeze the distilled student
    return nn.ModuleList([MedusaHead() for _ in range(k)])

# At position t, head i is trained against token t+i+1. At inference you run
# the base once, verify the drafted tokens, and keep the longest accepted
# prefix. Speculation is cheap; verification is the bill.

四、Medusa-1 和 Medusa-2 的差别，是运维差别

两种变体测出来的加速差不多，也都没有可感知的质量损失。Medusa-1 是先微调基座、冻结权重、再单独训练解码头；Medusa-2 把基座和解码头放在一个阶段里训练。作者选了 Medusa-2，理由不在分数，在流水线：单阶段训练更简单，而对同时维护多个模型的团队来说，「简单」本身就是可运维性。这条经验比任何一个倍数都更值得抄——你最终要维护的不是一篇论文，是十几条这样的小流水线。

# Step 3: speedup is not a constant, it is a function of acceptance rate.
# Measure it on your own traffic before you promise anyone "10x".
def tokens_per_pass(accept):
    """accept[i] = P(head i accepted | tokens 0..i-1 accepted)."""
    total, prod = 1.0, 1.0          # the base model always commits one token
    for a in accept:
        prod *= a
        total += prod
    return total

for rate in (0.9, 0.8, 0.7, 0.6):
    print(rate, round(tokens_per_pass([rate] * 4), 2))

# 4 heads at 0.9 acceptance ≈ 4.1 tokens per base forward pass.
# At 0.6 it collapses to ≈ 2.3. That gap is why the same technique gave 2x in
# one Booking.com feature and 3.6x in another — and it is an upper bound: the
# verification pass is not free.

五、三个功能的真实结果

Smart Filters 是延迟最敏感的：99% 的查询响应快了 15 倍。AI Trip Planner 的用户参与度相对 GPT-3.5 提升 2.9%。住宿描述生成在成本不到 GPT-4 三分之一的情况下，"有帮助"评价多了 1.4%。这三个数字放在一起看才有意思：前两个买的是延迟，第三个买的是成本。同一个流水线，三个不同的收益方向——所以复现之前你要先确定，你买的是哪一个。

# Step 4: grade the student on the task metric, not on perplexity.
# Perplexity moving tells you the student is close. It does not tell you the
# feature got better.
import statistics

def evaluate(model, cases, metric):
    scores = []
    for c in cases:
        out = model.generate(c["prompt"], max_tokens=c.get("max_tokens", 256))
        scores.append(metric(out, c["expected"]))
    scores.sort()
    return {
        "n": len(scores),
        "mean": round(statistics.mean(scores), 3),
        "p10": round(scores[len(scores) // 10], 3),
    }

# Booking.com went further than offline metrics: the numbers that decided the
# rollout came from live A/B tests — engagement for the trip planner, and the
# share of "helpful" votes on generated accommodation descriptions.

六、想抄的话，按这个顺序来

第一步先量基线：每个功能的 p50、p95、每千次请求成本、以及一个真正和业务挂钩的质量指标。第二步确认瓶颈维度：是显存带宽（模型太大）还是串行步数（token 太慢）。第三步才是做蒸馏，用未标注的线上文本让老师打标，这一步的钱是一次性的。第四步挂解码头，同时把接受率当成一等指标持续监控，因为加速倍数几乎完全由它决定。第五步用线上 A/B 而不是离线相似度决定是否放量。

# Step 5: cost per thousand requests, then the break-even.
def cost_per_1k(tokens_in, tokens_out, in_price, out_price, requests=1000):
    return requests * (tokens_in * in_price + tokens_out * out_price) / 1_000_000

teacher = cost_per_1k(600, 220, 2.50, 10.00)
student = cost_per_1k(600, 220, 0.05, 0.15)
print("teacher", round(teacher, 2), "vs student", round(student, 2), "per 1k requests")
print("ratio", round(teacher / student), "x cheaper")

# Then add the one-off: teacher labelling + GPU hours, divided by monthly
# request volume. If the break-even lands past the feature's expected life,
# the honest answer is that you should not build it yet.