Speculative Decoding Engineering: Accelerating LLM Token Generation Speeds via Parallel Small Draft Models

Nyckelpunkter

• Understand the bottleneck. Generation is sequential and each token requires a full forward pass; speculation parallelizes verification.
• Choose a draft model. It must share the tokenizer and be significantly faster than the target model.
• Set the speculation length. More proposed tokens increase throughput but raise rejection rates; tune per workload.
• Verify in parallel. The target model scores all proposed tokens in one forward pass and accepts the longest prefix.
• Handle rejection gracefully. When a token is rejected, fall back to the target model's prediction and continue.
• Measure acceptance rate. A low acceptance rate means the draft model is too weak or the domain is mismatched.
• Use temperature carefully. Speculation assumes greedy or low-temperature decoding; high temperature reduces acceptance.
• Profile end to end. Measure tokens per second and latency, not just acceptance rate, to confirm real gains.

Architecture

The draft model generates k tokens autoregressively. The target model then evaluates all k tokens plus the last accepted token in a single forward pass. Accepted tokens are emitted, and the first rejected position is corrected by the target model. The process repeats from the new position.

Production configuration

def speculative_decode(prompt, draft, target, k=4, max_tokens=128):
    tokens = tokenize(prompt)
    while len(tokens) < max_tokens:
        # draft proposes k tokens
        draft_tokens = []
        for _ in range(k):
            t = draft.predict(tokens + draft_tokens)
            draft_tokens.append(t)
        # target verifies in parallel
        logits = target.forward(tokens + draft_tokens)
        accepted = 0
        for i, dt in enumerate(draft_tokens):
            if dt == argmax(logits[i]):
                accepted += 1
            else:
                break
        tokens.extend(draft_tokens[:accepted])
        if accepted < k:
            tokens.append(argmax(logits[accepted]))
    return detokenize(tokens)

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70B \
  --speculative-model meta-llama/Llama-3-8B \
  --num-speculative-tokens 5 \
  --use-v2-block-manager

# benchmark
curl http://localhost:8000/v1/completions -d '{"prompt":"Explain TCP","max_tokens":100}'

import time
start = time.perf_counter()
out = speculative_decode(prompt, draft, target, k=5)
elapsed = time.perf_counter() - start
print(f"tokens/sec: {len(out)/elapsed:.1f}")
print(f"acceptance rate: {accepted_count/total_proposed:.2%}")

Edge cases from production

• Mismatched tokenizers between draft and target break speculation silently; verify tokenizer equality first.
• Very long prompts reduce the benefit because verification cost grows with context length.
• Draft models that are too small have low acceptance and can slow things down.
• Batching changes the memory profile; test with your actual batch size.

Vanliga frågor