Skip to content

[Spec Decode] Add draft-model distributions for probabilistic sampling - #1373

Draft
POWEHI2023 wants to merge 1 commit into
vllm-project:mainfrom
POWEHI2023:feat/lossless-speculative-sampling
Draft

POWEHI2023 wants to merge 1 commit into
vllm-project:mainfrom
POWEHI2023:feat/lossless-speculative-sampling

Conversation

@POWEHI2023

@POWEHI2023 POWEHI2023 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

The MLX draft-model proposer previously returned only greedy token IDs. Requests using a nonzero temperature or top-k/top-p sampling could not use draft-model speculation, so serving a model with its normal sampling settings fell back to target-only decoding. This draft PR works toward distribution-correct speculative sampling for that use case.

Design

  • Change MetalProposer.propose() to return (DraftTokenIds | None, dict[str, PendingDraftProbs] | None). Other proposers return None for the distribution result; this version implements probabilistic proposals only for DraftModelProposer.
  • Return logits from the draft model's ingest and subsequent draft steps. The greedy path still takes argmax. The probabilistic path applies the request's temperature, top-k, top-p, and min-p settings, samples each draft token, and retains the full processed distribution for every position. Temperature zero produces a point-mass distribution.
  • Initialize a per-request RNG in the model runner before proposal and pass it through ProposeContext. Proposal and verification consume successive draws from the same stream. The runner hands sampled draft IDs to the scheduler and retains the distributions for the next verification step.
  • Process the target model's verification logits with the same sampling controls. Accept draft token x with probability min(1, p(x)/q(x)); on rejection, sample from the normalized positive residual max(p-q, 0). If all drafts are accepted, sample a bonus token from the target distribution.
  • Keep requests with unsupported penalties, constraints, or sample logprobs on the target-only path. Validate any draft-bearing scheduler handoff before target model work.

Relation to #1278

#1278 samples each target verification row with sample_from_logits, using the request's output history plus earlier draft tokens, then compares the sampled target token with an argmax draft ID. This PR records the draft model's full proposal distribution and uses probability-ratio acceptance plus residual resampling. For a deterministic draft distribution, the two acceptance rules have the same distribution. If sampled draft IDs were simply fed into #1278's equality check, acceptance would be sum_x p(x)q(x) rather than the sum_x min(p(x),q(x)) of probability-ratio verification; when p=q, this PR can accept every draft.

Integration plan if #1278 merges

  • Reuse [Spec Decode] Verify drafts of sampled requests #1278's construction of per-row sampling metadata: each target row must see the committed output history plus only the draft prefix before that row. This is especially useful when later extending the supported controls to history-dependent penalties and token masks. Keep its native-greedy fast path for deterministic rows.
  • Share the logits-processing rules where possible, but expose normalized target probabilities to the probabilistic verifier. [Spec Decode] Verify drafts of sampled requests #1278's sample_from_logits returns sampled IDs, which are insufficient for the p/q acceptance test and residual distribution. Do not draw a target token for every proposal row in the probability-ratio path.
  • Keep [Spec Decode] Verify drafts of sampled requests #1278's sampled-target verify path for ID-only proposers. Route PendingDraftProbs from the draft-model proposer to verify_probabilistic, while preserving the common packed-row metadata and scheduler handoff checks in the runner.
  • Reconcile the shared can_draft eligibility and preflight checks by verifier capability: ID-only drafts may use [Spec Decode] Verify drafts of sampled requests #1278's sampled-target path; distribution-bearing drafts must have matching processed p and q. On rebase, resolve [Spec Decode] Verify drafts of sampled requests #1278's verify_greedy-to-verify rename without duplicating sampling or eligibility logic. Add parity tests for deterministic drafts and distribution/acceptance tests for stochastic drafts.

#1278 remains an open draft PR. This PR can proceed independently as a draft and will rebase on its final interface if it lands first.

Current draft status

  • The proposal, scheduler handoff, and probability verifier are wired together for supported sampling parameters in commit 77eae6a.
  • Known correctness issue for follow-up: preemption currently clears the per-request probability RNG while retaining request state. A seeded request recreates that RNG from its original seed after readmission and can replay draws, so its sampling sequence depends on preemption. Retain the RNG across preemption; clear it when the request finishes or its ID is reused.
  • This consolidated commit keeps only test changes required by the propose() interface. Probabilistic verification and RNG lifecycle regression tests will be added during the next development pass. Real-checkpoint baseline comparison, acceptance measurements, and B=1/B>1 serving benchmarks remain pending. No speedup claim is made.

Validation so far

  • pytest -q tests/test_spec_decode_metadata.py tests/test_draft_model_proposer.py tests/test_sampling.py: 180 passed on 77eae6a. These tests do not exercise the new probabilistic verifier.
  • scripts/lint.sh: shellcheck, Ruff lint and format, and mypy passed on 77eae6a.

Before marking ready

  • Fix the seeded RNG lifecycle across preemption and add a regression test.
  • Add focused regression tests for probabilistic prefill, mixed batches, zero-to-positive draft width transitions, and exact fixed-distribution verification.
  • Measure same-checkpoint acceptance and compare sampled output against target-only generation; then benchmark serving at B=1 and B>1 on representative workloads.

@github-actions github-actions Bot added the speculative-decoding Draft models and speculative decoding label Nov 18, 2026
@POWEHI2023
POWEHI2023 force-pushed the feat/lossless-speculative-sampling branch from 9d2f57c to 3aa3eea Compare November 19, 2026 10:17
Sample draft tokens from processed distributions and verify them against target probabilities with residual resampling. Preserve per-request draft distributions and RNG state across scheduler handoff. Return draft IDs and optional probabilities through one proposer interface.

Signed-off-by: POWEHI2023 <zxe6b885@163.com>
@POWEHI2023
POWEHI2023 force-pushed the feat/lossless-speculative-sampling branch from 25f952b to 77eae6a Compare November 19, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

speculative-decoding Draft models and speculative decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant