Why Is Speculative Decoding Fast?
Speculative decoding speeds up language model generation by having a smaller draft model propose several tokens ahead, which a larger model then validates in a single pass. This parallel prediction reduces the number of expensive full model evaluations per token, cutting latency and increasing throughput without sacrificing output quality. The method leverages cheap computation to accelerate inference.