Training Parallel Speculative Draft Models for LLM Inference
A method is proposed to train parallel and semi-autoregressive draft models by directly minimizing expected decoding rounds for speculative inference.
A method is proposed to train parallel and semi-autoregressive draft models by directly minimizing expected decoding rounds for speculative inference.