Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests
A new prefetching and interleaving strategy accelerates mixture-of-experts model inference by preventing GPU stalls during expert fetching.
A new prefetching and interleaving strategy accelerates mixture-of-experts model inference by preventing GPU stalls during expert fetching.