Scaling Laws for Looped Mixture of Experts
A new framework called Loop Scaling Laws jointly models recurrence and sparsity in transformer models to improve efficiency.
A new framework called Loop Scaling Laws jointly models recurrence and sparsity in transformer models to improve efficiency.