Pretraining Scale Reverses Muon-AdamW Fine-Tuning Gap
A study on optimizer performance shows that AdamW outperforms Muon in downstream fine-tuning tasks when pretraining data budgets are large.
A study on optimizer performance shows that AdamW outperforms Muon in downstream fine-tuning tasks when pretraining data budgets are large.