Copy the Same, Distill the Difference: Initializing Linear ViTs
Researchers propose a method to efficiently and effectively initialize linear Vision Transformers to replace Softmax attention without extensive from-scrat
Researchers propose a method to efficiently and effectively initialize linear Vision Transformers to replace Softmax attention without extensive from-scrat