How Local Mixing Encodes Relative Position in Global NoPE Attention
The work shows that local mixing layers can implicitly encode positional information in attention mechanisms without explicit position encodings.
The work shows that local mixing layers can implicitly encode positional information in attention mechanisms without explicit position encodings.