Reevaluating Self-Attention in Sequence Modeling: A Grassmann Flow Approach

This paper challenges the necessity of explicit self-attention in sequence modeling by introducing an attention-free architecture based on Grassmann flows, demonstrating competitive performance on language modeling and natural language inference tasks. The approach uses geometric features on a Grassmann manifold for information propagation, achieving 10-15% lower perplexity than Transformers while maintaining strong inference accuracy.

Comments

Read original article