Self-Attention Slimdown: From O(N²) to O(N·M), But Is It Worth It?
Interesting take on reducing self-attention complexity from O(N^2) to O(N*M) although results don’t look eye popping compared to SOTA like DeiTs. https://x.com/_akhaliq/status/1362221635571433481