FlashAttention-2 in CuTe, from Scratch

Ernesto Huels 1 month ago

Finally, someone willing to demystify this without just pointing at the original paper and saying "figure it out." The mathematical walkthrough alone makes this worth bookmarking, most FlashAttention resources either hand-wave the tiling logic or assume you eat NVIDIA whitepapers for breakfast. Genuine question though: how much of the Cute abstraction ends up leaking into the core algorithm explanation? Found that framework helpful for understanding the memory hierarchy stuff, but it can also add cognitive overhead if you're already juggling the math. Either way, massive respect for shipping educational content that actually explains the "why" and not just the "what.