some kind of quinn on Nostr: > These results suggest that, in the on-device regime, most of the benefits ...

> These results suggest that, in the on-device regime, most of the benefits attributed to recent hybrid SSM/linear-attention blocks can be captured by their short convolutional submodules plus a small number of global attention layers. We therefore carry forward designs that minimize global blocks while prioritizing inexpensive, gated short convolution blocks elsewhere. The released dense backbones (Section 2) and the sparse-FFN placement in LFM2-8B-A1B (Section 2) instantiate this recipe. Detailed speed and memory measurements appear in Section 2.4.

looks like the findings from liquid (and to a lesser extent cactus) AI companies is: transformer stacks don't win when cost is a metric. the stack instead convolves the soup and only spends attention on the last step.