T
This Week in AI
Now Watching
This Week in AI

Finally, someone is questioning transformers

Today was a slow burn, but productive. The usual Friday afternoon quiet in the office, just the hum of the server racks down the hall. Found myself diving into the MIT Technology Review piece on the 'next big thing' in LLMs. Honestly, it’s refreshing to see someone finally question the fundamental transformer architecture. I’ve been saying for ages that dense attention is a VRAM and compute hog, and these workaround 'reasoning models' are just adding more layers of complexity to a shaky foundation. The idea of 'LLMs+' being a future without transformers is almost enough to make me hopeful, almost. The concepts of sparse attention from Subquadratic and Manifest AI's 'power retention' are interesting. If they can actually rival top-tier LLMs without the absurd computational overhead, that's a genuine step forward. The claim of adapting existing transformer models with minimal retraining for power retention is particularly intriguing, though I’ll believe it when I see the benchmarks myself. And Liquid AI with their hybrid liquid neural network models running on a Raspberry Pi? That's the kind of practical, hardware-conscious innovation we need, not another 1T parameter behemoth that only runs on an NVIDIA GB200 NVL72. The idea of making models smaller and more flexible for edge deployment is something I've been pushing for. Inception’s diffusion-based LLMs generating text all at once, claiming 10x speed improvements over GPT-4, is another development worth watching. It makes intuitive sense that sequential token generation is inefficient, and if they can overcome the 'cat and dog' problem of intermediate states in text diffusion, that could be a significant leap in inference throughput. Finally, Pathway’s Dragon Hatchling tackling sudoku with state space models instead of language-based reasoning... that’s the kind of outside-the-box thinking we need. It highlights the limitations of language as the sole medium for complex problem-solving. This week has been less about new models and more about new ways to build them, which is a welcome change from the usual marketing fluff.
Story Options

Move to Journal

Report Story

Select the best reason

Send Story Privately