Beyond LLMs: A Post

The rapid ascent of large language models (LLMs)—and their growing role in everyday life—masks a fundamental problem: Generative Pre-trained Transformer (GPT) models hallucinate, struggle with memory-intensive tasks, consume enormous compute and energy resources, and at times behave unpredictably. Researchers have a name for the resulting unevenness: jagged intelligence. Systems solve graduate-level problems and then stumble over something a child would get right. What’s more, failures often arrive without any warning. This makes LLMs valuable for many tasks, but it renders them unreliable for predicting basic physical actions, such as how a ball bounces or a cat jumps. A popular test makes the point: “I put a coin inside a coffee mug. I carried the mug into the kitchen and turned it upside down on the counter. Then I walked back to the living room with the mug. Where is the coin?” Until recently, models routinely answered that the coin was in the living room, still inside the mug. They couldn’t simulate the coin falling out. Today’s leading models answer it correctly—but a correct answer to a widely circulated puzzle is not the same as intrinsic knowledge, or even a basic understanding of physics. Vary the puzzle slightly and the failure returns. As a result, researchers are exploring post-transformera AI frameworks. Rather than simply building bigger LLMs, the idea is to infuse models with a better understanding of spatial relationships, improve their memory, and incorporate new types of reasoning. Concepts include “world models”b that simulate how people, objects, and environments behave; agentic architectures that break down problems into smaller components and use multi-step reasoning; selective State Space Modelsc capable of long-sequence modeling with linear scaling; and biologically inspired neural memory modulesd that operate more like the human brain. Said Yann LeCun,e Jacob T. Schwartz Professor of Computer Science at New York University and a 2018 ACM Turing Award laureate: “We are never going to achieve human-level intelligence in AI simply by training text. The world is more complicated than language.” Model Behaviors If the last decade of AI has focused heavily on teaching computers to master language, the next decade might focus on teaching AI how to master reality and understand how things work in the physical world. Although large language models aren’t likely to disappear, the AI universe is likely to become more diverse. “Today’s LLM models are a bit more complex than similar neural networks from 1991, but not by a lot,” said Jürgen Schmidhuber,f co-chair of the GenAI Center at KAUST, Scientific Director at Swiss AI lab IDSIA and a pioneerg in the field of Recurrent Neural Networks (RNNs). “Passing the ‘Turing Test’h is much easier than [developing] true AI in the physical world. Predicting the next token by itself is not enough to produce artificial general intelligence (AGI)i,” he added. Much of this trouble traces back to memory, said Phillip Isola,j associate professor of electrical engineering and computer science at Massachusetts Institute of Technology. “Transformers don’t natively have memory in the sense of how we think about it in humans,” he said. The problem with LLMs is rooted in the way they process information. As their context window shifts forward, they overwrite tokens naively, he added. This causes errors and hallucinations, but it also contributes to models losing their train of thought as they tackle complex tasks. While the ability to tamp down hallucinations has improved thanks to guardrails, augmented retrieval, and reasoning-oriented training, the underlying LLM engine hasn’t evolved. Models still generate plausible, even convincing output—sometimes right and sometimes wrong—without any understanding of how the world actually works. They are far less likely to invent a citation, a statistic, or even a news event, but they still can’t reason about what would happen if a ball rolled into the street in front of an autonomous vehicle. Is it only a ball? Is there a child chasing it? Transformer models should ideally serve as just one layer in the overall AI stack, Isola said. Yet, more layers increase the odds that something can go astray. While LLMs appear fluent, they often struggle with facts, tracking goals, long-term planning, and adapting to changes. “Transformer models have many limitations, including a lack of understanding of spatial relationships that are a critical part of our world,” added Aditi Raghunathan,k Assistant Professor in the Computer Science Department at Carnegie Mellon University. Seeking Transformation Impressive language skills notwithstanding, transformer-based models function more like enormous databases than sources of intelligence, Schmidhuber said. Transformer models excel at “fuzzy” tasks like writing or brainstorming ideas, but they are ill-suited for scientific calculations or exact, step-by-step reasoning, such as reconciling financial records. The gap is not confined to the physical world; it shows up in formal reasoning too. “There are all kinds of simple problems that standard transformers cannot solve. An example is their ability to classify bit strings of arbitrary size according to whether the number of 1-bits is odd or even.” Unfortunately, tossing more resources at the problem—scaling up models—won’t fix the underlying problem, he noted. Since 1990, Schmidhuber’s research has centered on developing AI agents that learn a world model from data collected through “self-invented experiments” that identify regularities and patterns in the data. A controller uses simulations inside the world model to select action sequences that maximize long-term predicted rewards. “Goals change all the time because whatever the system learns becomes boring, and so it must continually invent new goals,” he said. “This is essentially how babies and scientists learn—not by downloading the web but through self-invented experiments.” LeCun is taking a different approach. His Joint Embedding Predictive Architecture (JEPA)l doesn’t generate text or pixels directly; it uses images, video, and other signals to learn and predict abstract representations of future states through self-supervised learning. This JEPA-based world model resides inside an agent that has perception, memory, an actor, and a cost module. The agent uses JEPA’s world model to simulate the consequences of actions and select those that appear most promising. LeCun’s goal is to escape the fundamental limitations of transformer models, including LLMs. “If you use a transformer model to predict future video frames, the results become a blurry mess because it has no idea what is actually taking place. As the system attempts to calculate an ‘average,’ it comes up with essentially nothing,” he explained. Unfortunately, using a probability distribution model to solve the challenge becomes an “intractable mathematical problem” and plugging in a diffusion model “doesn’t get us any closer to truly understanding reality.” JEPA uses sensor data from cameras, microphones, and an array of industrial or scientific instruments to identify the variables that actually matter and forecast how the scene will unfold. Its applications could range from robotics and aerospace to smart power grids and autonomous driving. In November 2025, LeCun announced that he was leaving his position as Chief AI Scientist at Meta and launched a startupm to fully develop the JEPA framework. “The idea that we can solve complex AI problems purely from language is ridiculous. It will never work,” he said. What makes a world model so powerful is that it can play out a situation before generating a response. For example, in the coin and the mug scenario, an agent doesn’t use language about these objects to produce a likely answer. It actually simulates the mug tipping, the coin leaving the mug, and the coin coming to rest on the counter before reporting where it ends up. Swap in a jar, add a second coin, or set the mug down twice along the way, and the simulation continues to track the objects. Likewise, a world model can project forward a few seconds and weigh the likelihood of a child following a ball into the street—along with the cost of a wrong decision. In Search of Better Reasoning Other post-transformer approaches are taking shape. For instance, researchers are exploring State Space Models (SSMs), such as Mamba,n which maintain a compact running “state” that’s updated as new words or data arrive rather than constantly comparing every single word. They can process longer sequences—and at times use less compute power and memory—than current transformer models. This running state makes them an attractive option for physical prediction: a system tracking a flight trajectory or a weather front can carry forward past events rather than re-collecting and re-weighing every observation from scratch. SSMs could help AI better predict complex trajectories, weather patterns, and other dynamic events. Google DeepMind is developing Titans,o a “neural memory” module that continues to learn and remember post-training. DeepMind researchers have also developed a technique called Nested Learning,p which reorients internal workflows to avoid catastrophic forgetting.q Meanwhile, AI21 Labs has developed Jamba,r a model that combines transformer elements with Mamba-style state spaces, while another group of researchers has created a framework called Retentive Network,s which takes aim at parallel training and recurrent low-cost inferencing. For now, a post-transformer world is still taking shape—and it isn’t clear when it will fully arrive or what form it will take. Some researchers, such as CMU’s Raghunathan, expect the space to advance gradually. She believes that model designers will mix and match components to achieve desired performance benchmarks, build more flexible frameworks, and produce more grounded results. This will lead to “creative and open-ended problem solving” as well as “improved quality and reliability,” she said. Concluded Isola: “We can do a lot of things with transformer models—and there is likely a lot of room to make them perform better—but there are also inherent limitations in these models. We must find ways to build a more robust AI stack.” Join the Discussion (0) Become a Member or Sign In to Post a Comment