Generative AI models have shown incredible capabilities, from composing poetry to generating functional computer programs. These models, such as large language models (LLMs), are trained primarily to predict the next word in a given text sequence.
As a result of their impressive outputs, many believe that these models may be learning some deeper understanding of general truths about the world. However, a new study challenges this assumption, revealing that while these AI models can perform tasks like providing accurate driving directions in New York City, they don’t necessarily have a coherent internal map of the city.
Unveiling the AI’s Limitations
The research showed that even though the AI could navigate New York with near-perfect accuracy, its performance significantly dropped when streets were closed, or detours were added. This raises concerns about the adaptability of such models in real-world applications where environments can change unexpectedly.
When the researchers examined the internal maps generated by the AI, they discovered that the model had fabricated many non-existent streets, with strange curves and connections between distant intersections. This highlights the issue that, while the AI can seem to perform well, it does not necessarily understand the underlying rules or structure of the tasks it’s executing.
Potential Risks for Real-World Applications
This finding has significant implications, particularly for the deployment of generative AI models in real-world environments. A model that appears to perform well under controlled conditions might fail when faced with unexpected changes or challenges. The study underscores the importance of evaluating whether LLMs are building accurate world models, especially when considering their use in other scientific fields.
“One hope is that, because LLMs can accomplish all these amazing things in language, maybe we could use these same tools in other parts of science as well,” says Ashesh Rambachan, assistant professor of economics at MIT. “But the question of whether LLMs are learning coherent world models is very important if we want to use these techniques to make new discoveries.”
New Metrics to Evaluate AI World Models
The research team developed two new metrics to better understand whether transformers, the backbone of LLMs, can form coherent world models. These metrics, sequence distinction and sequence compression, were designed to evaluate how well the AI distinguishes between different states and how accurately it compresses information about identical states.
They tested these metrics on two specific tasks: navigating New York City streets and playing the board game Othello. Surprisingly, the models trained using random sequences rather than strategic data performed better at forming coherent world models, possibly because they encountered a broader range of possibilities during training.
Transformers Can Perform Without True Understanding
Even though the AI models could generate accurate directions and valid game moves, the two metrics revealed that many models did not have a coherent grasp of the tasks they were executing. When researchers introduced detours in the New York City navigation task, the AI’s accuracy dropped dramatically—from near-perfect to just 67% with only 1% of streets closed.
The maps created internally by the AI were chaotic, with non-existent streets and impossible intersections. This suggests that, although transformers can perform complex tasks, they do so without truly understanding the rules or structures behind them.
Looking Ahead: Improving AI World Models
For generative AI models to be reliable in more unpredictable environments, researchers need to rethink the way these models are trained. The current approach may work well for specific tasks, but when the task changes slightly, the models can fail.
In the future, the team aims to apply these new evaluation techniques to a broader range of problems, including those where the rules are only partially known. This could lead to the development of more robust AI systems that better understand the world around them.
This research offers a critical look at the limitations of AI and transformers, a topic also explored in the article “Graph-based AI Model Paves the Way for Future Innovation”, where AI models’ structures and their ability to innovate are discussed further.
This study was supported by several organizations, including the Harvard Data Science Initiative and the National Science Foundation.