The Kepler Trap: When AI Mimics Understanding

🔊 Listen to the Bit & Being podcast on this topic. 🔊

In the early 1600s, Johannes Kepler mapped the heavens with astonishing precision. Night after night, he charted the planets’ dance around the sun, nailing the what and when of their motion. But ask Kepler why the planets move as they do, and he couldn’t tell you. It took Isaac Newton’s leap of insight into gravity to reveal the deeper truth. Today’s most advanced AI models are a lot like Kepler: brilliant at predicting patterns, yet often clueless about the principles behind them. They can plot the next word in a sentence or the next billion dollars in market trends, but when pressed on why those patterns occur, these systems shrug in digital perplexity. It’s as if we’ve built a superpowered oracle that can guess the plot twists without understanding the story. And in the boardrooms and innovation labs where results are king, this gap between prediction and understanding is the silent specter haunting AI’s rise, - a high-tech “Kepler syndrome” hiding in plain sight.

It wasn’t supposed to be this way. The whole promise of “foundation models”, - those giant AI systems trained on everything from literature to physics, was that if they predicted enough data, they’d eventually stumble into understanding. Feed an AI enough planetary orbits, the thinking went, and it might rediscover gravity on its own, - just as Kepler’s precise tables eventually set the stage for Newton (arXiv). This premise has driven a massive investment in AI. Tech leaders bet that sheer scale and pattern-recognition would yield deeper insights. For a while, the evidence seemed encouraging. After all, today’s AI can finish our sentences, label our photos, even draft passable business plans. In the field, we casually say these models “know” things. We anthropomorphize them as if a solid understanding lurks beneath their predictive text. But recently, researchers from Harvard and MIT decided to actually test that assumption, and what they found is raising eyebrows across the AI world. They developed something called an “inductive bias probe”, - essentially a clever experiment designed as an X-ray for a model’s mind (arXiv). Instead of just checking if an AI can predict correctly, this probe checks how the AI adapts when thrown into a new problem crafted from a known set of rules, - a “world model”. The question was simple. Does the AI’s learning bias line up with the true rules of the world, or is it just doing curve-fitting on the surface? The answer, in case your coffee isn’t already shaking: even our best AIs often cheat the test. As one summary of the study dryly noted, these models “excel at prediction” but “often learn task-specific heuristics or coarse representations rather than fundamental governing principles”. In other words, they’re pattern addicts, not real theorists (alphaXiv).

One striking experiment from the Harvard/MIT team drives the point home. They trained a model on thousands of simulated orbital trajectories, - planets circling a star, and indeed the AI became a virtuoso at predicting those orbits. Given any new planet configuration, it could forecast the path as accurately as Kepler’s tables. But then came the twist. The researchers gave this same model a new task in physics, something that required understanding gravity’s effects in a slightly different scenario. The result? The model flopped. It consistently failed to apply Newtonian mechanics to the new problem (arXiv). In essence, the AI had never truly learned why the planets move. It had just learned how to mimic the specific patterns it saw. It was as if a student aced the practice test by memorizing answers, only to fail the exam when questions were flipped. The researchers observed the model behaving as if it relied on “task-specific tricks” that collapsed outside the narrow context it was trained on. This is shortcut learning in action, - finding the cheap, narrow path to a correct answer, like a mathematical Clever Hans, …the horse. Our models, much like Hans, often latch onto incidental cues rather than the true substance. They’ll get the right answer, yes, but sometimes for all the wrong reasons.

This isn’t just an academic concern, - the real world has already seen what happens when AI “cheats” itself with shortcuts. Consider medicine, where AI models have been developed to read images like X-rays and retinal scans. In one case, an AI in an emergency room accurately flagged a patient as having pneumonia, and another system correctly identified signs of diabetes from an eye scan. Impressive results, right? The catch is that both predictions were correct by coincidence. The pneumonia flagger had essentially learned that anyone in the ICU probably has pneumonia, not something discernible in the X-ray itself. The eye-scan AI? It was keying off the patient’s age since older patients have higher diabetes risk, not truly detecting disease in the eye (siam.org). When doctors forced the model to ignore these telltale shortcuts, - controlling for ICU status or age, the magic evaporated. The AI went from “expert diagnostician” back to random guessing. As one report summed it up, “Both conclusions turned out to be correct, but for entirely coincidental reasons… the computer model took a shortcut by drawing an inference based on incidental details rather than actual diagnostic factors”. In plain terms, the AI wasn’t Dr. House, but a lucky statistician instead. Such shortcut learning is more than a quirky error, - in domains like healthcare it can be literally a matter of life and death. An algorithm that only seems to understand a disease while really chasing proxies could lead clinicians astray, recommending treatments based on phantom insights. The same goes for other high-stakes fields. An AI that evaluates loan risk might simply be redlining by another name if it picks up on zip codes as a proxy for creditworthiness. In finance, a trading model might dazzle on historical data, only to faceplant when market conditions shift, - precisely because it was leveraging correlations that held yesterday but not tomorrow. Wall Street quants have a saying for this: “Correlation is not causation,” a warning well-known long before AI, and now more pertinent than ever (https://alphaarchitect.com). The risk is that these models can lure us into complacency with spectacular predictive runs, all while a fundamental flaw lurks under the surface.

Why do these shortcuts and pseudo-understandings keep happening? In part, it’s a legacy of how we train AI. The algorithms are optimized to predict correctly, not to explain or generalize deeply. They will gladly take a statistical shortcut if it boosts accuracy on training data. In machine learning lingo, this is essentially “overfitting”, - when a model is so tuned to its training patterns that it misses the broader picture. We’ve seen overfitting in AI for years, - picture a model that learns the exact noise in your dataset as if it were signal, but now it’s appearing in exotic new forms. Today’s large language models operate by gobbling vast swaths of the internet and predicting the next word. They are incredibly good mimics. Yet even these linguistic wizards show signs of the same syndrome. They can lose the plot in a conversation by clinging to phrasing from their training data, or confidently generate nonsense when asked something outside their narrow familiarity, - a telltale sign that they’re still “stochastic parrots,” repeating data patterns without a true grasp of meaning. They might write a convincing paragraph on, say, economic policy, but slip in a subtle factual error or an absurd assumption, revealing that the model doesn’t actually know what it’s talking about. As AI researcher Samer Brahim dryly put it, “It will give you an answer, but not necessarily a correct one”. This is a trait that’s led some to dub AI “a lying machine,” when it bluffs with incorrect outputs. All of this underscores a crucial point: raw intelligence isn’t the same as tested wisdom. Without innate world models*, -* an internal compass of how things really work, AI will keep gorging on shortcuts like a kid eating dessert before dinner. It may be satisfying for the model’s objective function, but it’s not healthy in the long run.

So where do we go from here? For technology and innovation leaders, the takeaway is both a challenge and a call to action. First, we need to pierce the glamour of predictive prowess and ask the harder questions: How is our AI making this decision? Is it relying on real insight, or is it doing a digital form of reading the room? In practice, this means developing tests that go beyond the training data, - much like the inductive bias probes from our Harvard/MIT friends, to see if our models can handle the unexpected. Encourage your teams to perform “counterfactual” checks: tweak a detail and see if the AI still gets it right for the right reason. And when an AI-driven insight seems too good to be true, remember that sometimes it might be. Second, invest in understanding overfitting and shortcuts in your context. If you’re in healthcare, for example, simulate how your AI handles patients from different demographics or rare conditions, - does performance hold up, or does it crumble outside the familiar training patterns? If you’re in finance, stress-test your AI on economic scenarios that weren’t in the training data (hello, 2020 pandemic) to see if it truly understands risk or just mirrored the last decade’s bull market.

The goal isn’t to throw out predictive AI. It’s to make sure we pair it with domain wisdom and robust checks. Think of your AI like a prodigious intern, - extremely smart, astonishingly quick, but in need of supervision and mentorship to truly grow into the job. Finally, we should push the AI industry towards building models that incorporate causal reasoning and transparency. This might mean hybrid systems that blend data-driven learning with symbolic knowledge of the world, or it could mean new training objectives that reward explanation and penalize spurious shortcuts. The next generation of AI shouldn’t just be bigger, - it needs to be wiser. We owe it to ourselves and maybe society at large to demand AI that can go from Kepler to Newton, - not merely fit the curves, but figure out the laws. Until then, let’s approach the current AI with a healthy dose of peer-to-peer skepticism. Clever it may be, but we’ll keep one eyebrow raised until it shows it can do more than just predict but actually understand.

Further Readings

• What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models (Keyon Vafa et al. – July 2025). Introduces “inductive bias probes” as a method to test whether large AI models learn genuine underlying principles or just surface patterns. The paper’s experiments (across physics simulations and games) show that despite high performance on training tasks, these models often fail to develop true world-model understanding, revealing a significant gap between prediction and comprehension.

• When Artificial Intelligence Takes Shortcuts, Patient Needs Can Get Lost (Matthew R. Francis – May 2025). Explores real-world examples of AI “shortcut learning” in medicine. This article narrates how medical AI systems sometimes achieve correct diagnoses for coincidental reasons (like correlating disease with age or ICU admission). It discusses the dangers of such spurious correlations and the importance of ensuring AI decisions are based on relevant factors.

• Artificial Intelligence and the Risks of HARKing (Hypothesizing After-the-Fact) (Larry Swedroe – January 2025). A perspective on AI in financial research, highlighting the risk of data mining and spurious patterns. The author explains how AI’s predictive power can lead to “false discoveries” – patterns that look meaningful in historical data but lack causal basis – and warns that correlation without understanding can mislead investors and researchers alike.

---

Disclaimer: The perspectives shared in this article are my own and do not represent those of my employer or any affiliated organizations. All company names, product names, logos, and brands mentioned are the property of their respective owners and are used for identification and illustrative purposes only. No endorsement, sponsorship, or affiliation is intended or implied. References to specific companies or case studies are based on publicly available information and are used solely for educational and discussion purposes.

---