The Illusion of AI Thinking: How Apple’s New Paper Changed the Way I See AGI

4 min readJun 17, 2025

I was in a deep conversation with my friend Amulya a doctoral candidate at BU(Boston University) discussing large language models and the future of AGI. Midway through, she dropped a link into our chat.

It was a recent Apple research paper titled “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.” Published in June 2025, it was clear the moment I opened it wasn’t just another AI paper. It was something deeper. Something that challenged the very core of what we think LLMs can do.

Not Just Another Benchmark

We havee all seen pass or fail leaderboard screenshots, multiple choice scores, chain of thought traces, GSM(Grade School Math) type math reasoning sets. This paper is not that. Apple’s researchers shifted the focus entirely, instead of static accuracy, they measured the scalability of reasoning in modern LLMs.

What Makes This Paper Different

Instead of the usual benchmarks, Apple built something they call controllable puzzle environments. Here’s what sets this work apart:

Controllable Puzzle Environments
The team designed synthetic puzzle generators. These let them systematically increase the compositional complexity of problems without changing the underlying logic being tested. So, as the puzzles get harder, it’s not because the rules change but because the steps and reasoning required go up.

Four Core Reasoning Domains

Symbolic manipulation
Mathematical and logical deduction
Planning and multi- step problem solving
Algorithmic pattern detection

Relative-Complexity Index
Every puzzle instance is labeled with a single “relative complexity index,” so all results are plotted on the same scale no apples to oranges comparisons.

Reasoning-Trace Analysis
Instead of just looking at final answers, the researchers analyze every intermediate step. They study how the models’ reasoning processes degrade as complexity increases, not just when they get the answer wrong.

What Did They Find? (And Why It Matters)

The Phase Transition in Model Performance:

All the major models like GPT4, Claude Opus, Gemini 1.5 Pro, Mistral handle low-complexity tasks well.
As soon as complexity increases by even a single step, performance doesn’t just decline gradually, it collapses.
Example: In Boolean logic evaluation:

At depth 3, models perform with near-perfect accuracy.
At depth 4, accuracy drops by over half.
At depth 5, results are barely better than guessing.

This isn’t a slow fade. It’s a sharp “phase transition” one notch higher in complexity, and reasoning just falls apart.

The same pattern showed up across all domains.
This collapse was visible on performance curves, not just final scores.

Tools & Scale Weren’t the Answer

Tool Use:

You’d think giving models calculators, code interpreters, or scratchpads would help. It didn’t.
Longer explanations? Yes.
Better accuracy? No.
Tool use polished their syntax, but didn’t fix scalable reasoning.

Model Size:

Apple tested smaller and the very largest available LLMs.
Larger models delayed the collapse by about one complexity level.
But the “cliff” still came just a bit later.
This points to a limit in architecture, not just scale or training data.

Why Most Benchmarks Miss This

Benchmarking Problem:

Most public benchmarks are clustered at low-complexity indices.
Models look great when tasks are easy.
The “illusion of thinking” is really an illusion of benchmark design.

What Apple Proposes:

Evaluate accuracy as a function of problem complexity — not as a single static number.
Study when and how performance collapses using reasoning-trace analysis.

The Illusion We’re Buying Into

Let’s be clear:

Fluent language isn’t the same as deep reasoning.
These models are brilliant at mimicking understanding until the reasoning bar is raised.
At that point, they revert to being sophisticated pattern matchers, not genuine problem solvers.
The Apple paper argues we’re confusing output fluency with scalable, generalizable intelligence.

What Do We Do Now? (The Authors’ Suggestions)

Hybrid Systems

Combine LLM backbones with explicit symbolic modules, long-term memory, meta-reasoning layers, or planning agents.

Complexity-Aware Evaluation

Use benchmarks that highlight where and how performance collapses, not just a static pass or fail.

Reasoning Transparency

Make reasoning-trace analysis a standard part of model evaluation so we see not just answers, but how those answers break down.

Why I Wrote This

It wasn’t just the paper alone that compelled me to write. It was the conversation with Amulya the curiosity sparked by her sharing this deep dive into AI’s real limits. This research challenges what many of us have come to assume about LLMs and AGI.

If you’re following AI progress and wondering whether the hype reflects reality, take Apple’s findings seriously. Sometimes, all it takes is one thoughtful paper and one honest conversation to completely shift your perspective.

The future of AI depends on how well we understand these limits & how boldly we innovate beyond them.

References

Apple Machine Learning Research. (2025). The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. Retrieved from https://machinelearning.apple.com/research/the-illusion-of-thinking

Stay on the cutting-edge of AI! Follow me on Medium, connect on LinkedIn, and explore latest trends in AI technologies and models. Dive into the world of AI with me and discover new horizons!