A few days ago I reread “The Black Box of AI”, an article Davide Castelvecchi published in Nature in 2016. And the truth is, it’s still relevant: how can we trust machines whose reasoning we don’t understand?
The story begins in 1991. Jean Pomerleau, a student at Carnegie Mellon, tried to teach a Humvee to drive itself. His neural network learned something basic: “green on the sides = road.” Everything went fine until the vehicle reached a bridge with no grass and went off the road. An early failure that ended up being the perfect metaphor for what we now call the “black box” problem.
From simple networks to deep learning
In 25 years, those neural networks went from clumsy prototypes to massive systems trained with absurd amounts of data. Today they drive cars, detect cancer, and predict financial markets. But their internal logic remains opaque: a tangle of weighted connections that no one can fully decode.
We no longer write explicit rules. We let algorithms “discover” them on their own. As Michael Tyka from Google said: “The knowledge gets baked into the network, not into us.”
And there lies the dilemma: is it enough that it works if we don’t understand how?
When machines hallucinate
To open the black box, some researchers turned to visualization. Google’s DeepDream (2015) made the hidden layers of a neural network visible by exaggerating what each neuron recognized. The results were disturbing: flowers turning into animals, eyes appearing across the sky.
What they discovered is that neural networks don’t “see” like we do. They construct the world through statistical hallucination. Then came adversarial research: tiny pixel changes made an AI confuse a school bus with an ostrich. Unsettling: our most advanced models can be brilliant and fragile at the same time.
Transparent intelligence?
Scientists like Hod Lipson and Zoubin Ghahramani insist that AI must be interpretable by design. Their systems—Eureqa, the Automatic Statistician—try to rediscover physical laws or generate explanations in natural language. They show that clarity and computation can coexist.
But there are voices like Pierre Baldi’s that remind us: “You trust your brain all the time, and you have no idea how it works.” Maybe opacity is inherent to intelligence itself.
The philosophical turn
What Castelvecchi’s article really highlights is that this is not just a technical problem. It’s a philosophical one. It challenges our idea of what it means to “understand.” If reality itself is too complex to be fully reduced—weather, biology, consciousness—then maybe the black box is not a flaw but a reflection of the limits of human cognition.
AI’s opacity becomes a mirror. It doesn’t just hide how the machine works; it forces us to confront how little we understand our own minds.
In summary
“The Black Box of AI” reminds us that explainability is not just a technical frontier but an epistemological one. To open the black box, we may need more than better code. We may need a new philosophy about what it really means to understand something.