When your toaster fails to brown your bread evenly, you don’t lose sleep over it. But when an AI system denies your loan application or flags you for additional screening at the airport, you might want to know why. (Funny how stakes change our appetite for explanations.) As we hurtle toward a future where AI systems make increasingly consequential decisions about our lives, the question of explainability has moved from academic curiosity to urgent necessity.
The rush to make artificial intelligence more transparent – what the industry calls “explainable AI” or XAI – has become something of a holy grail in tech circles. It’s as if we’ve built a machine that can solve complex puzzles but can’t explain its methods any better than a dog can explain why it chases squirrels. The difference is, we’re letting these algorithmic black boxes make decisions that affect everything from our credit scores to our medical treatments.
How we got here #
Remember when computers were simple? (Neither do I, but I’ve read about it.) The evolution of AI explainability mirrors the evolution of AI itself, and it’s worth understanding how we got here. Looking at the recent academic literature, particularly the comprehensive work by Sun et al. and Gagnon et al., we can trace a fascinating progression:
-
The Age of Innocence (Traditional Machine Learning Era): Back when machine learning was young and optimistic, we had decision trees you could actually draw on a whiteboard. Models were simple enough that you could explain them to your grandmother over tea. Those were the days – limited capability, but crystal-clear logic.
-
The Deep Learning Revolution: Then came deep learning, and everything got… interesting. Suddenly we had neural networks with millions of parameters doing amazing things, but nobody could really explain how. It’s like we taught a computer to be a chess grandmaster by having it play itself millions of times, but it couldn’t tell you why it moved its bishop to e4.
-
The Foundation Model Explosion: Now we’re in the era of massive language models and multimodal AI systems. These are like digital savants – incredibly capable but often inscrutable. They can write poetry, generate images, and engage in seemingly intelligent conversation, but good luck getting them to explain their reasoning in a way that would satisfy a skeptical philosopher.
The Three Levels of the Explainability Problem #
Level 1: Technical Feasibility #
At its most basic level, the challenge is technical. Can we actually peek inside these neural networks and understand what’s happening? The papers suggest we can, to some extent, but it’s like trying to understand a city by looking at its electrical grid – you get some insights, but you miss the human story.
What makes this particularly challenging is that modern AI systems often learn patterns that are too complex for human comprehension. It’s not just about following a set of if-then rules anymore; it’s about recognizing subtle patterns across millions of data points. (Try explaining how you recognize a face – now imagine doing that with mathematics.)
Level 2: Human Understanding #
Even if we can technically explain how an AI system works, can we translate that explanation into something humans can meaningfully understand? This is where things get really interesting, because different stakeholders need different types of explanations:
- A developer needs to understand the technical architecture
- A doctor needs to understand the clinical reasoning
- A patient needs to understand why they should trust the diagnosis
- A regulator needs to understand if the system is fair and compliant
It’s like trying to explain quantum physics – there’s the mathematical explanation, the metaphorical explanation, and the “trust me, it works” explanation. Each serves a different purpose and audience.
Level 3: Societal Implications #
This is where things get philosophical (and a bit existential). Even if we achieve technical explainability and human understanding, what does that mean for society? Some key questions we need to grapple with:
- Should AI systems be held to a higher standard of explainability than human decision-makers?
- How do we balance transparency with performance?
- What role should explainability play in regulatory frameworks?
- How do we ensure explanations don’t become manipulation tools?
The Multimodal Challenge #
Here’s where things get really complicated …
One particularly thorny issue both papers touch on is multimodal AI – systems that combine different types of data (text, images, audio, etc.). It’s like trying to explain why you enjoy a particular song by separately analyzing the rhythm, melody, and lyrics. The whole is often more than the sum of its parts, and our traditional methods of explanation start to break down.
Consider a medical diagnosis system that combines:
- Patient history (text)
- Medical imaging (visual)
- Lab results (numerical)
- Genetic data (categorical)
- Sensor readings (time series)
How do you explain decisions that emerge from the complex interplay of all these different types of data? It’s a challenge that makes single-modality explainability look like child’s play.
The Sensemaking Revolution #
Here’s where things get interesting – and where Gagnon’s paper offers a perspective that’s either brilliant or obvious (often the same thing in retrospect).
Instead of trying to solve the Rubik’s cube of perfect AI explanations, what if we’re thinking about this all wrong? What if, rather than treating AI systems like vending machines of explanation – insert question, receive answer – we approached them more like jazz musicians in an improv session? Sometimes they follow familiar patterns, sometimes they take unexpected turns, but the magic happens in the back-and-forth between player and audience, each adapting to and building on the other’s cues.
This shift from monologue to dialogue isn’t just academic hair-splitting. It’s like the difference between reading a wine label and having a conversation with a sommelier. Sure, the label tells you the basics, but the sommelier helps you understand why this particular Bordeaux might make your Tuesday night pasta transcendent. Similarly, AI explanations work best when they’re part of a conversation, not a TED talk.
This perspective flip does four things (and like all good magic tricks, the fourth one’s the best):
- Makes us active participants rather than passive observers (goodbye, AI lecture mode)
- Brings human intuition back into the equation (turns out we’re not completely obsolete)
- Allows for context-specific understanding (because one size fits about as well as those “free size” hats)
- Enables our understanding to evolve over time (like how we finally figured out that maybe infinite scrolling wasn’t humanity’s greatest invention)
Think of it less like reading an instruction manual and more like having a conversation with a knowledgeable but sometimes cryptic friend. The understanding emerges through dialogue, not monologue.
What This Means for the Real World? #
Let’s get real for a moment. All this theory is about as useful as a chocolate teapot if we can’t put it to work. Here’s what actually matters:
- Context is King (and Queen, and the Whole Royal Family)
The level and type of explanation needed varies dramatically based on context. A medical diagnosis might require detailed justification, while a movie recommendation might need minimal explanation. Different stakes, different explanations.
- User-Centered Design
Explainability needs to be designed with specific users in mind. This might mean multiple layers of explanation, from high-level summaries to detailed technical documentation. Some users want the full technical breakdown; others just want to know it works better than a coin flip. (Spoiler: it usually does, but only slightly in some cases.)
- Interactive Explanations
Static explanations are often insufficient. Interactive systems that allow users to explore and question the AI’s reasoning might be more effective.
- Trust Building
Explanations aren’t just about understanding – they’re about building trust. Sometimes this means acknowledging uncertainty and limitations rather than trying to explain everything.
Looking Forward (With Both Eyes Open) #
As someone who still has a Netflix DVD somewhere in their drawer (purely for nostalgic purposes, I assure you), I’ve learned to be cautiously optimistic about tech predictions. Here’s what’s actually worth watching:
- Regulatory Pressure
Increased regulation around AI transparency will likely drive innovation in explainability methods.
The EU’s AI Act isn’t just another Brussels bureaucratic fever dream – it’s the canary in the regulatory coal mine. When the world’s largest trading bloc decides AI systems need to come with an explanation manual, Silicon Valley listens. (Or at least their lawyers do.)
With 37% of global companies already reporting regulatory compliance as their primary driver for XAI investment, this trend has more momentum than a runaway ML model. Watch for the ripple effects as other regions scramble to catch up – Japan’s already working on their own AI governance framework, and the U.S. is finally waking up to the party.
- User Expectations
As AI becomes more prevalent, user expectations around explainability will evolve.
A recent Gartner study suggests 85% of AI projects through 2025 will deliver erroneous outcomes due to bias in data or algorithms – and users are getting wise to it. The same way we now expect to know what’s in our food, consumers are developing a taste for AI transparency. Companies that can’t explain their AI decisions might soon find themselves with all the popularity of a Windows Vista reunion tour.
- Technical Innovations
New approaches to building interpretable AI systems from the ground up, rather than trying to explain black boxes after the fact.
The explainable AI market is projected to grow from $3.5 billion in 2020 to $21 billion by 2030, and it’s not just because everyone suddenly developed a passion for gradient descent algorithms. We’re seeing fascinating developments in “glass box” models that are interpretable by design (imagine that – AI systems that can actually explain themselves without needing an interpreter). Google’s TCAV and IBM’s AI FactSheets are just the beginning. The real breakthrough will come when we figure out how to make models that are both powerful and transparent – the holy grail of AI development, if you will.
- Cultural Shifts
Changes in how we think about human-AI interaction and what constitutes a satisfactory explanation.
This might be the most interesting trend of all – we’re watching society’s relationship with AI evolve in real-time. Just as we developed social norms around smartphone use (no phones at dinner, unless it’s to settle a debate about who starred in that one movie), we’re creating new expectations around AI transparency. A recent MIT study showed that 72% of consumers are worried about AI making decisions about them – yet 82% want AI systems in healthcare. We’re not afraid of AI; we’re afraid of AI we don’t understand.
The New Players #
Here’s a plot twist worthy of M. Night Shyamalan: some of the most interesting developments in explainable AI are coming from outside traditional tech circles. Insurance companies, of all people, are becoming unlikely innovators in this space (when your business model depends on explaining why you’re denying claims, you get creative about transparency). Healthcare providers are developing novel ways to make AI diagnoses comprehensible to both doctors and patients. Even financial institutions – not traditionally known for their transparency – are investing heavily in explainable AI systems.
The Path Forward #
If you’re looking for a neat, tidy conclusion, I’ve got bad news for you. The future of explainable AI looks about as straightforward as a Christopher Nolan plot timeline. But here’s what we can reasonably aim for:
- Explanations that don’t require a PhD to understand
- Systems that can engage in actual dialogue (not just “Computer says no”)
- Built-in transparency that doesn’t tank performance
- Acceptance that some mystery is okay (just like we don’t fully understand human consciousness, and we’re doing fine… mostly)
- Practical frameworks that work in the real world
- Better ways to measure if explanations actually help (beyond “Did you understand that?”)
- Standards that make sense for different contexts
Remember: perfect explainability in AI is like perfect communication in relationships – a noble goal we’ll never quite reach, but one worth pursuing anyway. At least the AI won’t get offended when we ask it to explain itself again.