Why Explainability Is Now a Market Access Requirement for AI/ML

The regulatory landscape for AI-enabled medical devices is undergoing a fundamental shift. FDA's intensifying focus on algorithmic transparency—coupled with high-profile recalls like Johnson & Johnson's recent Impella heart pump withdrawal following a patient death—signals that explainability is no longer a nice-to-have feature. It's rapidly becoming a prerequisite for market authorisation and a critical component of post-market risk management. For manufacturers developing AI/ML devices and the regulatory affairs teams shepherding them through clearance, this represents both a technical challenge and a strategic imperative that will define commercial viability over the next regulatory cycle.

FDA's Evolving Position on Algorithmic Transparency

FDA has historically evaluated medical devices based on demonstrated safety and effectiveness, with the internal workings of the device being secondary to clinical performance data. AI/ML systems have complicated this paradigm significantly. Unlike traditional software with deterministic logic flows, machine learning algorithms—particularly deep neural networks—often operate as 'black boxes' where even developers cannot fully articulate why a specific input produces a particular output.

This opacity presents profound regulatory challenges. How does FDA assess the safety of a diagnostic algorithm if the manufacturer cannot explain which features the model weighs most heavily? How do clinical evaluators validate that an AI hasn't learned spurious correlations that happen to work in training data but fail catastrophically in real-world deployment? These questions have moved from theoretical concerns to practical gatekeepers in premarket review.

Recent FDA guidance and public statements indicate the agency is hardening its position. Reviewers are increasingly requesting detailed algorithmic transparency documentation: feature importance analyses, ablation studies demonstrating which model components drive predictions, and evidence that the AI's decision-making aligns with clinical reasoning rather than dataset artifacts. For manufacturers accustomed to treating algorithms as proprietary black boxes, this represents a significant shift in what constitutes an approvable submission. The message is clear: if you cannot explain how your AI reaches its conclusions—at least to a degree that allows meaningful safety assessment—FDA may not be able to conclude that your device is safe and effective.

The Post-Market Dimension: When Unexplainable Systems Fail

The J&J Impella recall, while not explicitly an AI/ML device issue, provides a sobering parallel that regulatory professionals should study carefully. The recall followed a patient death and was linked to quality system findings during post-acquisition integration audits. What makes this particularly instructive is the cascade it represents: a quality system gap led to a product defect, which resulted in patient harm, triggering both a field action and heightened regulatory scrutiny.

Now transpose this scenario onto an AI/ML device where the 'defect' is not a manufacturing flaw but an algorithmic failure—perhaps a diagnostic system that misclassifies a particular patient subgroup because it learned an unintended bias during training. If the manufacturer cannot explain why the algorithm failed, they cannot effectively implement corrective actions. They cannot confidently define the scope of affected patients. They cannot demonstrate to regulators that proposed mitigations actually address the root cause. In short, lack of explainability transforms a contained post-market issue into an existential regulatory crisis.

This is why FDA's focus on transparency is fundamentally about risk management throughout the total product lifecycle. Explainable AI isn't just about getting through premarket review—it's about having the technical capability to investigate complaints, validate algorithm performance across diverse patient populations, and demonstrate continued safety when model updates are deployed. The Impella recall underscores what regulators already know: when things go wrong with complex medical devices, the ability to rapidly diagnose and remediate is just as important as initial validation.

What This Means for Your Team

For regulatory affairs professionals, this convergence of FDA expectations and post-market realities demands a fundamental reassessment of AI/ML device strategy. First, explainability must be architected into the product from the earliest development stages—it cannot be retrofitted during submission preparation. This means RA teams need to be involved in algorithm design discussions, ensuring that technical choices (model architectures, training methodologies, validation approaches) support regulatory requirements for transparency. If your data scientists are optimising purely for predictive accuracy without considering interpretability, you're building regulatory risk into your product.

Second, technical documentation packages need to evolve significantly. Traditional software documentation focused on requirements traceability and verification testing is insufficient for AI/ML. FDA now expects to see algorithm transparency reports that may include: feature importance rankings with clinical justification, visualization of model attention or decision boundaries, sensitivity analyses showing how predictions change with input variations, and evidence that the model's logic aligns with established clinical reasoning. These aren't optional enhancements—they're becoming standard expectations in premarket review.

Third, post-market surveillance systems must be designed with algorithmic monitoring in mind. For AI/ML devices, particularly those with adaptive algorithms, post-market performance monitoring cannot rely solely on traditional complaint handling and adverse event reporting. You need systems that continuously assess whether the algorithm is performing as intended across real-world patient populations, detect performance drift or emerging failure modes, and provide the data needed to explain any deviations. When FDA comes asking why your device failed in a specific case—and they will—'the algorithm made a mistake' is not an acceptable answer. You need to be able to explain what inputs led to the error, why the model responded that way, and what systemic factors contributed.

Finally, quality management systems need explicit processes for AI/ML governance. The J&J recall highlights how quality system gaps identified during integration audits can cascade into field actions. For AI/ML devices, this means establishing clear ownership of algorithm validation, defining what constitutes a significant algorithm change requiring regulatory notification, and maintaining the technical infrastructure to investigate and explain algorithm behavior throughout the product lifecycle. These processes should be in place and demonstrable to auditors long before your first submission.

Key Takeaways

• Algorithmic explainability is transitioning from a technical preference to a regulatory requirement—FDA increasingly expects manufacturers to demonstrate transparency in how AI/ML devices reach their outputs, not just that they produce accurate results.
• Post-market risk management for AI/ML devices is impossible without explainability—when algorithm failures occur, manufacturers must be able to diagnose root causes, define affected populations, and validate corrective actions, none of which is possible with black-box systems.
• Regulatory affairs teams must engage in algorithm development early to ensure interpretability is designed in, not bolted on—model architecture and training decisions have direct regulatory consequences that cannot be addressed during submission preparation.
• Technical documentation for AI/ML submissions must include algorithm transparency evidence such as feature importance analyses, sensitivity studies, and clinical justification for model logic—traditional software documentation is insufficient for FDA's current review expectations.
• Quality management systems need explicit AI/ML governance processes covering algorithm validation, change control for model updates, and post-market performance monitoring—gaps in these systems carry the same regulatory risk as manufacturing quality failures in traditional devices.

The convergence of FDA's explainability expectations and the practical lessons from recent recalls points to a regulatory environment where transparency is non-negotiable. Manufacturers who view explainability as a technical burden rather than a strategic asset will find themselves at a competitive disadvantage—both in time to market and in post-market resilience. Those who embed transparency into their development processes from the outset will not only navigate regulatory review more efficiently but will build devices that are fundamentally more defensible when performance questions inevitably arise. The question for your organisation isn't whether to invest in explainable AI—it's whether you can afford the regulatory and commercial consequences of not doing so. At SMEDTEC, we're working with clients to integrate these explainability requirements into their AI/ML regulatory strategies before they become submission bottlenecks, because in this regulatory environment, the devices that can explain themselves are the ones that make it to market.