FDA File Level Model
The five-level risk category system we described for AI predictions and explanations can be compared to the U.S. Food and Drug Administration's (FDA)regulatory framework for machine learning in medical devices. The FDA's approach, particularly in its guidance on the use of AI and machine learning in medical devices, tends to focus on the safety and effectiveness of these technologies in healthcare.
The FDA's approach to regulating AI and machine learning in medical devices, as outlined in their Artificial Intelligence/Machine Learning Action Plan, does not specifically categorize risks into a five-level model like the one we use in this course. Instead, the FDA focuses on a holistic approach to oversight that encompasses the entire lifecycle of AI/ML-based medical software (SaMD). This includes developing a regulatory framework for software that learns over time, supporting good machine-learning practices, ensuring patient-centered approaches, and advancing real-world performance monitoring. See FDA Releases Artificial Intelligence/Machine Learning Action Plan - January 12th
In the broader context of AI in healthcare, the FDA's role is to ensure the safety and effectiveness of AI-enabled products under its jurisdiction. The FDA considers adapting its review process for rapidly evolving AI-enabled medical devices, particularly those that can change in response to new data in ways that are difficult to predict. This process involves dealing with challenges such as ensuring the algorithms are trained on large, diverse datasets to be generalizable and unbiased, and validating their performance in real-world settings.
In contrast, our five-level risk category system appears to be more about the level of human interaction and oversight in decision-making, ranging from low-stakes assistance to high-stakes autonomous decision-making. While there are similarities in terms of escalating scrutiny and oversight based on risk, the FDA's current approach is more nuanced and specific to the complexities of AI/ML in medical devices, focusing on the totality of a product's lifecycle and the specific challenges of ensuring safety and efficacy in the rapidly evolving field of digital health.
Here's a comparison:
-
Passive Modification: This level, where AI modifies resources without high stakes, can be likened to the FDA's least restrictive category, possibly falling under "Software as a Medical Device" (SaMD) with minimal patient risk. These are typically subject to general controls.
-
Assisted Decision-Making: This level requires human confirmation for AI recommendations. It's similar to the FDA's approach for AI tools that assist healthcare providers in decision-making but don't directly impact patient care. These are closely monitored and may require specific controls to ensure safety and effectiveness.
-
Opt-In High-Stakes: This is akin to more critical FDA-regulated AI applications where the recommendation of AI can significantly impact patient outcomes, but the final decision is still human-dependent. These devices may be subjected to more rigorous premarket review to ensure their safety and effectiveness.
-
Opt-Out High-Stakes: This level parallels FDA-regulated AI applications that play a significant role in patient care, where the default is to accept the AI recommendation unless manually overridden. The FDA would likely require stringent premarket approval or de novo classification for such devices, ensuring high standards of safety and efficacy.
-
Autonomous High-Stakes: This matches the FDA's most critical category, likely akin to AI systems that operate independently in making healthcare decisions or diagnoses without human oversight. These would require the highest level of regulatory scrutiny, including extensive clinical trials and premarket approval, to ensure they meet the stringent criteria for safety and effectiveness.
In summary, while the FDA's regulatory framework is specifically tailored to medical devices and focuses on patient safety and device effectiveness, the five-level AI risk category system you've described has a broader application but aligns well with the FDA's approach in terms of escalating oversight and scrutiny based on the level of risk and autonomy in decision-making.