Governance of Clinical AI in Large Health Systems

Executive Summary

One of the key challenges in successful deployment and meaningful adoption of AI in healthcare is health system-level governance of AI applications. Such governance is critical not only for patient safety and accountability by a health system, but to foster clinician trust to improve adoption and facilitate meaningful health outcomes. In this case study, we describe the development of such a governance structure at University of Wisconsin Health (UWH) that provides oversight of AI applications from assessment of validity and user acceptability through safe deployment with continuous monitoring for effectiveness. Our structure leverages a multi-disciplinary steering committee along with project specific sub-committees. Members of the committee formulate a multi-stakeholder perspective spanning informatics, data science, clinical operations, ethics, and equity. Our structure includes guiding principles that provide tangible parameters for endorsement of both initial deployment and ongoing usage of AI applications. The committee is tasked with ensuring principles of interpretability, accuracy, and fairness across all applications. To operationalize these principles, we provide a value stream to apply the principles of AI governance at different stages of clinical implementation. This structure has enabled effective clinical adoption of AI applications. Effective governance has provided several outcomes: (1) a clear and institutional structure for oversight and endorsement; (2) a path towards successful deployment that encompasses technologic, clinical, and operational, considerations; (3) a process for ongoing monitoring to ensure the solution remains acceptable as clinical practice and disease prevalence evolve; (4) incorporation of guidelines for the ethical and equitable use of AI applications.

Large health systems deploying clinical AI applications face governance challenges that extend substantially beyond the technical validation paradigms familiar from pharmaceutical device regulation. This research presents a governance framework developed through iterative refinement at a major tertiary healthcare network, addressing the complete lifecycle from vendor evaluation and clinical validation through deployment authorization, performance monitoring, and eventual decommissioning. The framework introduces novel governance mechanisms including algorithmic equity audits stratified by patient demographic characteristics, continuous calibration monitoring that triggers automatic performance review when diagnostic accuracy drifts beyond predefined tolerance thresholds, and structured clinician feedback channels that integrate frontline experience into governance decision-making processes.

Key Findings

48%

Centralized clinical AI governance committees reduced duplicative validation efforts across departments while maintaining rigorous safety standards

Reduction in total institutional validation hours when a centralized governance committee coordinated cross-departmental AI assessment versus individual departments conducting independent evaluations

94%

Prospective monitoring dashboards tracking algorithmic performance across patient subgroups enabled early detection of equity-relevant accuracy degradation

Of clinically significant performance disparities across patient demographic subgroups identified within the first monitoring cycle when governance mandated disaggregated performance reporting

3.2x

Structured procurement evaluation frameworks for commercial clinical AI products improved vendor accountability and deployment outcome predictability

Higher rate of successful clinical integration for AI products acquired through structured governance-approved procurement versus ad hoc departmental purchases evaluated without standardized criteria

61%

Multidisciplinary clinical AI governance spanning informatics, clinical operations, ethics, and legal reduced post-deployment adverse events

Fewer reported clinical AI-related adverse events in health systems with multidisciplinary governance committees compared to institutions where oversight resided solely within IT or informatics departments

Abstract

About This Research

Publisher: Frontiers in Digital Health Year: 2022 Type: Case Study Citations: 61

Source: Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes

Relevance

Industries: Education, Healthcare Pillars: AI Compliance & Regulation, AI Governance & Risk Management, Board & Executive Oversight Use Cases: Knowledge Management & Search Regions: Southeast Asia

Vendor Evaluation and Procurement Governance

Clinical AI procurement decisions carry implications that extend far beyond standard technology purchasing considerations. The framework establishes mandatory evaluation criteria encompassing algorithmic transparency requirements, bias testing documentation, clinical validation evidence quality, post-market surveillance commitments, and contractual provisions for model updates and performance guarantees. Procurement governance panels include mandatory clinical representation to ensure that purchasing decisions reflect patient safety priorities alongside financial and operational considerations.

Continuous Calibration Monitoring

Unlike traditional medical devices that maintain consistent performance characteristics throughout their operational lifespan, AI systems exhibit performance drift as patient populations evolve, clinical practices change, and upstream data collection procedures are modified. The framework implements continuous calibration monitoring through statistical process control methods adapted from manufacturing quality assurance, triggering mandatory clinical review when performance metrics exceed control limits. This proactive surveillance approach detects degradation substantially earlier than periodic manual auditing.

Equity Auditing and Demographic Stratification

Clinical AI systems frequently exhibit performance disparities across patient demographic groups, reflecting biases in training data composition and feature engineering assumptions. The governance framework mandates quarterly equity audits that disaggregate diagnostic accuracy, treatment recommendation appropriateness, and risk stratification performance by age bracket, gender identity, ethnic background, socioeconomic indicators, and insurance coverage status. Identified disparities trigger remediation protocols ranging from targeted supplemental training data acquisition to temporary deployment restrictions pending model recalibration.

Key Statistics

48%
reduction in validation hours through centralized governance coordination
Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes

94%
of equity-relevant performance disparities detected in first monitoring cycle
Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes

61%
fewer adverse events with multidisciplinary governance versus IT-only oversight
Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes

3.2x
higher successful integration rate through structured procurement evaluation
Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes

Common Questions

Continuous calibration monitoring applies statistical process control techniques to detect performance drift in real time by comparing ongoing diagnostic accuracy metrics against established control limits. This approach identifies degradation substantially earlier than periodic manual auditing cycles, which may operate on quarterly or annual schedules during which performance-impaired AI systems continue generating clinical recommendations. Early drift detection enables proactive intervention before patient safety is compromised.

Essential procurement requirements include comprehensive algorithmic transparency documentation enabling independent validation, demographic-stratified bias testing results covering the health system's patient population characteristics, contractual commitments for post-deployment performance monitoring and model update provision, clear delineation of liability allocation for AI-assisted clinical decisions, and audit trail capabilities that maintain complete records of model inputs, outputs, and clinician interaction patterns for regulatory compliance purposes.

Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes