Executive Summary: Research from Gartner, McKinsey, and Stanford shows 70% of AI projects fail—not because the technology is impossible, but because organizations skip fundamental best practices. This comprehensive checklist consolidates lessons from hundreds of AI failures into a practical, stage-by-stage framework covering readiness, data quality, vendor selection, governance, adoption, integration, ROI, scope, technical debt, and security. Use this master checklist to systematically prevent the failures that derail most AI initiatives.
How to Use This Checklist
For New Projects: Work through sections 1–10 sequentially from project kickoff through launch and operations.
For In-Flight Projects: Jump to relevant sections based on current challenges (e.g., Section 6 for adoption issues, Section 8 for scope creep).
For Audits: Use the entire checklist to assess project health and identify gaps.
Scoring: Track completion percentage per section. Aim for >80% completion in each section before advancing to the next phase.
Section 1: AI Readiness Assessment (Pre-Project)
Strategic Readiness
- Clear business problem defined with measurable success criteria
- Executive sponsor identified with authority and time commitment
- Budget allocated covering full lifecycle (not just initial build)
- Realistic timeline (6–12 months for meaningful AI projects)
- Alternative solutions evaluated ("Is AI the right approach?")
Organizational Readiness
- Cross-functional team assembled (ML, engineering, product, domain experts)
- Stakeholder alignment on goals and constraints
- Change management plan in place
- Training budget allocated for users
- Success metrics agreed upon by all stakeholders
Technical Readiness
- Data availability and quality assessed
- Infrastructure capacity evaluated
- Integration requirements documented
- Security and compliance requirements identified
- Technical debt in existing systems evaluated
Section Score: _/15 (__%)
Section 2: Data Quality & Preparation
Data Availability
- Sufficient volume of data (typically 1K–1M+ examples depending on problem)
- Data representative of real-world conditions
- Historical data covers relevant time periods
- Edge cases and rare scenarios included
- Data access permissions secured
Data Quality
- Data accuracy validated (>95% for most use cases)
- Missing data quantified (<10% missing for critical features)
- Duplicate records identified and handled
- Outliers and errors detected
- Data freshness verified (not stale)
Data Governance
- Data lineage documented (where data comes from)
- Data versioning implemented
- Privacy and security controls in place
- Compliance requirements met (GDPR, HIPAA, etc.)
- Data access audit trail maintained
Bias & Fairness
- Protected characteristics identified (race, gender, age, etc.)
- Data distribution checked across demographics
- Historical bias in data acknowledged
- Fairness metrics defined
- Bias mitigation strategies planned
Section Score: _/20 (__%)
Section 3: Vendor & Tool Selection
Vendor Evaluation
- Multiple vendors evaluated (minimum 3)
- POC/trial conducted with real data
- Vendor financial stability assessed
- Customer references checked (minimum 3)
- Vendor roadmap alignment verified
Technical Fit
- Integration requirements validated
- Performance benchmarks on your data confirmed
- Scalability limits understood
- API capabilities and limits documented
- Customization options explored
Commercial Terms
- Pricing model understood (per-user, per-transaction, per-API-call)
- Hidden costs identified (training, support, professional services)
- SLA commitments documented (uptime, response time, accuracy)
- Lock-in risks evaluated (data portability, switching costs)
- Contract terms negotiated (exit clauses, renewal terms)
Build vs. Buy
- Total cost of ownership compared (build vs. buy)
- Time to value compared
- Long-term maintenance costs estimated
- Internal capabilities assessed
- Strategic importance evaluated (core vs. non-core)
Section Score: _/20 (__%)
Section 4: AI Governance & Oversight
Governance Structure
- AI steering committee established
- Decision-making authority clarified
- Meeting cadence defined (typically bi-weekly)
- Escalation process documented
- Stakeholder representation balanced
Policies & Standards
- AI ethics policy defined
- Model risk management framework established
- Data usage policies documented
- Explainability requirements defined
- Human-in-the-loop requirements specified
Monitoring & Auditing
- Performance metrics tracked continuously
- Bias audits scheduled (quarterly minimum)
- Model drift monitoring implemented
- Decision audits conducted for high-stakes decisions
- Compliance reporting automated
Documentation
- Model cards created (purpose, training data, limitations)
- Risk assessments documented
- Decision rationale recorded
- Incident response procedures defined
- Lessons learned captured
Section Score: _/20 (__%)
Section 5: User Adoption & Change Management
User Involvement
- Users involved from project start (not end)
- User needs and pain points documented
- User feedback incorporated into design
- Beta testing with real users conducted
- User champions identified in each department
Training & Support
- Role-specific training created
- Just-in-time training provided (not months before launch)
- Training effectiveness measured
- Support resources available (help docs, videos, live support)
- FAQs and troubleshooting guides published
Communication
- Clear value proposition communicated
- Change rationale explained (why AI, why now)
- Timeline and milestones shared transparently
- Concerns and resistance addressed openly
- Success stories celebrated and shared
Incentives & Accountability
- Performance metrics aligned with AI adoption
- Managers held accountable for team adoption
- Early adopters recognized and rewarded
- Non-adoption consequences clarified
- Adoption rates tracked and reported
Section Score: _/20 (__%)
Section 6: Integration & Workflow Design
System Integration
- Integration architecture designed (APIs, data pipelines, etc.)
- Authentication and authorization configured
- Data synchronization approach defined
- Error handling and retry logic implemented
- Performance and latency requirements met
Workflow Integration
- AI fits naturally into existing workflows (not bolted on)
- User journeys mapped with AI touchpoints
- Handoffs between human and AI defined
- Fallback procedures for AI failures documented
- Workflow testing with real users completed
User Experience
- AI outputs presented clearly and actionably
- Confidence scores displayed when appropriate
- Explanations provided for AI decisions
- Users can override AI recommendations
- Feedback mechanism for incorrect predictions built-in
Invisible Integration Principles
- AI integrated where users already work (not separate tool)
- Minimal training required (intuitive design)
- Value delivered immediately (no setup hassle)
- AI enhances (not replaces) human judgment
- Progressive disclosure (simple by default, complex when needed)
Section Score: _/20 (__%)
Section 7: ROI & Business Value
ROI Framework
- Baseline metrics measured (current state performance)
- Target improvement quantified (specific, measurable)
- Cost components identified (development, infrastructure, operations)
- Benefit components quantified (time saved, revenue increase, cost reduction)
- Payback period calculated (typically 12–24 months)
Realistic Expectations
- ROI timeline communicated clearly (not immediate)
- Phased value delivery planned (quick wins + long-term gains)
- Risk-adjusted returns calculated
- Sensitivity analysis performed (best/worst/likely scenarios)
- Opportunity cost considered (what else could resources do?)
Value Tracking
- KPIs defined and baselined
- Measurement systems implemented
- Attribution methodology agreed upon
- Regular ROI reporting scheduled
- Value realization tracked against projections
Financial Accountability
- Budget owner identified
- Cost tracking implemented
- Budget variance monitored monthly
- Investment decision criteria documented
- Go/no-go checkpoints defined
Section Score: _/20 (__%)
Section 8: Scope & Project Management
Scope Definition
- Single primary success metric defined
- MVP features list (maximum 5–7 features)
- Explicit exclusions documented (what we won't build)
- Phase 2 backlog created
- Constraints documented (timeline, budget, resources)
Change Control
- Change control process defined
- Impact assessment template created
- Approval authority clarified
- Change log maintained
- Scope health metrics tracked weekly
Agile Delivery
- 2-week sprints defined
- Sprint planning process established
- Demo cadence agreed upon
- Retrospectives scheduled
- Velocity tracked and trends monitored
Risk Management
- Risk register maintained
- Risk mitigation plans documented
- Risk owners assigned
- Risk reviews conducted monthly
- Escalation triggers defined
Section Score: _/20 (__%)
Section 9: Technical Debt & Quality
Code Quality
- Version control (git) from day 1
- Unit tests written (>80% coverage for critical code)
- Code reviews conducted on all changes
- Linting and code style automated
- Documentation maintained (README, API docs)
MLOps Practices
- Model versioning implemented (track every model change)
- Experiment tracking configured (hyperparameters, metrics)
- Training reproducibility verified (can recreate any model)
- Model registry maintained (catalog of all models)
- A/B testing framework for model updates
Data Engineering
- Data pipelines versioned and tested
- Data quality checks automated
- Feature engineering documented
- Data lineage tracked
- Data versioning implemented (DVC, Pachyderm, etc.)
Infrastructure
- Infrastructure as code (Terraform, CloudFormation)
- CI/CD pipelines configured
- Auto-scaling configured
- Disaster recovery plan documented
- Cost monitoring and alerts enabled
Maintenance Planning
- 70/20/10 allocation (features/debt/learning)
- Refactoring sprints scheduled quarterly
- Technical debt backlog maintained
- Onboarding time tracked (<2 weeks target)
- Deployment frequency tracked (multiple per week target)
Section Score: _/25 (__%)
Section 10: Security & Compliance
Data Security
- Data encryption at rest and in transit
- Access controls implemented (least privilege)
- Authentication and authorization configured
- Audit logging enabled
- Data retention policies defined
Model Security
- Model access controls implemented
- API rate limiting configured
- Input validation and sanitization implemented
- Adversarial robustness tested
- Model extraction risks mitigated
Privacy
- Privacy impact assessment conducted
- PII handling procedures documented
- Anonymization/pseudonymization applied where appropriate
- Data minimization practiced (only collect what's needed)
- User consent obtained and tracked
Compliance
- Regulatory requirements identified (GDPR, HIPAA, etc.)
- Compliance documentation maintained
- Audit trail complete and accessible
- Right to explanation implemented
- Regular compliance audits scheduled
Incident Response
- Security incident response plan documented
- Incident detection systems configured
- Response team identified and trained
- Communication plan defined (internal and external)
- Post-incident review process established
Section Score: _/25 (__%)
Master Scoring & Interpretation
Total Score: _/225 (__%)
Scoring Guide
90–100% (203–225 points): Excellent
Your project has strong foundations across all dimensions. Continue monitoring and maintaining these practices. Small gaps should be addressed but shouldn't block progress.
75–89% (169–202 points): Good
Solid foundation with some gaps. Identify and address gaps in lower-scoring sections before launch. Focus on sections scoring <75%.
60–74% (135–168 points): Fair
Significant gaps exist that increase failure risk. Do not proceed to production until critical gaps (especially in governance, security, and data quality) are addressed.
Below 60% (<135 points): High Risk
Project is at high risk of failure. Recommend project pause to address fundamental gaps. Consider bringing in external expertise or conducting a full project reset.
Priority Gap Analysis
Critical Gaps (Must fix before launch):
- Data quality issues
- Security vulnerabilities
- Compliance gaps
- Missing governance structure
- No user adoption plan
High Priority Gaps (Fix within 30 days of launch):
- Integration issues
- Monitoring gaps
- Technical debt accumulation
- Scope creep patterns
- Insufficient training
Medium Priority Gaps (Address in first 90 days):
- ROI tracking improvements
- Documentation completeness
- Process optimizations
- Tool upgrades
Key Takeaways
- 70% of AI projects fail—this checklist systematically addresses the most common failure modes.
- Use sequentially for new projects, jumping to relevant sections for in-flight projects or audits.
- Aim for >80% completion per section before advancing to the next project phase.
- Critical sections (data quality, governance, security) require 100% completion before production launch.
- Track completion over time—regular checklist reviews reveal drift and emerging risks.
- Adapt to your context—this is a comprehensive framework; customize based on project size, risk, and industry.
- Share with stakeholders—the checklist creates common language and shared accountability for project success.
Frequently Asked Questions
Do I really need to complete 100% of this checklist?
No—aim for >80% overall, with 100% in critical areas (data quality, security, governance). Smaller, lower-risk projects may reasonably skip some items. Use judgment based on your context, but document why items are skipped.
What if we're already in production and scoring <60%?
Don't panic, but do take action. Conduct a rapid gap assessment, prioritize critical issues (security, data quality, compliance), create a remediation plan with timeline, and consider rolling back features while addressing gaps. Many in-flight projects can be rescued with focused effort.
How often should we review this checklist?
New projects: weekly during development, biweekly after launch. Mature projects: monthly reviews. Major changes (new features, vendor changes, regulatory updates) trigger full re-assessment. Treat it as a living document, not a one-time exercise.
Who should be involved in completing the checklist?
Cross-functional team: AI/ML lead, project manager, data engineer, security officer, compliance lead, user representative, executive sponsor. No single person can complete it alone—this is a team exercise that reveals gaps through diverse perspectives.
What if we don't have resources to address all gaps?
Prioritize ruthlessly: security and compliance first (non-negotiable), data quality second (garbage in, garbage out), governance third (prevents chaos), then everything else. Consider reducing scope, extending timeline, or securing additional budget rather than launching with critical gaps.
How does this checklist differ for pilot vs. production projects?
Pilots can defer some operational items (scalability, full monitoring, comprehensive training) but must maintain rigor in data quality, ethics, and security. Define pilot objectives clearly and use a simplified version of this checklist. Promote to production only when full checklist is >80% complete.
Can we use this checklist for AI vendor selection?
Yes—Section 3 addresses vendor selection, but also use the full checklist to assess vendor solutions: Do they help with data quality (Section 2)? Do they provide governance tools (Section 4)? Does their solution integrate well (Section 6)? A good vendor should enable high scores across all sections.
Additional FAQs
Do I really need to complete 100% of this checklist?
No, but aim for >80% overall with 100% in critical areas like data quality, security, and governance. Smaller or lower-risk projects may reasonably skip some items, but document why and reassess if risk level changes.
What if we're already in production and scoring below 60%?
Conduct a rapid gap assessment focusing on critical issues (security, data quality, compliance). Create a remediation plan with clear timelines and ownership. Consider feature rollback or limited deployment while addressing the most severe gaps.
How often should we review this checklist during a project?
Weekly during active development, biweekly in the first 90 days post-launch, then monthly for mature projects. Major changes (new features, vendor switches, regulatory updates) trigger full reassessment.
Who should be involved in completing the checklist?
A cross-functional team: AI/ML lead, project manager, data engineer, security officer, compliance lead, user representative, and executive sponsor. No single person can accurately assess all dimensions—diverse input reveals blind spots.
What if we lack resources to address all identified gaps?
Prioritize ruthlessly: security and compliance are non-negotiable, data quality is next (garbage in, garbage out), then governance. Consider reducing scope, extending timelines, or securing additional budget rather than launching with critical gaps unresolved.
How does this checklist apply differently to pilot vs. production projects?
Pilots can defer some operational elements (full scalability, comprehensive monitoring, extensive training) but must keep rigor in data quality, ethics, and security. Define clear pilot success criteria and use a streamlined version. Only promote to production when the full checklist reaches >80%.
Can this checklist help evaluate AI vendors?
Yes. Section 3 is vendor-specific, but assess vendors against the full framework: Do they support good data practices (Section 2)? Provide governance tools (Section 4)? Integrate smoothly (Section 6)? Strong vendors help you score higher across all sections, not just procurement criteria.
Frequently Asked Questions
No—aim for more than 80% overall, with 100% completion in critical areas such as data quality, security, and governance. For smaller or lower-risk projects, you can skip some items, but you should document the rationale and revisit those decisions if the project’s risk profile changes.
Run a rapid gap assessment focused on security, data quality, and compliance. Define a remediation plan with clear owners and deadlines, and consider rolling back or limiting high-risk features until critical issues are resolved.
Review weekly during active development, biweekly in the first 90 days after launch, and monthly for mature systems. Any major change—such as new features, vendor changes, or regulatory updates—should trigger a full reassessment.
A cross-functional group including the AI/ML lead, project manager, data engineer, security officer, compliance lead, user representative, and executive sponsor should complete the checklist together to surface blind spots and ensure shared ownership.
Treat security and compliance as non-negotiable, address data quality next, then governance and operational issues. Reduce scope, extend timelines, or seek additional budget rather than going live with unresolved critical gaps.
Pilots can relax some scalability, monitoring, and training requirements, but must still meet high standards for data quality, ethics, and security. Only promote a pilot to production once the full checklist is at least 80% complete with no critical gaps.
Yes. Use Section 3 for direct vendor evaluation and the rest of the checklist to test whether a vendor supports strong data practices, governance, integration, and security. Strong vendors should help you achieve high scores across multiple sections.
Most AI failures are preventable
Analysts estimate that around 70% of AI projects fail to reach their intended business impact. The majority of these failures stem from gaps in readiness, governance, data quality, and adoption—not from limitations of the underlying models.
Estimated share of AI projects that fail to deliver expected value
Source: Gartner, McKinsey, Stanford HAI syntheses (2024–2025)
"AI success is less about model accuracy and more about disciplined execution across data, governance, integration, and change management."
— Enterprise AI implementation best-practice synthesis
References
- AI Project Failure Analysis. Gartner (2025)
- Enterprise AI Success Factors. McKinsey & Company (2024)
- AI Implementation Best Practices. Stanford HAI (2025)
- AI Governance Framework. MIT Sloan (2024)
- AI Risk Management Guide. NIST (2024)