Third-party AI introduces third-party risk. This guide provides a systematic methodology for conducting security audits of AI vendors.
Executive Summary
- Vendor AI extends your risk surface — Their security gaps become your security gaps
- AI adds unique risks — Model security, data handling, and AI-specific vulnerabilities
- Audit before and after — Pre-contract assessment plus ongoing monitoring
- Right-size to risk — Audit depth should match vendor risk level
- Document findings — Audit evidence protects you and enables vendor management
- Remediation matters — Findings without follow-up are wasted effort
- Continuous relationship — Security isn't one-time; it's ongoing
AI Vendor Security Audit Methodology
Phase 1: Scoping
Determine audit depth based on:
- Sensitivity of data shared with vendor
- Criticality of AI service to your operations
- Regulatory requirements
- Previous audit findings
- Contract requirements
Audit types:
| Type | Depth | When to Use |
|---|---|---|
| Documentation review | Low | Low-risk vendors, renewal assessments |
| Questionnaire-based | Medium | Standard vendors, initial assessment |
| Remote audit | Medium-High | Significant vendors, verification needed |
| On-site audit | High | Critical vendors, regulatory requirement |
| Third-party attestation | Variable | When vendor provides SOC2/ISO27001 |
Phase 2: Documentation Request
Request from vendor:
- Security policies and procedures
- Architecture documentation
- Data flow diagrams
- Access control documentation
- Incident response plan
- Business continuity plan
- Compliance certifications (SOC2, ISO27001)
- Penetration test results
- Previous audit reports
Phase 3: Assessment Areas
1. Data Protection
- How is your data encrypted at rest and in transit?
- Where is data stored (geographic location)?
- Who has access to your data?
- How is data isolated from other customers?
- What is the data retention and deletion policy?
2. Model Security
- How are models protected from unauthorized access?
- Is there protection against model extraction attacks?
- How is training data secured?
- What controls exist against adversarial inputs?
3. Access Control
- How is authentication managed?
- What is the authorization model?
- How are privileged accounts managed?
- Is multi-factor authentication required?
4. Logging and Monitoring
- What events are logged?
- How long are logs retained?
- Is there security monitoring?
- Can you access audit logs for your data?
5. Incident Response
- What is the incident response process?
- What are notification timelines?
- How will you be informed of incidents affecting your data?
- What is the breach notification process?
6. Business Continuity
- What is the vendor's RTO/RPO?
- Is there a disaster recovery plan?
- How often is it tested?
- What happens to your data if the vendor fails?
7. Personnel Security
- What background checks are performed?
- What security training is provided?
- How is access revoked when employees leave?
8. Third-Party Risk
- Does the vendor use subprocessors?
- How are subprocessors assessed?
- Are subprocessors disclosed?
Phase 4: Testing (if applicable)
For higher-risk vendors, consider:
- Verification of controls described in documentation
- Technical testing (with vendor permission)
- Review of actual configurations
- Interviews with key personnel
Phase 5: Finding Documentation
For each finding:
- Description of the gap or issue
- Risk level (Critical, High, Medium, Low)
- Evidence or observation
- Recommendation
- Vendor response
- Agreed remediation and timeline
Phase 6: Remediation Tracking
- Establish remediation timelines
- Define verification approach
- Track remediation progress
- Verify closure
AI Vendor Security Audit Checklist
Data Protection
- Encryption at rest verified
- Encryption in transit verified
- Data location documented
- Access controls reviewed
- Data isolation confirmed
- Retention/deletion policy documented
Model Security
- Model access controls reviewed
- Training data protection verified
- Adversarial input controls assessed
- Model extraction protections in place
Access Control
- Authentication mechanisms reviewed
- MFA requirement confirmed
- Privileged access management assessed
- Access review process documented
Logging and Monitoring
- Security logging in place
- Log retention adequate
- Monitoring capabilities verified
- Audit log access available
Incident Response
- Incident response plan reviewed
- Notification timelines acceptable
- Breach notification process documented
- Contact information current
Compliance
- SOC2 report reviewed (if available)
- ISO27001 certification verified (if claimed)
- PDPA compliance confirmed
- Industry-specific requirements met
Contractual
- Security terms in contract
- Right to audit preserved
- Data processing agreement in place
- Exit provisions documented
Common Audit Findings
1. Inadequate Data Isolation — Customer data not properly segregated.
2. Weak Access Controls — Excessive access, no MFA for administrative accounts.
3. Missing Encryption — Data not encrypted at rest or in certain transit paths.
4. Insufficient Logging — Security events not logged or retained inadequately.
5. Incomplete Incident Response — No clear customer notification process.
6. Subprocessor Opacity — Vendor uses subprocessors not disclosed or assessed.
Frequently Asked Questions
Q: How often should we audit AI vendors? A: Annually for critical vendors; every 2-3 years for standard vendors; upon significant changes regardless of schedule.
Q: Can we rely on SOC2 reports? A: They're valuable but not complete. Verify scope covers your needs; supplement with AI-specific questions.
Q: What if the vendor refuses an audit? A: Red flag. Consider whether this is acceptable for the risk level. Document the refusal.
Q: Should we use a third-party auditor? A: Consider it for critical vendors or when you lack internal expertise. Third parties provide independence.
Ready to Assess Your AI Vendors?
Book an AI Readiness Audit to get expert help assessing AI vendor security.
[Contact Pertama Partners →]
References
- NIST. (2024). "Cybersecurity Supply Chain Risk Management."
- ISO 27001. (2024). "Supplier Relationships."
- Singapore CSA. (2024). "Third-Party Security Assessment Guidelines."
Frequently Asked Questions
AI audits must examine training data handling, model security, prompt injection defenses, and AI-specific incident response—areas not covered in traditional software security assessments.
Assess data handling practices, model security, API security, access controls, incident response, compliance certifications, and contract terms for audit rights.
Conduct initial assessment before deployment, annual reviews, and additional assessments after significant changes or incidents. Risk-based frequency for different vendors.
References
- Cybersecurity Supply Chain Risk Management.. NIST (2024)
- Supplier Relationships.. ISO (2024)
- Third-Party Security Assessment Guidelines.. Singapore CSA (2024)
