AI Incident Classification: How to Categorize and Prioritize

When an AI incident hits, the first critical decision is: how serious is this? Get it wrong—classifying a critical incident as routine, or treating a minor issue as an emergency—and you either miss your response window or waste resources on over-reaction.

Effective incident classification ensures the right level of response for each situation. It prevents panic over minor issues and complacency over serious ones.

This guide provides a framework for categorizing AI incidents by type and severity, with clear criteria for prioritization.

Executive Summary

• Classification drives response: Severity level determines response time, escalation, resources, and notification
• AI incidents have unique categories: Model failures, bias events, data exposures, and adversarial attacks require AI-specific classification
• Objective criteria prevent subjective judgment: Define clear thresholds to ensure consistent classification
• Classification can change: Initial assessment may be wrong; build in reassessment
• Over-classification is better than under-classification: When uncertain, classify higher
• Document classification decisions: Reasoning should be recorded for review and learning

Why This Matters Now

Without a classification framework, incident response becomes chaotic:

• Critical incidents get delayed because they weren't recognized as critical
• Routine issues consume executive attention because everything gets escalated
• Response is inconsistent because different responders apply different standards
• Resources are misallocated because priority isn't clearly determined
• Post-incident review is difficult because there's no standard to compare against

A clear classification framework creates shared understanding of what different severity levels mean and what response they require.

AI Incident Type Categories

Category 1: Model Performance Incidents

Incidents where AI models produce incorrect, degraded, or unexpected outputs.

Category 2: Data Incidents

Incidents involving data handled by or exposed through AI systems.

Category 3: Bias and Fairness Incidents

Incidents where AI produces discriminatory or unfair outcomes.

Category 4: Security Incidents

Incidents involving security threats to or via AI systems.

Category 5: Governance and Compliance Incidents

Incidents involving policy violations or compliance failures.

Category 6: Operational Incidents

Incidents affecting AI system operations.

AI Incident Classification Matrix

Severity Level Definitions

Classification Decision Tree

START: AI incident detected

1. Is there active harm occurring to individuals or customers?
   YES → CRITICAL
   NO → Continue

2. Is sensitive personal data exposed or at risk?
   YES (large scale) → CRITICAL
   YES (limited scale) → HIGH
   NO → Continue

3. Is there regulatory notification required within 72 hours?
   YES → CRITICAL
   LIKELY → HIGH
   NO → Continue

4. Is there significant business operation disruption?
   YES → HIGH
   LIMITED → MEDIUM
   NO → Continue

5. Is there potential for harm if unaddressed?
   SIGNIFICANT → HIGH
   MODERATE → MEDIUM
   MINIMAL → LOW

6. Is there a policy or compliance violation?
   SERIOUS → MEDIUM (minimum)
   MINOR → LOW
   NO → Continue

7. Default classification based on operational impact:
   Significant → MEDIUM
   Limited → LOW

Severity Criteria Matrix

Response Requirements by Severity

Classification Examples

Example 1: AI Chatbot Hallucination

Scenario: Customer-facing AI chatbot tells a customer they're entitled to a refund they're not actually entitled to. One customer affected, no data breach.

Classification process:

• Active harm? No (minor financial misstatement)
• Sensitive data exposed? No
• Regulatory notification? No
• Business disruption? No
• Harm potential? Low (one incorrect response)

Classification: LOW

But note: If the chatbot is making this error systematically, reclassify to MEDIUM or HIGH.

Example 2: AI System Data Leak

Scenario: AI system accidentally includes personal data from training set in outputs. 50 customers' names and emails exposed.

Classification process:

• Active harm? No (exposed, not exploited)
• Sensitive data exposed? YES (personal data)
• Regulatory notification? LIKELY (personal data breach)
• Business disruption? Limited
• Harm potential? Moderate

Classification: HIGH

Example 3: Bias in Hiring AI

Scenario: Analysis reveals hiring AI has been systematically rating female candidates lower. Has affected hiring decisions for 6 months.

Classification process:

• Active harm? YES (ongoing discriminatory decisions)
• Sensitive data? Employment decisions (sensitive context)
• Regulatory notification? Potentially (discrimination laws)
• Business disruption? Significant (hiring process affected)
• Harm potential? Significant (career impacts)

Classification: CRITICAL

Example 4: Model Accuracy Drop

Scenario: Internal predictive model accuracy drops from 90% to 85% based on monitoring alerts. No customer-facing impact yet.

Classification process:

• Active harm? No
• Data impact? No
• Regulatory notification? No
• Business disruption? Not yet
• Harm potential? Moderate if unaddressed

Classification: MEDIUM

Classification Procedures

Initial Classification

1. Gather available facts (don't wait for complete information)
2. Walk through decision tree using known information
3. Apply criteria matrix where judgment needed
4. When uncertain, classify higher
5. Document classification reasoning
6. Communicate classification to response team

Reclassification

Initial classification is preliminary. Reassess as information develops:

Reclassification triggers:

• New information changes impact assessment
• Scope expands or contracts
• Root cause identified that changes understanding
• External factors change (media attention, regulatory inquiry)

Reclassification process:

1. Identify trigger for reclassification
2. Re-walk classification criteria with new information
3. Adjust response if severity changes
4. Document reclassification and reasoning
5. Communicate change to stakeholders

Common Failure Modes

1. Under-Classification

Serious incidents classified as routine, causing delayed response. When in doubt, classify higher.

2. Over-Classification

Every incident treated as critical, causing alert fatigue and resource exhaustion. Use criteria objectively.

3. Inconsistent Classification

Different responders classify similar incidents differently. Use documented criteria, not gut feel.

4. Static Classification

Failing to reclassify as information develops. Build in reassessment points.

5. Missing Categories

AI-specific incident types (bias, model failure) not in classification scheme. Ensure all AI incident types are covered.

6. Unclear Thresholds

"Significant impact" is subjective without defined thresholds. Quantify criteria where possible.

Implementation Checklist

• Define incident categories relevant to your AI systems
• Establish severity levels with clear definitions
• Create quantified criteria for each level
• Build decision tree for classification
• Define response requirements per severity
• Train responders on classification process
• Create classification templates/tools
• Establish reclassification procedures
• Test classification with scenarios
• Review and calibrate regularly

Metrics to Track

Frequently Asked Questions

Who should perform initial classification?

The first responder or on-call person should perform initial classification using the framework. They can escalate to the Incident Commander for confirmation on borderline cases.

What if an incident spans multiple categories?

Classify based on the highest-impact category. An incident that's both a data breach (HIGH) and policy violation (LOW) should be classified as HIGH.

How do we handle near-misses?

Near-misses (incidents that almost happened) should be logged at LOW severity but reviewed for lessons learned. They may indicate systemic issues.

Should we publish our classification criteria internally?

Yes—transparency helps everyone understand what constitutes incidents at different levels. This improves reporting and appropriate escalation.

How often should we review and update classification criteria?

After each significant incident (did criteria work?) and at least annually. Update when new AI systems are deployed or regulations change.

Taking Action

Effective incident classification is the foundation of effective response. It ensures critical incidents get urgent attention while routine issues follow standard processes.

Build your classification framework now, when you have time to think clearly—not in the chaos of an active incident.

Ready to build your AI incident classification capability?

Pertama Partners helps organisations develop AI-specific incident classification frameworks. Our AI Readiness Audit includes incident response capability assessment.

Book an AI Readiness Audit →

References

1. NIST. (2023). Incident Handling Guide (SP 800-61).
2. ITIL. (2024). Incident Management Practice Guide.
3. ENISA. (2024). Good Practice Guide for Incident Management.
4. ISO/IEC 27035. Information Security Incident Management.
5. Singapore PDPC. (2024). Data Breach Management Guide.