Detection of Baseline Deviation

[All Actors]
1. Define baseline states and metrics for every relevant layer: configurations, infrastructure, service behaviour, model-performance KPIs and security settings.

2. Continuously monitor telemetry and logs against those baselines by using rules-based or ML-driven detectors (e.g., SIEM, APM, drift-detection services).

3. Generate automated alerts (severity, owner, escalation path) as soon as deviations exceed predefined thresholds.

4. Record every deviation event with root cause, corrective action and outcome; store results in a change / incident repository for audit and learning.

5. Review and tune baselines, thresholds and detection logic on a regular basis or whenever major incidents, architectural change or new threat intelligence.

1. Inquiry with Control Owners

1.1 Interview monitoring and operations personnel responsible for detecting changes to AI infrastructure. Obtain and review the organization's monitoring strategies, alert thresholds, and notification workflows for: AI accelerator (GPU/TPU) performance characteristics, distributed computing environment configurations, high-performance storage system metrics, specialized networking fabric performance, hardware driver and firmware versions, and resource allocation and scheduling policies. Verify the existence of documented detection mechanisms for: hardware performance degradation patterns, driver compatibility or stability issues, storage throughput and latency deviations, network fabric performance regression, resource contention and scheduling anomalies, and infrastructure capacity constraints. Identify monitoring tools used for: hardware telemetry collection and analysis, accelerator-specific performance profiling, storage I/O pattern monitoring, network packet flow analysis, resource utilization heat mapping, and distributed system synchronization monitoring.

1.2 Review Notification and Response Procedures: Examine documentation describing notification pathways when infrastructure issues are detected. Understand escalation procedures based on customer impact and resource criticality. Verify integration between detection systems and infrastructure engineering teams. Assess emergency response capabilities for high-severity infrastructure incidents impacting multiple customers. Review response playbooks for different types of infrastructure-related issues: accelerator hardware performance degradation, storage system throughput or latency issues, network fabric congestion or packet loss, resource scheduler inefficiencies, driver or firmware compatibility problems, and distributed system synchronization failures.

2. Obtaining and Verifying the Population of Records

2.1 Define the complete population of monitoring records by inventorying monitoring systems for AI infrastructure, including hardware telemetry collection platforms, accelerator (GPU/TPU) monitoring systems, storage performance tracking tools, network fabric monitoring infrastructure, resource manager logging and metrics, virtualization and container monitoring, driver and firmware version tracking, and capacity planning and forecasting systems. 

2.2 Verify completeness of the population by cross-referencing monitoring coverage against the inventory of AI infrastructure component. Verify monitoring covers all regions, availability zones, and deployment models.

3. Inspection of Evidence

3.1 Monitoring System Verification: Verify that monitoring systems are configured to detect deviations in the following categories. For AI Accelerator Performance: computational throughput (FLOPS, operations/second), memory bandwidth and utilization, power consumption and thermal characteristics, training/inference benchmark performance, error rates and correction events, and utilization efficiency across workloads. For Storage System Characteristics: I/O operations per second (IOPS), throughput (GB/s) for sequential access, latency distributions for different operation types, queue depths and blocking operations, cache hit rates and effectiveness, and storage capacity utilization trends. For Network Fabric Performance: bandwidth utilization for collective operations, latency profiles for inter-node communication, packet loss rates and retransmissions, congestion events and back pressure signals, quality of service enforcement effectiveness, and network topology efficiency. For Resource Management Effectiveness: allocation efficiency and resource fragmentation, scheduling fairness across customers, queue wait times for resource types, preemption rates and impact, resource affinity and locality effectiveness, and quota enforcement accuracy.

3.2 Alert Configuration Assessment: Examine alert configuration to verify: tiered thresholds based on resource type and cost, graduated alerting based on deviation persistence, different sensitivity for different customer tiers, correlation between related infrastructure metrics, seasonality and workload-aware baselines, forecasting-based proactive notifications, and hardware generation-specific threshold adjustments.

3.3 Sample-Based Testing of Detection Capabilities: Select a representative sample of infrastructure components and perform controlled tests: induce synthetic accelerator load patterns, create storage I/O contention scenarios, simulate network congestion conditions, generate resource allocation imbalances, inject driver or firmware compatibility issues, and test distributed system synchronization edge cases. Verify that monitoring systems: accurately detect the simulated issues, generate appropriate alerts with correct severity, include sufficient diagnostic context, trigger within expected timeframes, follow defined notification workflows, and properly identify fault domains and impact scope.

3.4 Alert Notification Workflow Verification: Trace the notification path for different types of infrastructure issues: initial detection and enrichment with telemetry, routing to appropriate infrastructure teams, escalation for customer-impacting issues, hardware vendor coordination workflows, customer notification processes, maintenance scheduling integration, and cross-region incident coordination. 

3.5 Response Effectiveness Evaluation: Review historical infrastructure incidents to evaluate: time to detect performance deviations, quality of diagnostic information, response time to critical infrastructure issues, effectiveness of remediation actions, 
customer impact minimization, vendor coordination effectiveness, and root cause analysis thoroughness.

3.6 Automated Remediation Assessment: Verify implementation of automated remediation for common issues: accelerator thermal throttling management, storage path failover mechanisms, network route optimization and reconfiguration, resource rebalancing and workload migration, driver rollback capabilities, self-healing distributed system recovery, and preemptive resource capacity expansion.

3.7 Integration with Capacity Planning: Assess how detection systems feed into capacity management: early warning indicators for capacity constraints, trend analysis for resource utilization, predictive analytics for hardware procurement, seasonal demand pattern recognition, capacity risk assessment automation, hardware lifecycle and refresh monitoring, and geographic expansion trigger indicators.