Change Management Baseline

[All Actors]
1. Identify and define critical baseline elements that must be monitored for deviation, including system configurations, AI model versions, training datasets, hyperparameters, infrastructure specifications, and access control settings.

2. Maintain version-controlled records of all approved baselines, ensuring traceability and auditability of original configuration states.

3. Implement automated tools to detect configuration drift or unauthorized deviations from the approved baselines in real-time or near real-time.

4. Establish materiality threshold criteria for what constitutes a material deviation that warrants investigation, rollback or emergency change.

5. Document procedures for responding to baseline deviations, including root cause analysis, corrective actions, and optional incident escalation.

6. Ensure deviation detection mechanisms are tested periodically and integrated with the organization’s broader change control and security operations workflows.

1. Inquiring with Control Owners
 
1.1 Interview key personnel to understand if the AI Infrastructure Provider has established and maintains effective change management baselines for all relevant authorized changes to computational resources, storage systems, and network capabilities used for training, fine-tuning, and deploying AI models.

2. Obtaining and Verifying the Population of Infrastructure Records

2.1 Infrastructure Component Population Definition: Define the complete population of records for testing, including: inventory of AI accelerator hardware (GPUs, TPUs, specialized accelerators), compute server configurations and specifications, hardware driver and firmware versions, AI framework optimizations and libraries, storage system configurations and performance profiles, network topology and interconnect specifications, resource management and scheduling policies, container images and virtualization configurations, infrastructure-as-code templates, cluster management software versions, and performance tuning parameters and configurations.

2.2 Perform procedures to verify the completeness and accuracy of the obtained population.

3. Inspection of Infrastructure Evidence

3.1 Infrastructure Baseline Scope Definition: Verify that clear documentation exists defining what constitutes a baseline for infrastructure components. For Compute Components: server hardware specifications, GPU/TPU/accelerator configurations, CUDA/ROCm/driver versions and compatibility, CPU optimization settings, and memory configuration and management. For Storage Components: storage system architectures and configurations, I/O optimization parameters, data caching strategies, performance tuning configurations, and data transfer protocols and settings. For Network Components: network topology and bandwidth specifications, interconnect configurations, network protocol optimizations for AI workloads, load balancing configurations, and quality of service policies. For Platform Components: container runtime versions and configurations, orchestration platform settings, resource scheduling algorithms and weights, quota management policies, and multi-tenancy isolation controls.

3.2 Infrastructure Management Tool Assessment: Examine the implementation and usage of: version control for infrastructure-as-code templates, configuration management databases (CMDBs), hardware monitoring and telemetry systems, resource management and scheduling platforms, performance profiling and optimization tools, and automated provisioning and deployment systems.

3.3 Infrastructure Sample-Based Verification: Select a representative sample of infrastructure components and verify: baseline establishment documentation for each component, version control implementation for hardware and software configurations, compatibility testing between component versions, change history completeness across configuration changes, approval documentation for infrastructure updates, performance testing before and after changes, and 
regression testing documentation.

3.4 Infrastructure Baseline Integrity Controls: Review controls protecting infrastructure baseline integrity: access controls to infrastructure configuration repositories, approval workflows for hardware and software changes, configuration integrity verification mechanisms, prevention controls for unauthorized infrastructure modifications, detection controls for identifying unexpected configuration changes, and audit trails for infrastructure modifications.

3.5 Infrastructure Version Change Controls: Trace a sample of infrastructure changes to verify: change approval documentation, performance and compatibility evaluation reports, corresponding baseline updates in configuration management systems, documentation of dependency changes and impacts, 
customer communication regarding infrastructure updates, and backward compatibility assessments and migration paths.

3.6 Infrastructure Environment Baselines: Verify infrastructure environment baseline management through: firmware and driver version control, operating system and kernel parameter management, library and runtime environment versioning, containerization and image management, infrastructure-as-code template versioning, and configuration drift detection mechanisms.

3.7 Integration Between Infrastructure Baseline Systems: Assess how different infrastructure baseline management systems are integrated: connections between hardware inventory and configuration management, integration between infrastructure monitoring and change management, synchronization between capacity planning and deployment systems, coordination of infrastructure updates across dependent components, and management of compatibility between hardware, drivers, and frameworks.

3.8 Other Baseline Effectiveness Assessment Considerations: Consider the following other assessments: evaluate how well baselines ensure reliability of AI infrastructure, assess the completeness of baseline coverage across all infrastructure components, determine if baseline granularity is appropriate for infrastructure complexity, review instances where infrastructure baseline controls failed to prevent issues, evaluate performance impact assessment effectiveness, and assess effectiveness of infrastructure deprecation procedures.