Exception Management

[All Actors]
1. Define a formal exception workflow that specifies: eligibility (regular vs. emergency), required risk assessment, approvers, maximum duration and/or compensating controls.

2. Embed the workflow in change-/config-management tooling (e.g., CI/CD gates, IaC pull-requests) so unauthorised changes are blocked or tagged until the exception is approved.

3. Log every exception with full metadata (requester, reason, scope, approvals, expiry date); store in the same repository used for normal change records.

4. Implement an emergency fast-track that allows rapid change when service impact or security risk is imminent; require retrospective approval and review within organization defined SLA.

5. Review the open-exception register on a regular, risk-based cadence and either close, renew with justification, or convert to standard policy updates.

6. Align all steps with GRC-04 (policy-exception requirements) to ensure single governance across policies, risk assessments, and stakeholder notifications.

1. Inquiry with Control Owners

1.1 Understand Infrastructure Exception Handling Practices: Interview infrastructure operations leaders, hardware engineers, and data center managers responsible for exception handling. Review documented exception policies covering: emergency hardware/firmware updates and infrastructure maintenance, expedited capacity expansion and resource reallocations (e.g., GPU/TPU), disruption response (e.g., network, storage, power, environmental), and post-incident review and documentation requirements. Verify that exception criteria are clearly defined for: emergencies requiring immediate changes (e.g., failures, vulnerabilities), expedited patches or reallocations due to performance constraints, and authorization levels needed based on severity and customer impact.

1.2 Review Exception Process Documentation: Examine procedures and artifacts that detail: exception request templates and approval workflows, risk assessment steps for infrastructure-related exceptions, temporary approval and escalation pathways, required documentation and post-change validation, and exception tracking and status monitoring.

1.3 Assess Emergency Response Protocols: Evaluate documented procedures for handling critical infrastructure events: hardware failures and firmware vulnerabilities, storage/data integrity issues or cache corruption, network fabric disruptions or latency spikes, power, cooling, or physical plant failures, and emergency resource quota adjustments.

1.4 Evaluate Governance and Oversight Structures: Confirm existence of: designated approval authorities and escalation paths, on-call emergency response teams per infrastructure domain, exception review boards and governance charters, executive oversight and GRC-04 alignment, and integration with enterprise risk and incident management. 

2. Define and Verify Population of Exception Records

2.1 Complete Exception Inventory: Obtain a full inventory of exception records, including: emergency hardware/firmware updates, capacity expansion approvals, resource reallocation (e.g., accelerator pooling), and unplanned maintenance and retroactive exceptions.

2.2 Cross-Verify for Completeness: Ensure population accuracy by cross-referencing monitoring alerts and change tickets, incident and escalation records, service status reports and customer impact notifications, post-incident reviews, and risk registers.

3. Exception Sample Selection and Testing

3.1 Select Representative Exceptions: Choose samples that vary by: type (e.g., hardware update, network fix, quota increase), affected infrastructure (compute, storage, network), customer impact (high, medium, low), approval level and timeframe, justification category (performance, failure, security).

3.2 Evaluate Lifecycle of Each Exception: Review the following categories. Justification: clear rationale and urgency documented, evidence from monitoring or capacity thresholds, risk assessment and consideration of alternatives, and fit within defined exception criteria. Approval: approval by appropriate authority (or retroactively for emergencies), conditions/time limitations documented and followed. Implementation: verified through logs or infrastructure management tools, confined to approved scope and components, monitoring and mitigation applied during exception, stakeholder communication documented (e.g., customer alerts). Closure and Follow-up: timely closure and rollback (if applicable), validation tests conducted, lessons captured and documented, reintegration into standard processes completed.

4. Exception Tracking, Governance, and Continuous Improvement

4.1 Assess Tracking and Oversight: Verify centralized tracking of infrastructure exceptions, expiration tracking for temporary approvals, governance reporting and executive visibility, trend analysis and identification of recurring issues, and integration with customer impact and risk reporting. 

4.2 Evaluate Improvement Mechanisms: Assess maturity of the CSP’s improvement processes: regular exception pattern reviews, incident-driven process refinements, reductions in emergency change frequency, improved emergency response calibration, updates to exception criteria as operations evolve, and 
infrastructure architecture adaptations to minimize exceptions. 

From CCM:

1. Verify that the organization establishes and documents mandatory configuration settings for information technology products employed within the information system, as determined by adoption of the latest suitable security configuration baselines.
2. Confirm that the process identifies, documents, and approves exceptions from the mandatory established configuration settings for individual components based on explicit operational requirements.
3. Determine that the organization monitors and controls changes to the configuration settings in accordance with organizational policy and procedures.