Human supervision
Specification
Establish, execute, and assess processes, procedures, and technical measures to ensure human oversight and control of the AI system in compliance with regulatory requirements and organizational risk management.
Threat coverage
Architectural relevance
Lifecycle
Data storage
Guardrails
Validation/Red Teaming, Re-evaluation
Orchestration, AI Services supply chain, AI applications
Operations, Maintenance, Continuous monitoring, Continuous improvement
Not applicable
Ownership / SSRM
PI
Shared Cloud Service Provider-Model Provider (Shared CSP-MP)
The CSP and MP are jointly responsible and accountable for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer.
Model
Owned by the Model Provider (MP)
The model provider (MP) designs, develops, and implements the control as part of their services or products to mitigate security, privacy, or compliance risks associated with the Large Language Model (LLM). Model Providers are entities that develop, train, and distribute foundational and fine-tuned AI models for various applications. They create the underlying AI capabilities that other actors build upon. Model Providers are responsible for model architecture, training methodologies, performance characteristics, and documentation of capabilities and limitations. They operate at the foundation layer of the AI stack and may provide direct API access to their models. Examples: OpenAI (GPT, DALL-E, Whisper), Anthropic(Claude), Google(Gemini), Meta(Llama), as well as any customized model.
Orchestrated
Shared Model Provider-Orchestrated Service Provider (Shared MP-OSP)
The MP and OSP are jointly responsible and accountable for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer.
Application
Owned by the Application Provider (AP)
The Application Provider (AP) is responsible for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer. The AP is responsible and accountable for the implementation of the control within its own infrastructure/environment. If the control has downstream implications on the users/customers, the AP is responsible for enabling the customer and/or upstream partner in the implementation/configuration of the control within their risk management approach. The AP is accountable for carrying out the due diligence on its upstream providers (e.g MPs, Orchestrated Services) to verify that they implement the control as it relates to the service/product develop and offered by the AP. These providers build and offer end-user applications that leverage generative AI models for specific tasks such as content creation, chatbots, code generation, and enterprise automation. These applications are often delivered as software-as-a-service (SaaS) solutions. These providers focus on user interfaces, application logic, domain-specific functionality, and overall user experience rather than underlying model development. Example: OpenAI (GPTs,Assistants), Zapier, CustomGPT, Microsoft Copilot (integrated into Office products), Jasper (AI-driven content generation), Notion AI (AI-enhanced productivity tools), Adobe Firefly (AI-generated media), and AI-powered customer service solutions like Amazon Rufus, as well as any organization that develops its AI-based application internally.
Implementation guidelines
Auditing guidelines
1. Verify that the Cloud Service Provider (CSP) has defined processes, procedures, and technical measures to ensure that AI systems are designed and developed in such a way that human operators can oversee their functioning and intended performance throughout their entire lifecycle. Ensure that the processes are documented in detail, covering scope, objectives, roles and responsibilities. 2. Examine the above-mentioned processes, procedures, and technical measures to confirm their compliance with relevant regulatory requirements and industry best practices. 3. Examine whether the above-mentioned processes, procedures, and technical measures adopt a risk-based approach. 4. Confirm that the above-mentioned processes, procedures, and technical measures are concretely and appropriately implemented by responsible parties over the entire AI systems' lifecycle (from the design and market placement to the maintenance/upgrade and decommission phases). 5. Inspect whether the above-mentioned processes, procedures, and technical measures are monitored against sets of efficacy and efficiency metrics / indicators. 6. Inspect whether the above-mentioned processes, procedures, and technical measures are periodically reviewed and updated by responsible parties.
Standards mappings
42001: B.5.1 AI system risk assessment and treatment 42001: B.5.3 (Documentation of AI system impact assessments) 42001: B.6.1.3 (Processes for responsible design and development of AI systems) 42001: B.5.3 Documentation of AI system impact assessments 42001: B.6.1.4 Accountability and human oversight mechanisms 42001: B.6.2.1 Technical robustness and security 42001: B.7.1.1 Monitoring and review of AI system behavior 42001: B.7.1.2 Incident management for AI systems 42001: B.8.2.1 Compliance with legal and regulatory requirements
Addendum
N/A
Article 14 Article 15 Article 17
Addendum
N/A
GV-3.2-001 GV-4.1-003 MG-2.2-003 MP-2.3-001
Addendum
N/A
C4 PC-02 RE-01 RE-04 C5 PSS-03
Addendum
N/A
AI-CAIQ questions (1)
Are processes, procedures, and technical measures to ensure human oversight and control of the AI system in compliance with regulatory requirements and organizational risk management, established, executed and assessed?