TVM-12AI-Specific

Threat Analysis and Modelling

Specification

Define implement and evaluate threat analysis process and procedures to identify, assess and review the threat landscape for Cloud and AI systems. Build threat models according to industry best practices to inform the risk mitigation strategy.

Threat coverage

Model manipulation

Data poisoning

Sensitive data disclosure

Model theft

Model/Service Failure

Insecure supply chain

Insecure apps/plugins

Denial of Service

Loss of governance

Architectural relevance

Physical infrastructure

Network

Compute

Storage

Application

Data

Lifecycle

Preparation

Data collection, Data curation, Data storage

Development

Design, Training, Guardrails

Evaluation

Evaluation, Validation/Red Teaming, Re-evaluation

Deployment

AI Services supply chain, AI applications

Delivery

Operations, Maintenance, Continuous monitoring, Continuous improvement

Retirement

Data deletion

Ownership / SSRM

Shared Cloud Service Provider-Model Provider (Shared CSP-MP)

The CSP and MP are jointly responsible and accountable for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer.

Model

Owned by the Model Provider (MP)

The model provider (MP) designs, develops, and implements the control as part of their services or products to mitigate security, privacy, or compliance risks associated with the Large Language Model (LLM). Model Providers are entities that develop, train, and distribute foundational and fine-tuned AI models for various applications. They create the underlying AI capabilities that other actors build upon. Model Providers are responsible for model architecture, training methodologies, performance characteristics, and documentation of capabilities and limitations. They operate at the foundation layer of the AI stack and may provide direct API access to their models. Examples: OpenAI (GPT, DALL-E, Whisper), Anthropic(Claude), Google(Gemini), Meta(Llama), as well as any customized model.

Orchestrated

Shared Model Provider-Orchestrated Service Provider (Shared MP-OSP)

The MP and OSP are jointly responsible and accountable for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer.

Application

Owned by the Application Provider (AP)

The Application Provider (AP) is responsible for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer. The AP is responsible and accountable for the implementation of the control within its own infrastructure/environment. If the control has downstream implications on the users/customers, the AP is responsible for enabling the customer and/or upstream partner in the implementation/configuration of the control within their risk management approach. The AP is accountable for carrying out the due diligence on its upstream providers (e.g MPs, Orchestrated Services) to verify that they implement the control as it relates to the service/product develop and offered by the AP. These providers build and offer end-user applications that leverage generative AI models for specific tasks such as content creation, chatbots, code generation, and enterprise automation. These applications are often delivered as software-as-a-service (SaaS) solutions. These providers focus on user interfaces, application logic, domain-specific functionality, and overall user experience rather than underlying model development. Example: OpenAI (GPTs,Assistants), Zapier, CustomGPT, Microsoft Copilot (integrated into Office products), Jasper (AI-driven content generation), Notion AI (AI-enhanced productivity tools), Adobe Firefly (AI-generated media), and AI-powered customer service solutions like Amazon Rufus, as well as any organization that develops its AI-based application internally.

Implementation guidelines

[All Actors]
1. Model / System Understanding – document the architecture, data sources, algorithms and use-case context of each AI or cloud component under review.

2. Threat Identification – use recognised frameworks (e.g., CSA GenAI Risk Scenarios, Mitre ATLAS) to enumerate malicious actors, insider threats and unintentional failure modes; create concrete scenarios (data poisoning, model inversion, availability attacks, etc.).

3. Vulnerability Analysis – perform data reviews, code reviews, penetration testing and other assessments to discover weaknesses; rate each finding for severity and likelihood.

4. Risk Analysis & Mitigation Strategy – combine impact and likelihood into a risk matrix, prioritise events and define counter-measures (hardening data pipelines, model-integrity checks, drift / skew monitoring, access controls).

5. Documentation & Governance – keep a transparent record of threat logic, data sources, mitigation plans and moderator / human-in-the-loop rules; review the documentation periodically to reflect regulatory change or new attack vectors.

6. Continuous Monitoring – deploy telemetry and alerting to detect emerging threats; include advanced techniques such as AI honeypots (decoy data lakes, dummy APIs, fake model endpoints) to attract and study attackers.

7. Risk Acceptance & Compensating Controls – when residual risks are formally accepted, document rationale, stakeholder sign-off and compensating measures (enhanced monitoring, restricted access); revisit accepted risks on a scheduled basis.

Auditing guidelines

1. Verify that the Cloud Service Provider (CSP) has defined processes, procedures, and technical measures to systematically identify threats to which AI systems and models are potentially exposed. Ensure that the processes are documented in detail, covering scope, objectives, roles and responsibilities. 

2. Verify that processes, procedures, and technical measures are in place to systematically assess threats to AI systems and models previously identified.

3. Inspect whether the above-mentioned processes, procedures, and technical measures of threat analysis are compliant with relevant regulatory requirements and industry best practices.

4. Verify that countermeasures against identified threats are timely defined, prioritized, accordingly applied, monitored, reviewed and updated by relevant parties.

5. Inspect whether the above-mentioned processes, procedures, and technical measures of threat analysis are monitored against sets of efficacy and efficiency metrics / indicators.

6. Inspect whether the above-mentioned processes, procedures, and technical measures of threat analysis are periodically reviewed and updated by responsible parties.

Standards mappings

ISO 42001Partial Gap

42001: A.6.2.3/B.6.2.3 - Documentation of AI system design and development
42001: A/6.2.6/B.6.2.6 - AI system operation and monitoring
42001: A.7.2/B.7.2 - Data for development and enhancement of AI system
27001: A.5.7 - Threat intelligence

Addendum

Although 42001 and 27001 speaks to performing threat analysis, they do not speak specifically to the TVM-04 topic of establishing process and procedures for threat analysis.

EU AI ActPartial Gap

Article 9 (1)
Article 9 (2)
Article 15 (1)
Annex IV (2) (f)

Addendum

Define or require the use of threat modeling frameworks to include cloud-specific risks or shared responsibility models, periodic review or update of threat models based on evolving threats or real-world incidents, and embedding threat modeling during system architecture or lifecycle stages.

NIST AI 600-1Partial Gap

MS-2.7-001
GV-1.1-001
GV-6.1-004

Addendum

Needs requirement to use formal threat modeling frameworks. No coverage of cloud-specific threat landscape analysis. No mandate to evaluate the effectiveness of the threat analysis process over time.

BSI AIC4No Gap

C4 SR-01
C4 SR-02
C4 SR-03
C4 SR-04
C4 SR-05
C4 SR-06
C4 RE-05
C5 OPS-18

Addendum

N/A

AI-CAIQ questions (2)

TVM-12.1

Are threat analysis processes and procedures defined, implemented, and evaluated to identify, assess, and review the threat landscape for Cloud and AI systems?

TVM-12.2

Are threat models built according to industry best practices to inform the risk mitigation strategy?