Application Security Metrics
Specification
Define and implement technical and operational metrics in alignment with business objectives, security requirements, and compliance obligations.
Threat coverage
Architectural relevance
Lifecycle
Resource provisioning
Guardrails, Design
Validation/Red Teaming, Evaluation, Re-evaluation
AI applications, AI Services supply chain
Continuous monitoring, Continuous improvement, Operations
Archiving
Ownership / SSRM
PI
Shared Cloud Service Provider-Model Provider (Shared CSP-MP)
The CSP and MP are jointly responsible and accountable for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer.
Model
Owned by the Model Provider (MP)
The model provider (MP) designs, develops, and implements the control as part of their services or products to mitigate security, privacy, or compliance risks associated with the Large Language Model (LLM). Model Providers are entities that develop, train, and distribute foundational and fine-tuned AI models for various applications. They create the underlying AI capabilities that other actors build upon. Model Providers are responsible for model architecture, training methodologies, performance characteristics, and documentation of capabilities and limitations. They operate at the foundation layer of the AI stack and may provide direct API access to their models. Examples: OpenAI (GPT, DALL-E, Whisper), Anthropic(Claude), Google(Gemini), Meta(Llama), as well as any customized model.
Orchestrated
Shared Orchestrated Service Provider-Application Provider (Shared OSP-AP)
The OSP and AP are jointly responsible and accountable for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer.
Application
Owned by the Application Provider (AP)
The Application Provider (AP) is responsible for the design, development, implementation, and enforcement of the control to mitigate security, privacy, or compliance risks associated with Large Language Model (LLM)/GenAI technologies in the context of the services or products they develop and offer. The AP is responsible and accountable for the implementation of the control within its own infrastructure/environment. If the control has downstream implications on the users/customers, the AP is responsible for enabling the customer and/or upstream partner in the implementation/configuration of the control within their risk management approach. The AP is accountable for carrying out the due diligence on its upstream providers (e.g MPs, Orchestrated Services) to verify that they implement the control as it relates to the service/product develop and offered by the AP. These providers build and offer end-user applications that leverage generative AI models for specific tasks such as content creation, chatbots, code generation, and enterprise automation. These applications are often delivered as software-as-a-service (SaaS) solutions. These providers focus on user interfaces, application logic, domain-specific functionality, and overall user experience rather than underlying model development. Example: OpenAI (GPTs,Assistants), Zapier, CustomGPT, Microsoft Copilot (integrated into Office products), Jasper (AI-driven content generation), Notion AI (AI-enhanced productivity tools), Adobe Firefly (AI-generated media), and AI-powered customer service solutions like Amazon Rufus, as well as any organization that develops its AI-based application internally.
Implementation guidelines
Auditing guidelines
1. Verify that technical and operational metrics are defined and documented. 2. Verify that technical and operational metrics are verifiable (specific and measurable). 3. Assess the technical and operational metrics for sufficiency and relevance (e.g., vulnerability management, access control, data protection, incident response). 4. Verify that the technical and operational metrics are being utilized. 5. Evaluate if the technical and operational metrics are aligned with business objectives, security requirements and compliance requirements. 6. Verify that the technical and operational metrics are being monitored and evaluated (e.g., evidence of monthly tracking trends, adjustments from management reports). 7. Confirm metrics around API usage, access control, resource allocation, and network security boundaries. 8. Validate visibility into automated remediation actions triggered by metric thresholds (e.g., anomaly in tenant resource use). 9. Review internal reports used to assess compliance with ISO, CCM, and NIST 800-53 style frameworks. From CCM: 1. Examine policies and procedures for definition of operational metrics, security, and compliance requirements.
Standards mappings
42001: 9.1 42001: B.6.2.6 27001: 9.1 27701: 5.7.1
Addendum
N/A
Article 15 (2) Article 15 (3) Article 53 and Annex XI Article 55 Article 74 Article 75
Addendum
N/A
MS-2.7-002 MS-2.7-004 MEASURE 3.2 MEASURE 3.3 MEASURE 4.2 MG-3.1-002 MS-2.11-002 MS-2.7-002 MS-2.7-004 MG-3.1-002 MS-2.11-002
Addendum
Missing elements include explicit security baseline metrics for applications. Define baseline security metrics for application security. Define operational security KPIs that align with compliance and industry best practices. Enhance risk tracking for application security to ensure structured data collection, processing, and evaluation.
COM-04
Addendum
C5 looks at information security only, no business objectives and compliance obligations.
AI-CAIQ questions (1)
Are technical and operational metrics defined and implemented in alignment with business objectives, security requirements, and compliance obligations?