Data Provenance and Transparency

[Applicable to all providers]
1. Establish standards and guidelines to track the origin, history, and transformations of data. This should include tracking data from its source through all stages of processing, storage and usage.

2. Identify and document all data sources, along with associated metadata and contextual information. 

3. Implement mechanisms to automatically track data lineage and ensure data integrity through immutable logs and data versioning. Use digital signatures, checksums, or hash values to validate the integrity of incoming data.

4. Implement mechanisms to track changes and updates to data.

5. Establish and implement processes to make data sources available according to legal and regulatory requirements.

6. Implement access controls to ensure only authorized personnel can access data.

7. Establish and implement measures to track and manage consent and licensing agreements for data use.

8. Implement data catalogs to manage and audit data access and usage.

9. Establish and implement data handling processes in accordance with data protection laws and regulations, such as GDPR, HIPAA data traceability, AI model traceability under the EU AI act.

10. Perform regular audits to ensure compliance with legal and regulatory requirements.

11. Establish processes to inform stakeholders about data access policies and any changes to them.

12. Document all data transformations, such as imputation, normalization, and encoding.

13. Implement procedures to assign data stewardship roles and implement automated tracking systems.Protect audit logs from unauthorized access or tampering.

14. Implement monitoring and anomaly detection to enforce compliance and adjust processes.

15. Document data gaps and shortcomings that prevent compliance with regulations.

16. Implement data provenance checks as part of data quality and governance audits.

17.Establish mechanisms to integrate provenance tracking with data governance and ETL tools.

18.Implement segregation of duties among users who manage, consume, and audit provenance data to reduce fraud or manipulation risk.

19. Establish a process to securely dispose of provenance records when no longer needed, in alignment with organizational data lifecyle policies.

1. Verify that all infrastructure-level data sources (e.g., logs, metrics, model checkpoints) are documented with source, type, and format.

2. Verify that lineage information is maintained for operational data, showing data flow from ingestion to storage or processing.

3. Verify that dictionaries or schemas exist for metadata and logs captured during AI processing.

4. Verify that provenance tracking includes infrastructure actions such as resource provisioning, access history, and pipeline changes.

5. Verify that automated systems monitor changes to infrastructure datasets and logs (e.g., audit logs, object storage access).

6. Verify that system-level controls ensure the integrity of data and metadata in transit and at rest.

7. Verify that operational processes handle data volume growth, privacy-sensitive logs, and infrastructure-specific complexities.

8. Verify that infrastructure-level data practices comply with laws and cloud service obligations (e.g., data residency, retention).

9. Verify that encryption and granular access controls are in place for all customer and operational data.

10. Verify that data cleanup and deletion are performed according to retention schedules.

11. Verify that infrastructure personnel are trained in secure data lifecycle handling and monitoring practices.

12. Verify that metadata about AI data processing and resource usage can be produced for audit or forensic purposes.

13. Verify that versioning is applied to deployment templates, pipelines, and logs associated with infrastructure-level AI workloads.

14. Verify that disclosure protocols are defined for infrastructure data and comply with legal and contractual frameworks.