Our previous post introduced the three-layer framework that structures the scope of AI assurance: the management system, the technical stack, and real-world impact. Now we go deeper and examine: What are the specific objects a practitioner must actually examine, and what does “assuring” them actually entail?
Layer 1: The Organisational Operating System
Assuring the management system of a deployer means examining four key objects: 1) their governance and management system, 2) their monitoring and maintenance processes, 3) their assurance governance and process, and 4) their assurance workforce and competence management.
The primary evidence artefacts in this layer are organisational documents: AI governance policies, risk management frameworks, compliance registers, audit reports, training curricula, competence matrices, and org charts defining roles and responsibilities.
Assurance activities in this layer must work through three levels of scrutiny. First, do these objects exist and are they sufficiently defined? Does the organisation have a risk management process at all? Have roles and responsibilities for AI governance been assigned? Second, are these objects of adequate quality? Are compliance registers accurate, complete, and kept up to date? Are policies comprehensive and appropriate for the organisation's AI risk profile? An AI tool assessing benefits eligibility in public administration deserves closer scrutiny and risk mitigation than an enterprise chatbot suggesting employee wellness activities. Third, are the objects actively managed over time? How are processes updated as the regulatory landscape evolves? How are learnings from incidents integrated into governance frameworks?
In practice, this means looking at things like: whether a risk management framework systematically identifies risks across the full AI lifecycle, meaningfully justifies risk tolerances, and produces mitigation measures that are demonstrably followed through; whether monitoring processes define clear triggers for re-validating deployed systems and whether those triggers are actually acted upon; whether supplier assurance protocols cover the right vendors at the right depth, or whether critical upstream dependencies are being waved through unchecked; and whether those performing assurance tasks hold the right qualifications, manage conflicts of interest appropriately, and maintain their competence as the field evolves.
The critical point about this layer is that good documentation is necessary but not sufficient. Assurance here is fundamentally about evidence of implementation - a polished policy manual with no evidence of use tells an assurer very little.
Layer 2: The Technical Stack
Assuring the technical stack means examining six objects, best understood in three clusters: 1) the AI model and algorithm and the AI system as a whole, capturing what the AI system does and how it behaves; 2) the data assets and compute infrastructure, addressing what the AI system is built on; and 3) the technical documentation set and the deployed solution in its specific application context, covering documentation and implementation in practice.
The evidence artefacts here are considerably more varied than in Layer 1: model cards, architecture diagrams, training configuration records, benchmark and evaluation results, red-team reports, dataset documentation, security audit reports, infrastructure and usage logs, and deployment configuration files, among others.
Assurance activities must again work at multiple levels. Do the necessary artefacts exist - is there a model card, a datasheet, a documented ODD? Are they of adequate quality - is the model card complete and accurate, does the datasheet cover known biases and limitations, is the ODD specific enough to be meaningful? And are they maintained - are documentation and configurations updated when the system changes, and are those changes traceable?
In practice, this means examining things like: whether model architecture and training configuration are documented with enough clarity to support reproducibility and audit; whether guardrails and alignment mechanisms are tested for robustness against circumvention, not just described in policy; whether training and evaluation datasets are assessed for representativeness across relevant subgroups, including rare but high-impact edge cases; whether infrastructure logging is sufficient to support forensic investigation of incidents; whether local deployment configurations - prompts, thresholds, access controls - are consistent with the provider's instructions for use and the deployer's own risk assessment; and whether the technical documentation set is kept current as the system evolves, rather than reflecting only its state at launch.
The key insight across this layer is that assuring the technical stack is about far more than the AI model. Every component involved in training and operation, every deployment context, and the management of all these artefacts over time falls within scope.
Layer 3: What Actually Happens
Assuring AI system impact means examining three key objects: 1) the real-world impact of the deployed system, 2) human-in-the-loop and operator competence in practice, and 3) stakeholder engagement, complaints, and redress mechanisms as they actually function.
The primary evidence artefacts here are outputs and records rather than designs: post-deployment impact assessments, incident and complaint logs, case files and redress outcomes, operator override and escalation records, worker surveys, and independent evaluation reports.
Assurance activities in this layer must ask first whether these objects are generating evidence at all - are impacts systematically measured, are incidents captured across relevant channels, are complaints being logged with sufficient granularity? Second, is the quality of that evidence adequate - are impact assessments methodologically robust, are incident analyses identifying root causes rather than proximate ones, are redress outcomes consistent and proportionate? Third, is the evidence being used - are patterns in complaints feeding back into system improvements, are operator override logs informing retraining decisions, are impact assessment findings leading to concrete changes in deployment practice?
In practice, this means examining things like: whether post-deployment monitoring detects fairness issues across intersectional groups, not just aggregate performance metrics (a chatbot that very rarely uses offensive slurs may appear safe if testing focuses on averages); whether human oversight arrangements give operators true agency to intervene - including the practical ability to pause or override the system - or whether workload and incentive structures make rubber-stamping the path of least resistance; and whether complaints procedures are realistically accessible to those most likely to be harmed, including people with limited digital literacy or those who may not know an AI system was involved in a decision affecting them.
The critical insight in this layer is that the existence of procedures is not sufficient. Inaccessible redress and nominal human oversight are among the most common governance failures in practice, and they are often invisible and unnoticed until a serious incident or a lawsuit draws attention to them. The central question in assuring this layer is not "does a process exist?" but "how well does it serve the people it is intended to protect, and does it generate meaningful change, even when that change comes at commercial cost?"
Using the Framework as a Gap Analysis Tool
The three layers and their objects provide a concrete basis for identifying blind spots in any existing assurance approach: which objects are already covered, which are missing entirely, and which are addressed only at the surface. These past two blog posts help assess if the AI assurance ecosystem is covering all the right things. Future posts in this series will turn to who does the assuring and how it gets done in practice across the ecosystem.