The case for AI assurance is well-established. What has proven harder is putting it into practice. Organisations procuring AI systems have no reliable, independent basis for knowing whether what they are buying will perform as promised in the conditions it will actually face.
Vendors cannot credibly differentiate good systems from weak ones. And the principles that governance frameworks have spent years developing around fairness, reliability, and robustness remain largely unverifiable in any given deployment.
Titled ‘Pathways for Operationalising AI Assurance’, this paper is the second in Resaro's collaboration with Partnership on AI (PAI) under the Strengthening the AI Assurance Ecosystem initiative. Where the first mapped what a functioning assurance ecosystem requires, this one asks what sits at its centre: how do you actually verify that an AI system is good?
It works through the reasons that question has been so difficult to answer, and puts forward the AI Solutions Quality Index (ASQI) as a framework for doing so, translating quality principles into assessments that are measurable, context-specific, and meaningful to the people who have to act on them.
Three case studies across public administration, legal services, and healthcare show what that looks like when applied to real systems. The findings are instructive. A system can be faithful to its sources and still consistently retrieve the wrong ones. It can meet accuracy targets and still produce an unacceptable rate of missed findings. Generic benchmarks would not have caught either. That is precisely the gap this paper is trying to close.