Procuring Third-Party AI Solutions: Best Practices & Key Factors for Decision Makers

Before venturing into the AI marketplace, companies must align their AI objectives with their broader organisational goals. What does an organisation want to achieve with AI? Whether automating mundane tasks, deriving insights from data, or pioneering discoveries in a niche field, understanding the ‘why’ of procuring third-party AI solutions is the first step in the procurement journey.

Then, companies must start thinking about integrating the business and technical requirements of the AI system. Failing to do this at an early stage of the procurement process can lead to significant unnecessary downstream costs. If key stakeholders across technical, business, legal and procurement teams are not involved early on to define the performance and risk metrics, it can result in post-deployment modifications, integration challenges, and potential compliance hurdles.

For instance, an AI system that isn’t properly aligned with a company's existing IT landscape can create disruptions, compatibility issues, or even security vulnerabilities. Without input from technical, operational, and compliance experts from the outset, there's a heightened risk of overlooking critical concerns, which can translate to regulatory fines or compromised system efficiency.

Even as the AI landscape evolves, best practices for procuring AI can help provide senior decision-makers clarity in ensuring that they adopt the most suitable third-party AI systems for their organisations. Take the World Economic Forum’s “AI Procurement in a Box” framework as a reference. It is a comprehensive set of guidelines and tools designed to assist governments and businesses to procure AI in an innovative, efficient and ethical way.

One of the central best practices emphasised by the WEF is the need for “algorithms to be interpretable as a means of establishing accountability and contestability”. This means vendors should share information such as what training data was used and which variables have contributed most to the results. Vendors can do so through model cards, results from audit and assurance practices etc.

Other best practices that the WEF framework touches on include the importance of assessing AI systems for potential biases and continuous oversight and iteration of AI systems to keep up to date with changing data landscapes or evolving contexts

Below are several other factors that organisations should consider to achieve AI procurement excellence:

1. Technical AI evaluations — Beyond the marketing jargon and dazzling demos, it’s essential to assess a vendor's actual technical capabilities. This could be done in several ways:
   1. When in discussion with AI vendors, companies could consider pre-procurement technical evaluation of the AI system. At Resaro, we audit at key points where important business decisions are made, for example, in procurement evaluation of different AI vendors. By auditing specific AI applications and their algorithms, we quickly identify the risks, as well as safeguards, that the buyer and vendor need to put in place for a successful and safe deployment of the AI solution.
   2. Running pilot projects can help offer insights into the AI solution’s real-world efficacy. For instance, in 2018, the UK's National Health Service (NHS) initiated pilot projects with Google's DeepMind to test AI-powered diagnostic tools. While the technology showed promise, real-world tests revealed challenges related to data handling and system integration.
   3. Companies can explore scalability assessments to ensure that the AI solutions do not just work in present-day controlled environments but can scale in line with business growth. Take the case study of Dropbox, for example. After its initial phase of relying on third-party AI services, it had to build its machine learning infrastructure to handle growing data volumes to ensure scalability.
2. Emerging regulatory compliance — Given AI’s influence on data and decision-making, more governments are considering ways to govern its use to mitigate risks. In New York City, the Automated Employment Decision Law requires employers that use AI as part of their hiring process to perform an annual audit of their recruitment technology. The aim is to prevent AI bias in corporate hiring that could perpetuate acts of discrimination or unintended biases against certain groups of people (e.g. pregnant women). Companies adopting third-party AI solutions in regulated or soon-to-be regulated jurisdictions should also pay close attention to emerging compliance requirements and design them into the procurement requirements.
3. Continuous monitoring and validation — Companies should be prepared to conduct regular audits, both by internal and third-party experts, to ensure that the system operates optimally. Even after going live, AI systems should also undergo rigorous testing, not just for functionality but for potential risks. Ongoing technical audits and risk assessments serve as objective documentations of the AI performance across its lifecycle and deployment in operational environments.

AI offers a transformative power that can redefine industries. However, its procurement requires a blend of technical understanding and strategic foresight. Senior management and C-suite decision-makers play a pivotal role in ensuring the responsible and effective use of AI. The key components above can help senior decision-makers make better decisions in procuring AI systems.