Making progress with AI governance (Part 2): procuring an AI system
04/06/26 – In Part 1 of Making progress with AI governance we looked at the key elements which are likely to form part of an organisation’s cross-functional AI Policy. In this Part 2 we focus on the procurement of an AI system (which could be generative and/or agentic) and consider some of the key questions and issues that a potential customer may want to keep in mind.
Due diligence
Questions that the customer may want to ask as part of the procurement process:
- What third party service providers does the supplier rely on, including LLM provider(s)? Generally, how transparent is the supplier about its supply chain and third-party dependencies?
- What data is the AI system trained on? Has the supplier secured all necessary rights for the data, both for past and future training? Will the supplier want to use some or all of the customer’s data and/or output data for further training? If so, what confidentiality and anonymisation safeguards are available?
- Can the supplier demonstrate that the AI system is free of bias or meets acceptable bias testing benchmarks?
- Does the supplier have ISO 27001 security certification and/or a SOC2 (ideally Type 2) attestation report?
- Are the arrangements between the supplier and its service providers compliant with UK data protection law, including rules on international data transfers?
- What is the supplier’s data retention policy, including post-contract deletion timelines?
- Does the supplier maintain PI and cyber insurance with adequate indemnity limits?
Supply contract
When negotiating the supply contract, the customer may want to pay particular attention to the following areas:
Ownership of AI output
Whether AI output can be copyright protected under English law is still a moot point. But to the extent copyright does subsist in the output, the supplier and the customer can agree that ownership in the copyright (and any other IP) vests in the customer.
If the supplier wants to use the customer’s data and output data for further training of the AI system, and therefore for the benefit of other customers, the customer will need to consider whether this is acceptable from legal and commercial perspectives. If it is, the customer will want to consider whether use by the supplier (and potentially by third parties) needs to be subject to anonymisation and confidentiality safeguards. The customer may also want to consider whether some form of compensation should be payable for its contribution to the development of the AI system.
IP Infringement
Infringement of IP may arise in two areas:
- The AI system generates outputs which infringe third party IP rights, e.g. instead of coding from scratch the GenAI tool opts to re-use proprietary software code that it finds online. When I asked Gemini about the risk of Claude Code doing this, Gemini replied that Claude Code (like all other GenAI) may generate code that “incorporates, resembles, or is inspired by third-party licensed software” but didn’t venture a view how likely this is. Gemini did however suggest that to mitigate the risk software developers may want to “Run software composition analysis (SCA) or license scanning tools as part of [their] CI/CD pipeline to identify any third-party code that has been inadvertently incorporated“.
- The training of the AI system has resulted in the infringement of the copyright in the material used as training data. Although the lawfulness of using copyrighted works for LLM training is still subject to extensive litigation and government review, the wind has been blowing in favour of the LLM providers with several courts in the U.S. supporting the view that the intermediate, and allegedly infringing, copies created in the course of training LLMs constitutes fair use (see for example Judge Alsup’s comments in Bartz v. Anthropic (pp 13, 14)).
If the supplier is reluctant to give indemnities for 3rd party infringement claims, arguing that infringement by LLMs is out their control, the customer may want to point out that most of the LLM providers offer robust IP infringement indemnities as part of their Terms of Use, certainly for their professional, subscription-based AI models (see for example section K.1 of Anthropic’s Commercial Terms of Service and section 13.1 of the OpenAI Services Agreement).
Hallucinations and accuracy
As much as it goes against conventional contracting normal, the customer may need to accept that hallucinations are an integral feature of GenAI and that the supplier is unlikely to be able to provide warranties regarding the accuracy and completeness of the AI system’s outputs of the type that are commonplace in SaaS contracts. The exception to this is where the supplier has suggested that its AI system meets accuracy thresholds; if so the customer may want to repurpose these into contractual service levels, supported by meaningful service credits. The customer may also want to negotiate a critical service level failure threshold (e.g. accuracy falls below x% in n consecutive months), resulting in the customer having an early termination right.
Bias and discrimination
Depending on the nature of the AI system, the types of data on which it was trained, and how the AI system will be deployed, the customer may want contractual commitments regarding bias and discrimination, and the effectiveness of the AI system’s guardrails. For example if the AI system is to be used for screening and filtering CVs, the customer may want to require the supplier to measure rejection rates broken down by protected characteristics (including sex, race, disability, age) on a 6- or 12-monthly basis. The customer can incorporate the agreed bias testing benchmarks as contractual service levels, again supported by meaningful service credits and a critical service level failure/early termination trigger.
EU AI Act
If the customer will be using the AI system in the EU, or if the output of the AI system will be used in the EU, they will want the supplier to not only warrant the AI system’s compliance with the EU AI Act but also to help the customer comply with its own obligations regarding transparency, explainability of output and employee literacy as a deployer of the AI system. In practice the customer will want contractual commitments that the supplier will provide sufficient information and documentation regarding: how the AI system was developed, what data was used to train it, how it works; how the AI system is tested for bias; and how the AI system performs over time.