AI vendor selection
in eight steps

How does an AI vendor selection process avoid being decided by demos or internal preferences?

An AI vendor selection in eight steps starts with the desired outcome and with criteria and weightings set in writing before a single vendor name is on the table. Every vendor then gets the same scenario on representative data of your own, assessors score independently before discussing, and a three-year cost calculation follows that includes integration, upkeep and exit. The selection closes with negotiating on real terms and recording why the other vendors were rejected.

Is price the most important point to negotiate in an AI vendor selection?

No. In an AI vendor selection, a pricing agreement for consumption at growth, recorded exit terms, a performance commitment on your own scenario, and the right to your own quality measurement weigh more heavily. It should also be recorded what happens if the vendor replaces the underlying model.

Does a no-purchase option belong in an AI vendor selection comparison?

Yes. An AI vendor selection should include at least one no-purchase option in the comparison. That option gives a reference point for what the purchase actually adds.

Over how many years are the costs calculated in an AI vendor selection?

Three years is the period over which the total costs are calculated in an AI vendor selection. The calculation includes at least the licence, consumption costs at expected and double volume, one-off integration, ongoing upkeep, training and estimated exit costs.

Source: article by Dave Willemstein. Updated: October 2026.

Why AI selection is different

With ordinary software you can tick off functionality. With AI applications, the core question isn't "can the system do this?" but "how well does it do this on my data, and what happens when it gets it wrong?" That's a fundamentally different question, and the standard RFP isn't built for it.

On top of that, demos in this field are deceptively good. A presentation on carefully chosen examples says little about performance on your messy reality. This process is designed to make that difference visible.

Step 1: Define the decision, not the need

Start with the outcome you want to achieve, expressed in your own process. Not "we're looking for a document-processing platform" but "we want 80% of incoming files complete within a day, with less manual checking than now."

This framing keeps the door open to solutions you don't yet know about, including the possibility that no purchase is needed at all.

Step 2: Set criteria and weightings before you explore the market

This is the most important step, and the one most often skipped. Business, IT, security and procurement decide together which criteria matter and how heavily, before a single vendor name is on the table.

A workable set of categories:

• Functional fit: does it cover your scenarios?
• Performance on your own data: how well does it work on your material?
• Integration: how much work is connecting it to your systems?
• Data and privacy: where does the data sit, what happens to it, is it used for training?
• Manageability: can you adjust thresholds, rules and exceptions yourself?
• Cost: licence, consumption, integration, upkeep, exit.
• Vendor: continuity, roadmap, support in your language and time zone.

Record the weightings in writing and have them signed off. Criteria adjusted after the first demo describe that demo.

Step 3: Build a longlist on fit, not familiarity

Look wider than the three names everyone knows. Look at specialist players in your sector, at what's already built into your existing software landscape (often more than you think), and at the option of assembling it yourself from standard components.

Include at least one "no purchase" option in the comparison. That's not a formality: it gives you a reference point for what the purchase actually adds.

Step 4: Send everyone the same scenario

No open RFP with two hundred questions. Send a concrete scenario: here's a representative set of our data (anonymised), this is the task, this is what we want to see. Same set, same questions, same amount of time for everyone.

The set needs to be representative, not polished. Include the messy cases: incomplete documents, inconsistent formats, edge cases. That's exactly where the difference between vendors shows up, and exactly what every demo leaves out.

Step 5: Score independently, then discuss

Have assessors fill in their scorecard individually before discussing it as a group. Otherwise the senior person's or the enthusiast's opinion dominates, and the information in the dissenting scores disappears.

Focus the discussion on the differences. If two assessors score the same answer very differently, that usually means the criterion isn't sharp enough. You want to know that before you decide.

Step 6: Ask the questions that make the difference

A selection of questions that deliver the most in practice:

• Is our data used to train your models? If so, can we opt out, and what does that cost?
• Which underlying models do you use, and what happens if that provider changes its terms or pricing?
• How do you measure quality, and can we see and reproduce that measurement method?
• What happens when the system gets it wrong? How do we find out, and what's your liability?
• Can we adjust thresholds and rules ourselves, or does that have to go through you?
• How do we get our data and configuration out if we stop, in what format and within what timeframe?
• Which customer you lost last year can we call?

That last question rarely produces a name, but the reaction to it says a lot.

Step 7: Work out the real costs

A three-year TCO, with at least these items: licence, consumption costs at expected and double volume, one-off integration, ongoing upkeep on your side, training, and estimated exit costs.

In practice, this calculation regularly reshuffles the ranking. The vendor with the lowest licence fee often has the most expensive integration, and consumption costs that exceed the licence at full rollout.

Build the calculation the same way for every shortlisted vendor and have procurement check it. A TCO that only the consultant understands won't survive the first critical question.

Step 8: Negotiate on what you actually need

Price isn't the most important point to negotiate. More important are:

• A pricing agreement for consumption at growth, so success doesn't unexpectedly turn out to be expensive.
• Exit terms recorded in writing, including the format and timeframe of data transfer.
• A performance commitment on your scenario, not a generic SLA on uptime.
• The right to run your own quality measurement and use the results internally.
• Clarity on what happens if the vendor replaces the underlying model.

That last point is new compared with classic software and is almost always forgotten. Vendors swap out their underlying models, sometimes without notice, and that can noticeably change performance on your specific task, in either direction. Record that you'll be informed about this and that you can revalidate afterwards.

Finally: document why the others were rejected

Not just why the winner won. The reasons for rejection are most valuable when, a year later, something disappoints and someone asks why vendor B wasn't chosen. Without a record, that becomes a conversation about gut feeling. With one, it's a conversation about criteria you agreed together at the time.