We sit in on a lot of vendor calls with clients. The decks are usually accurate. They are also usually built to avoid a specific set of questions.
We ask the nine questions below. A good vendor answers all nine comfortably and the call gets better afterwards.
1. What does this do when it is not confident?
The answer separates products built for demos from products built for production. “It gives its best guess” is a red flag. “It routes to a queue with the reason attached” is not.
2. Show me the error case
Ask for a live demo on a document that is deliberately messy. A photo of a page. A form with two pages swapped. The vendor’s willingness to do this tells you more than the result does.
3. What is the evaluation methodology behind that accuracy number?
Accuracy figures quoted without a test set definition are marketing. Ask what corpus, how large, who labelled it, and whether it resembles your inputs.
4. Can we run it against our own historical data before we sign?
If the answer is no, understand why. Sometimes there is a legitimate reason and often there is not.
5. Who owns the data we put in, and what is it used for?
Specifically: is it used for training, is it retained after termination, where is it stored, and who at the vendor can see it. Get this in writing rather than on the call.
6. What happens when the underlying model is deprecated?
Every model behind every product has a retirement date. Ask what happened the last time one moved, how much notice customers got, and whether behaviour changed.
7. What does the integration actually require from us?
The demo runs on the vendor’s sample data. Ask what your side has to build, who maintains it, and what breaks when your source system updates.
8. Who else in our sector runs this at our volume?
Ask for a named reference you can call who runs comparable volume, rather than a logo slide.
9. What is this bad at?
The single most useful question in the call. Vendors with a good product have a ready answer and it builds trust. Vendors without one will tell you it handles everything, which is the answer that should worry you.
Why we sit in on these calls
A vendor call goes better when somebody in the room has read the architecture and is not afraid to be unimpressed. That is a large part of what an advisory retainer is used for in practice, and it is usually the cheapest line in the engagement.