A field guide to evaluating AI vendors

The market is currently flooded with vendors promising to remake your business with AI. Most of them are selling thin wrappers around OpenAI’s API, and many lack the fundamental software engineering discipline required to build durable systems.

When evaluating an AI vendor, ignore their demo. Ask how they handle evaluations. Ask what happens when the underlying model’s API goes down. Ask how they manage prompts in version control. Ask how they measure hallucinations.

A competent vendor will have boring, rigorous answers to these questions. They will talk about CI/CD, evaluation datasets, fallback mechanisms, and error rates. If they talk about magic, look elsewhere.