How to choose an AI agent developer: 10 questions to ask
The most reliable way to choose a team to build an AI agent is to ask about method rather than be guided by the demo. How success will be measured, where the data goes, what happens when the model is wrong, which steps need human approval, how cost is capped and what happens if the model changes. A team that answers these concretely, and in writing, understands the risks of the work.
This piece is a list of criteria. Which team suits you is your decision, made by listening to the answers. Questions that apply to any software project — who owns the source code, confidentiality agreements, changes in scope — are collected in how to choose a software company. Here we cover only the questions specific to AI agent projects.
Why does an AI project need different questions?
Conventional software gives the same output for the same input every time. An AI agent works on probabilities: it may answer the same question slightly differently, it can be confidently wrong, and it may use the tools it is given in an unexpected order.
That difference changes the selection criteria. “Can you build it?” almost always gets a yes. The real question is how the team measures how well the agent works, and how they have designed for the moments it gets things wrong. We describe what an agent is, and its limits, in what is an AI agent.
Ten questions to ask
1. “Does this job really need AI?”
A good team will happily tell you that part of the work could be done more cheaply and reliably with rule-based automation. An answer of “yes, an agent” to every request may mean the work has not been examined. You can draw the line yourself using the table in multi-agent, single agent or conventional automation?
2. “How will we measure success? Who prepares the test set?”
You should hear a concrete method: a test set of real examples, an expected result for each, a way of scoring, and tests rerun after every change. “We’ll see as we use it” is not a measurement method. How a measurement set-up is built is covered in how to evaluate an AI system.
3. “Do you start with a pilot? When would the pilot be stopped?”
A team that proposes a narrow pilot with a fixed duration and a stopping threshold agreed in advance knows how to manage uncertainty. We set out the parts of a pilot in how to start an AI project in your company.
4. “Which model, and which country’s servers, will our data go to?”
The answer should be clear: which provider, which region, how long the data stays there, and whether the provider uses it to train models. If personal data leaves the country, the legal position needs discussing too. For Turkey, we cover it with reference to the data protection authority’s guidance in personal data in an in-house AI assistant.
5. “What happens when the model gives a wrong answer?”
Look for an answer that accepts hallucination cannot be eliminated but lists concrete ways to shrink it: tying answers to sources, showing those sources, testing that the model says “I don’t know” when it should, and validating output in code. “Our system doesn’t make mistakes” is a warning sign. The methods are in hallucination, data privacy and human approval.
6. “Which steps will require human approval?”
The team should propose a table that classifies the agent’s steps: which run without approval, which are prepared as drafts for a person, and which always need sign-off. For steps that are hard to undo — payments, deletions, anything sent outside the company — the answer should be unambiguous.
7. “How are the agent’s permissions limited? Is every step logged?”
Expect the tools the agent may use to be listed one by one, with “no” as the default, and every step recorded with its reasoning. Ask too how the agent is protected from hidden instructions in the emails or web pages it reads. That risk arises in any project where an agent handles outside content.
8. “How do you cap model costs?”
Model usage is a monthly cost, and in agent systems it grows with the number of steps. A good answer covers step and spending limits, matching the model to the task, and how usage will be monitored. We break down the cost items in what determines the cost of an AI project.
9. “What happens if the provider changes or retires the model?”
Providers update models, retire old versions and change prices and terms. Ask how tightly the system is tied to one provider, what moving to another model would take, and whether the tests would be rerun on the switch. Defining tools through a standard protocol such as MCP is one way to reduce that dependency.
10. “Once it is live, who monitors what?”
After launch, accuracy, cost and usage should all be watched. The answer should say who does the monitoring, what triggers an alert and how the system can be stopped from a single point. The scope and length of maintenance belong to this question too.
How to weigh the answers
The table below is for laying your interview notes side by side. Answers on the left suggest a method is in place; answers on the right suggest one is missing.
| Topic | Reassuring answer | Warning sign |
|---|---|---|
| Need | “This part is rule-based automation” | An agent for everything |
| Measurement | Test set, scoring, threshold | “We’ll see as we go” |
| Starting point | Time-boxed pilot, stopping threshold | Straight to the full system |
| Data | Provider, region, retention in writing | “Your data is safe” and nothing more |
| Errors | Source grounding, “I don’t know” tests | “Our system doesn’t make mistakes” |
| Approval | Step-by-step approval table | Approval left for later |
| Permissions | Tool list, logging, limits on outside content | Starting with broad access |
| Cost | Step and spending limits, monitoring | A vague “depends on usage” |
| Dependency | A plan for switching models | Architecture locked to one model |
| Operation | Monitoring, alerts, a stop point | Relationship ends at handover |
No team has to be flawless on every line. But asking whether an answer will be given in writing turns a verbal reassurance into a concrete commitment.
What should the deliverables include?
In conventional software, deliverables are mostly source code and documentation. An AI agent has a few more parts, and they should be named in the proposal or contract:
- The test set. The examples used for measurement, the expected results and the scoring method. If another team takes the system over later, comparing like with like depends on this set.
- Prompts and configuration. The instructions given to the model, the tool definitions and the settings. Much of the agent’s behaviour is defined in these texts.
- The model provider account. The account that pays for usage, and its access keys, should be opened in the company’s own name. If the account sits with the developer, so do the bills, the logs and the access.
- Logs. Where the agent’s step logs are kept, for how long, and how they will be handed over.
- An operating note. How to stop the system, where spending limits are set and what must be retested when the model version changes.
How to prepare for the meeting
For the questions to work, a few things need to be ready on your side:
- A one-paragraph description of the task. Which team does which job, and how often?
- Real examples. A few examples from the task’s history, each with its correct outcome — masked, if personal data is involved.
- The systems to connect. For each, is read access enough, or is write access needed?
- Red lines. Which data must never leave the company, which step must never be automatic.
- A decision-maker. Who inside the company is accountable for the outcome?
This preparation lets you compare answers from different teams on the same footing. How to read the proposals that follow is covered in how to read a software proposal.
Frequently asked questions
What is the most important criterion when choosing an AI agent developer?
The measurement method. A team that can explain concretely how it will measure the agent’s accuracy on real examples will usually answer the other questions soundly too. Without measurement there is neither improvement nor a basis for decisions.
Is an impressive demo enough evidence?
No. Demos are built on chosen examples. Bring a few real examples from your own work to the meeting and ask how the system would handle them. The real evidence is a measurement made with your data.
A large agency or a small team?
What matters more than size is who you will actually work with on the project, and how those people answer these ten questions. Ask whether the person in the meeting is the one who will run the project.
How can I judge a team’s AI experience?
Ask how they measured an AI system they have already put live, which mistakes they ran into and what they changed. A team that can tell a specific story about something going wrong has seen a system in real use.
If you would like to talk these questions through, see our AI systems architecture page or write to us.