How to start an AI project in your company: a pilot guide
The soundest way to start an AI project in a company is a small pilot built around a single task. A pilot has five parts: a narrow scope, a success criterion written as a number, data drawn from real examples, a table showing which steps need human approval, and a stopping rule agreed before the pilot begins. With those five in place, “carry on or stop?” can be answered with evidence within a few weeks.
Here we go through the five parts in order, and at the end share a one-page pilot card we recommend filling in before any pilot. Which kinds of work suit a language model is a separate question, covered in our piece on ChatGPT integration for businesses; this piece is about how to trial the task you have chosen and reach a decision.
Why start with a pilot rather than a large project?
AI projects carry more uncertainty than conventional software projects. You cannot know in advance how accurately the model will work with your data, whether people will actually adopt it, or where the cost will settle under real use. A pilot measures those three unknowns in a small, contained area.
The typical outcome of starting big is an impressive demo, months later, that nobody can vouch for in day-to-day work. A pilot is tied from the outset to one question: “Can this task be done, with this data, to this standard?”
1. Scope: which single task?
A pilot’s scope is one repetitive task, done by one team, whose output can be checked. “Improve customer service with AI” is not a scope. “Sort incoming support emails into five categories” is.
A good pilot task:
- Happens often. Dozens of times a day, say. Rare tasks never build up enough examples to measure.
- Produces checkable output. Someone on the team can say in seconds whether a result is right.
- Has recoverable mistakes. A wrong category can be corrected; a wrong payment is hard to undo. Pilots start with the first kind.
- Has an owner. There is a named person who does the work today and will judge the result.
To weigh whether a task suits an AI agent or conventional automation, the decision table in multi-agent, single agent or conventional automation? helps. Most pilots end up with the simplest option.
2. Success criterion: what does “it works” mean?
The success criterion is written down as a number before the pilot starts. A criterion chosen afterwards will always find a way to fit the result.
Consider three kinds of measure together:
- Quality. What share of outputs were accepted without correction? How accurate is the classification?
- Time. How long did one instance of the task take before, and how long now — including the time spent checking?
- Use. Is the team actually using the tool, or drifting back to the old way?
Alongside each measure, write a threshold: below which value does the pilot count as a failure? Keep the test set — real examples with known correct answers — fixed for the whole pilot, so that every change is measured against the same yardstick. How to build that test set is covered in how to evaluate an AI system.
3. Data: what do you actually have?
Pilot data means real examples from the task’s history: past emails, completed forms, answers already given, documents already processed. A pilot run on invented examples will not show the messiness of real work.
Before starting, we recommend answering three questions:
- Are there examples? Are enough past records accessible to measure against?
- Is the right answer known? For each example, is there a record or a person who can say “this was correct”? If not, the first job is for the team to label a small set themselves.
- Is there personal data? If so, decide at the start which fields the model genuinely needs and which can be masked. We cover the legal side under Turkey’s data protection law in personal data in an in-house AI assistant.
If answers will come from company documents, preparing that data is a job of its own: which document is current, and who may see what. We walk through it in what is RAG.
4. Human approval: which step belongs to whom?
During a pilot, every output the AI produces passes in front of a person before it has any effect. That keeps the risk small, and it also produces the measurement: every output a person accepts, corrects or rejects is a data point.
The pilot card has one line per step:
| Step | What the AI does | What the person does |
|---|---|---|
| Reading the request | Summarises it, suggests a category | Confirms or changes the category |
| Preparing a reply | Writes a draft | Edits and sends it |
| Recording it | Fills in the fields | Checks before saving |
| Irreversible step | Nothing | Does it themselves |
If the pilot succeeds, some lines can be relaxed later. If category suggestions prove accurate enough, for instance, low-risk categories might be processed without approval. The measurement makes that decision. One way to classify steps by risk is set out in hallucination, data privacy and human approval.
5. The stopping rule: when do you let it go?
The stopping rule is written before the pilot starts and says under what conditions the pilot ends. The most common loss in AI projects is a trial that is not working but runs on for months on the strength of “let’s improve it a little more”.
A stopping rule contains three things:
- Duration. How many weeks will the pilot run? When time is up, a decision is made; an extension is a separate decision.
- Threshold. The value of the success criterion below which you stop.
- Red lines. Events that stop the pilot even if they happen once — personal data shown to the wrong person, or text reaching a customer without approval.
At the end, one of three decisions is taken:
| Outcome | Decision | Next step |
|---|---|---|
| Threshold passed, team using it | Expand | Open the same task to a second team or document set |
| Close to the threshold, cause known | One more round | Make a single change and measure again on the same criterion |
| Below threshold, or unused | Stop | Write up what was learnt and consider another task |
Stopping should not be treated as failure. Learning in a few weeks that a task cannot be done this way is far cheaper than learning it a year later.
What drives the cost of a pilot?
The task itself: how ready the data is, how many systems it connects to, and how much effort measurement takes. At pilot scale, model usage fees are usually a small item; most of the effort goes into data preparation and measurement. We take the items one by one in what determines the cost of an AI project.
The one-page pilot card
We recommend filling in this card with the team involved before the pilot begins. Our view is that if a line is empty, the pilot should not start until it is filled.
- Task: Which team does which task, and how often?
- Today: How long does one instance take now, and how often do mistakes happen?
- Success criterion: One number each for quality, time and use.
- Threshold: Below what value does the pilot fail?
- Data: Which past records will be used, and who decides the right answer?
- Personal data: Which personal data goes to the model, and which is masked?
- Human approval table: At each step, what does the AI do and what does the person do?
- Red lines: Which single event stops the pilot?
- Duration and decision date: When does the pilot end, and when is the decision meeting?
- Owner: Who is accountable for the outcome?
The card fits on one page. If it does not, the scope is too large.
Common mistakes
- Choosing the tool first. “We’ve bought an AI tool — where shall we use it?” pushes the work to fit the tool. Choose the task first.
- Deciding on a demo. A system that shines on five hand-picked examples can behave quite differently on a hundred messy real ones.
- Setting the criterion afterwards. A criterion written after seeing the result cannot support a decision.
- Opening it to everyone at once. A pilot runs with one team. With a wide user base, finding the source of an error gets much harder.
- Leaving the team out. If the people who do the work today are not part of the design, the criterion will not measure the real work.
Frequently asked questions
How long should an AI pilot run?
Long enough for sufficient examples to build up for measurement. For a task done dozens of times a day, a few weeks is often enough. The duration is written down before the pilot starts, and any extension is decided separately.
How much data does a pilot need?
Enough real examples for the measurement to be reliable, each with a known correct answer. If correct answers are not on record, the first job is for the team to label a small set themselves.
If the pilot fails, is the investment wasted?
No. A well-run pilot shows what did not work and why, even when it fails: was the data inadequate, was the task unsuitable, or did the team not adopt the tool? That knowledge feeds directly into choosing the next task.
Should we run the pilot ourselves or with an outside team?
Ownership of the task and its evaluation should always stay inside the company. Outside help can cover the technical build, the measurement set-up and the architecture. If you are bringing in an outside team, we have collected the questions to ask in how to choose an AI agent developer.
What happens when the pilot succeeds?
The same task is opened to a second team or a second set of data, and measurement continues. Some lines in the approval table are relaxed as far as the measurements allow. At this stage, maintenance, monitoring and spending limits also become permanent.
If you would like to think through which task could be your first pilot, see our AI systems architecture page or write to us.