From idea to pilot: what is a PoC, prototype, and pilot?

Why start with a scoped test phase?

Projects involving language models or machine learning contain uncertainties that cannot always be resolved on paper. It may be unclear if there is enough data, if the model can handle the organization's language, if users trust the results, or if the solution can be integrated with existing systems.

A scoped test phase makes uncertainties visible and transforms them into decision points. The purpose is not to build an impressive demo, but to gather enough evidence to choose the next step.

A good pilot should not prove that the technology is exciting. It should provide the basis for a business decision.

Feasibility study, PoC, prototype, and pilot – the difference

Feasibility study: is the problem worth solving?

The feasibility study investigates needs, affected users, available data, risks, and possible solution paths. Here you should be able to describe which work step is being changed, who owns the benefit, and how a better outcome can be measured.

The deliverable is usually a scope definition, an initial set of requirements, identified risks, and a recommendation. A feasibility study does not need to contain working code.

How to choose the right use case for your first AI project

Proof of Concept: is the critical idea technically possible?

A PoC tests the most uncertain technical hypothesis. Can a matching model rank relevant alternatives? Can a language model retrieve the correct section from the organization's documents? Can the necessary data source be connected?

A PoC can be small and technically raw. It does not need a finished interface or support for many users. However, the test must have a clear passing result: what needs to work for the hypothesis to be considered proven?

Prototype: how can the solution work for a user?

The prototype makes the idea usable and evaluable in a cohesive flow. It can contain a simple interface, several technical components, and realistic sample data. Here, feedback can be gathered on usability, workflow, and which features are actually needed.

A prototype is still not built for full production. Permissions, monitoring, incident handling, and capacity may be simplified. It must be clear to the testers what is temporary.

Pilot: does the solution work in the organization?

The pilot tests the solution with real users, real tasks, and a scoped part of the actual environment. The focus shifts from purely technical performance to usage, benefit, risk, and maintenance.

A pilot therefore needs to have defined user groups, permissions, support, metrics, and a plan for how errors are handled. It must end with a decision: scale up, change direction, perform a new scoped test, or terminate.

Production: does the solution hold up over time?

In production, the solution must work for intended users with stable integrations, correct access, monitoring, documented quality, and clear ownership. It must withstand changes in data, models, costs, and organizational requirements.

Which phase fits your needs?

Start with the biggest unanswered question.

• If you cannot formulate the problem or the benefit: start with a feasibility study.

• If technical feasibility is uncertain: do a PoC.

• If you need to understand the user flow: build a prototype.

• If the technology works but the impact on the organization is unknown: run a pilot.

• If benefit, quality, and risk are sufficiently proven: plan for production.

It is not always necessary to use all terms or make separate projects of each phase. In a small initiative, a PoC and prototype can be carried out together. The important thing is that the questions are not skipped.

What should be measured in a pilot?

The metrics should be linked to the problem the pilot is intended to solve. For a knowledge assistant, it could be time to find the right information, the proportion of answers backed by the correct source, and the number of cases that need to be escalated. For a forecasting model, it could be forecast error compared to a simple baseline and how the outcome affects inventory or planning.

A balanced pilot measures at least four areas:

• Business benefit: time, quality, cost, revenue, or other relevant impact.

• Technical quality: precision, relevance, stability, response time, and capacity.

• Risk and security: incorrect answers, permission violations, data leaks, and incidents.

• Usage: how often the solution is used, in which tasks, and why users opt out.

Always measure against a baseline. Without a baseline, it is impossible to know if the pilot has improved the work.

Data, security, and users must be included from the start

Postponing security and data protection to the production phase often leads to redesigns. Already in the testing phase, the organization needs to know what information is used, who is allowed to see it, and whether personal data or confidential material is processed.

Users also need to participate early. A technically correct solution can fail if it falls outside the actual workflow or requires more verification work than it saves.

The NIST framework emphasizes that risk management needs to run throughout the system lifecycle, not be added as a final check.[1] Digg similarly recommends that public sector actors start from needs, information, and responsibility when making decisions about generative technology.[2]

What happens to your company's data when you use AI?

Three examples of scoped tests

Example 1: Matching profiles and opportunities

A PoC can test whether a multilingual embedding model can rank jobs or courses based on a limited number of anonymized profiles. The next step is to review relevance, bias, and which parts of the personal information should affect the match.

AI Act, NIS2, DORA & GDPR – which requirements affect your company's AI solution?

Example 2: Demand forecasting

An initial model can be trained on a clearly scoped time series and compared with a simple statistical baseline. If the model provides stable improvement, more factors, locations, and workflows can be added in a pilot.

Example 3: Chat with internal documents

A prototype can be connected to a limited set of documents and tested by a selected group. Questions should cover correct answers, missing sources, conflicting documents, and information that the user should not see.

How to connect a secure chat to your organization's own data

When is the solution ready for production?

A pilot is ready to move forward when the business owner can answer yes to the following questions:

• Have we demonstrated a measurable improvement compared to the current state?

• Is the quality sufficient for the task and documented with representative tests?

• Are data sources, legal basis, and information classification handled?

• Does the access follow the user's actual permission level?

• Does the user know when the result must be verified by a human?

• Are there owners for the product, data, security, and maintenance?

• Can we monitor cost, capacity, errors, and changes over time?

If the answer is no, it does not necessarily mean the idea should be terminated. The pilot may have done its job by showing exactly what needs to change.

Permission-controlled AI with Microsoft Entra ID – how access works

Common mistakes

The most common mistake is choosing technology before the problem. Another is building too many features before the critical hypothesis is tested. Projects also get stuck when a pilot lacks a decision owner or when it is evaluated based on enthusiasm instead of metrics.

Also, avoid treating prototype code as automatically production-ready. A demo may lack almost everything required for stable operation: identity, logging, monitoring, error handling, cost control, and a plan for model updates.

Buy a service, build it yourself, or use an AI platform?

Checklist before a pilot

• A scoped business problem and a named problem owner.

• A prioritized user group and real work tasks.

• Available, permitted, and quality-assured data.

• Identified security, privacy, and regulatory requirements.

• Metrics, baseline, and acceptance criteria.

• Timeframe, budget, and clear decision points.

• A plan for maintenance or decommissioning after the pilot.

Sources

1. NIST, AI Risk Management Framework:
   https://www.nist.gov/itl/ai-risk-management-framework

2. Digg, Make conscious decisions about acquiring generative AI:
   https://www.digg.se/ai-for-offentlig-forvaltning/riktlinjer-for-generativ-ai/ta-medvetna-beslut-om-att-skaffa-generativ-ai