What is RAG? How businesses use their own data in a language model

A language model can formulate a convincing answer without having access to the information that applies specifically to your business. RAG, from retrieval-augmented generation, is a way to narrow that gap. The technology allows a solution to search for relevant information in approved data sources and use it as a basis when generating the answer.

This makes it possible to build, for example, an internal knowledge chat that answers based on policies, manuals, reports, or product documentation. However, RAG is not a guarantee that every answer will be correct. The result depends on how the information has been structured, what access the user has, how the search works, and how the solution is tested.

What is RAG?

RAG combines two functions: information retrieval and text generation. When a user asks a question, the system first searches for relevant content in one or more connected sources. The most relevant text passages are then sent, along with the question, to a language model. The model formulates an answer using the source material as context.

The concept was established in research on how language models can be combined with an external, searchable memory. An important point is that knowledge does not only need to be stored in the model's parameters. It can also be retrieved from a source that the organization itself can update and control.[1]

In practical enterprise solutions, RAG often includes more layers than just the search itself: document processing, identity, permission filtering, logging, citation of sources, and follow-up.

How RAG works step by step

1. Sources are connected

The solution is linked to approved data sources, such as SharePoint, a document archive, an intranet, or a database. Each source needs an owner and clear rules regarding which users are allowed to see the content.

How to connect a secure chat to your business's own data

2. Content is processed

Documents need to be readable by the system. Text is extracted, usually split into smaller sections, and supplemented with metadata, such as document type, date, department, and permission information. Poorly scanned files, duplicates, and conflicting versions degrade the result.

3. Text is indexed

The text sections are often converted into numerical representations, known as embeddings. They are stored in a search index where content with similar meaning can be found even if the question does not use the exact same words as the source.

4. The question is verified

When a user asks a question, the user and their access rights are identified. The search must be restricted to information that the user actually has the right to see. A shared chat must not become a shortcut around the organization's permission model.

Permission-controlled AI with Microsoft Entra ID – how access works

5. Relevant sections are retrieved

The system selects the sections deemed to be most relevant. Here, semantic search can be combined with keywords, filters, re-ranking, and rules for freshness. If the wrong sections are retrieved, the model is given a poor foundation, even if the language model itself is powerful.

6. The model formulates an answer

The question and the retrieved sections are sent to the chosen language model. The instructions can require the model to stick to the sources, state uncertainty, and refrain from answering when the basis is insufficient.

Local language models or cloud models – how to choose?

7. Sources are shown and the result is followed up

The user should be able to see which documents or sections support the answer. Logs and user feedback make it possible to detect recurring errors and improve indexes, instructions, and test cases.

RAG does not make the model infallible. It makes the answers easier to ground, verify, and update.

RAG compared to training or fine-tuning a model

RAG and fine-tuning solve different problems. RAG is primarily used to give the model access to current or business-specific knowledge at the time of answering. When a policy changes, the source can be updated and re-indexed without re-training the model.

Fine-tuning, on the other hand, affects the model's behavior or ability within a specific task. This can be relevant for, for example, classification, tone, format, or a recurring decision pattern. Fine-tuning is rarely the best way to store continuously updated facts.

In some solutions, the methods are combined. A model can be customized for the task while simultaneously using RAG to retrieve current information. The choice should be based on the problem, the data, and the requirements – not on which method sounds most advanced.

What problems can RAG solve?

RAG is suitable when the correct answer exists within a defined set of information but is difficult or time-consuming to find. Common areas of use are:

• support for employees searching policies, procedures, and manuals

• technical support based on manuals and product documentation

• document analysis with summaries and source references

• support in regulated operations where the origin of information must be traceable

• customer service where answers must be grounded in current terms and approved content

• research in report archives or other large, controlled bodies of knowledge

The technology is less suitable if the task primarily requires structured calculations, deterministic rules, or real-time data that should be retrieved directly via an API. In such cases, RAG can be part of the solution, but not necessarily the core.

Secure document analysis with AI in regulated industries

What determines the quality of the answers?

Quality is not determined by a single component. At least five parts need to work together.

Quality of the sources

Outdated, contradictory, or unclear documents provide an unreliable foundation. Information ownership and version control are therefore just as important as the choice of model.

Document processing

Headings, tables, attachments, and metadata must be handled in a way that preserves context. If text is split in the wrong place, key terms can be separated from their exceptions.

The search

The solution needs to find the right text for different types of questions. This should be tested separately from the quality of the final answer.

Instructions to the model

The model must know when to use sources, how to express uncertainty, and when to decline. However, the prompt is not a sufficient security safeguard on its own.

The evaluation

A few impressive demo answers say little about how the solution works over time. A representative test library needs to contain simple questions, ambiguous questions, permission cases, insufficient documentation, and questions the solution should refuse to answer.

NIST recommends that generative systems are managed throughout their entire lifecycle with documented risk assessment, measurement, and follow-up.[2]

Permissions, source citations, and traceability

An enterprise solution needs to distinguish between the right to use the platform, the right to use a specific function, and the right to retrieve content from a specific source. If two employees ask the same question, their source materials may therefore need to be different.

Source citations allow the user to check the answer, but they also need to be tested. A reference can be relevant without actually supporting the entire claim. For sensitive or business-critical information, the solution should be designed so that human review is part of the workflow.

Logging is needed to be able to monitor usage, troubleshoot, and investigate incidents. At the same time, logs may contain personal data or sensitive information. What is logged, who is allowed to read the logs, and how long they are kept must therefore be a conscious decision.

What happens to your company data when you use AI?

When is RAG not enough?

RAG cannot compensate for an organization's lack of reliable sources. Nor does the technology eliminate all risks associated with generative text. The model can misunderstand a correct section, mix up multiple sources, or formulate an overly confident conclusion.

For tasks with a very low error tolerance, a more controlled flow may be required: fixed rules, structured data validation, mandatory human approval, or only allowing the model to extract information rather than drawing its own conclusions.

OWASP highlights, among other things, risks related to manipulated instructions, sensitive information disclosure, and insecure handling of the model's outputs.[3]

AI assistant or AI agent – what is the difference?

How to get started

Before choosing a model or platform, you need to define what the solution should be able to answer and which sources may be used.

• Appoint a business owner for the problem and an owner for each data source.

• Audit document formats, quality, duplicates, and update frequency.

• Classify the information and define permissions.

• Decide how sources and uncertainty should be displayed to the user.

• Create representative test questions and expected answers or rejections.

• Set acceptance criteria for search, answers, security, and response time.

• Plan how content, tests, and models will be maintained after launch.

A good first pilot uses a limited information area and a clear group of users. This makes it possible to measure whether the solution actually reduces search time, improves quality, or simplifies a workflow – without opening up the entire organization's information landscape all at once.

From idea to pilot: what is PoC, prototype, and pilot?

Read more about Violet AI Platform

Sources

1. Lewis, P. et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020:
   https://arxiv.org/abs/2005.11401

2. NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, published 2024 (webpage updated 2026):
   https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

3. OWASP, Top 10 for Large Language Model Applications / GenAI:
   https://genai.owasp.org/