A RAG chatbot on your company knowledge base: what it is and when it pays off

The request I hear most often this year sounds the same every time: "We want a bot that answers from our documents and does not make things up." The technical name is RAG, retrieval-augmented generation. Below, without jargon: what it is, who it pays off for, what your company needs to bring, and where these projects break.

RAG in plain language

A language model answers from memory. It read the internet up to some date and knows nothing about your returns policy, September price list or warehouse manual. Ask it directly and it will either admit it does not know or confidently invent something.

RAG changes the order. First the system searches your documents for the passages that relate to the question. Then it hands them to the model with the question and asks it to answer only from them, naming the source document. The model still writes the text, but the source of facts becomes your files, not its memory.

To the user it looks like a chat. To the business it is a way to give people hundreds of pages of documentation without making them read those pages.

The citation in every answer is not decoration. It is the main mechanism of trust.

Why RAG is in 13.6% of postings and vector databases in only 4.5%

In a sample of 903 US AI Engineer postings (Glassdoor, April 2026), RAG is required in 13.6% of ads. That is more often than agents (10.6%) and prompt engineering (8.9%). Vector databases, usually called the heart of RAG, appear in only 4.5%.

The gap between those numbers shows where the work actually is. A database for semantic search can be stood up in a day, and there are plenty of ready options. The hard part comes before and after.

• Documents. Three versions of the same policy in different folders, scanned PDFs, tables that lose their structure when copied.
• Chunking. How to split a document so a rule and its exception do not land in different pieces.
• Permissions. Who is allowed to see which documents, and how to carry that into search.
• Evaluation. How to tell that the bot answers correctly rather than just confidently.

In my review of the AI job market in 2026 I quoted the KORE1 survey: employers name building evaluation systems as the number one skill, not choosing a database. For RAG that is doubly true.

Who it pays off for

Not every company needs a bot on its documents. It pays off where there is a lot of text and many repeated questions about it.

• Support with a big FAQ. Hundreds of articles, agents answering the same questions, first response measured in hours.
• Internal policies and manuals. A new hire spends weeks asking colleagues where things are. HR and accounting answer the same questions about leave and expenses.
• Sales teams with a catalog. Thousands of SKUs, specifications, compatibility, regional terms. A rep digs through ten files while the customer waits.
• Agencies with an archive of proposals. Past pitches, case studies, estimates. A new proposal is assembled from old ones, if they can be found.

The common trait: the answer already exists in the documents, but finding it costs more than asking a person.

Twenty pages of documentation and five questions a day do not need a bot. One FAQ page will do.

What the company needs to bring

I raise this in the first meeting, because it matters more than the choice of model.

1. 01Current documents. The bot answers from what it was given. If a policy is out of date, the bot will confidently relay it, with a precise citation.
2. 02An owner for updates. A named person responsible for keeping the base current. Without one the system degrades within a quarter.
3. 03A list of questions with correct answers. Fifty or a hundred real questions from customers or staff, and what counts as a right answer. Without it nobody can say whether the bot got better or worse after a change.

The third item gets the most resistance and delivers the most value. A couple of days with support logs or email threads is enough to assemble it.

Where these projects break

Four typical failures, each with a defense.

Stale documents. The bot cites last year's price list. Defense: an update owner, and the document date shown in the answer so the user sees how fresh the source is.

Answering outside scope. A customer asks about the legal consequences of terminating a contract, and a bot built on delivery documents tries to answer. Defense: scoped retrieval and a hard rule: if the documents do not contain the answer, say so and hand over to an operator.

Leaking documents. The bot answers a customer with a passage from an internal manual listing cost prices. Defense: permissions enforced at the retrieval level, not in the prompt. The model must never see a document the user may not see.

Hallucinated citations. The model cites a section that does not exist. Defense: citations are generated by the system from the retrieved passages, not written by the model, plus an evaluation set that tracks such cases.

Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. From what I have seen, document projects die for the same reasons, and almost always nobody built an evaluation set before starting.

A realistic timeline

A pilot on one set of documents takes two weeks. One source (say, the support knowledge base), one scenario (answers for customers or staff), an evaluation set of 50 questions, handoff of hard cases to a person. After two weeks you have a working version to show the team and measure.

Other sources and permissions for several roles come in the second stage. I would not try to cover everything at once: every new source brings its own document quality problems.

How this fits my services

I build this kind of bot under the "AI inside your existing product" service, from 1,200 EUR. That covers RAG, LLM features and integrations into what you already run: a website, a CRM, a support portal, an internal tool. If it is unclear whether the company has suitable documents and a process, start with an AI readiness audit from 450 EUR.

If staff or customers keep asking the same questions about your knowledge base, send me a short brief: what the documents are, who asks, how many questions per week. I will tell you whether it is worth doing and how long it would take.

FAQ: RAG chatbot on a company knowledge base

How is RAG different from just chatting with ChatGPT?

ChatGPT answers from the model's memory and knows nothing about your documents. A RAG bot first finds the relevant passages in your base, then answers only from them and cites the source. The answer can be checked and the data stays under your control.

Do we need to fine-tune a model on our documents?

For a knowledge base, almost never. Fine-tuning changes a model's style and behavior, but it is poor at memorizing facts and gives no citations. RAG inserts current documents at the moment of answering, and updating them means replacing a file, not retraining.

Is it safe to give the bot internal documents?

Yes, if permissions are enforced at the retrieval level: the bot searches only the documents a given user is allowed to see. A restriction written into the prompt does not count as protection. Where data is stored and which model is used is agreed separately with the company.

How many documents do we need for a bot to make sense?

It is about repeated questions, not page count. If staff or customers ask dozens of questions a day whose answers already sit in the documents, the bot pays for itself. If it is five questions a day, an FAQ page is enough.

How much does it cost and how long does it take?

A pilot on one set of documents takes two weeks and falls under the "AI inside your existing product" service from 1,200 EUR. If you first need to know whether documents and process are ready, an AI readiness audit starts from 450 EUR.