AI & data

RAG and AI chatbot development: how to build it right

What RAG development involves, what separates a reliable assistant from a demo, realistic first-project scope, and how to keep company data confidential.

Marco Pravato Published 6 min read

RAG development - retrieval-augmented generation - is how you build an AI assistant that answers from your company’s own documents instead of improvising. The model retrieves the relevant passages from an indexed knowledge base, then writes a grounded, cited answer. Getting the demo working takes days; getting it right takes document preparation, retrieval evaluation against real questions, permission handling and guardrails. Build on model APIs rather than fine-tuning, keep the data in your accounts, and scope the first assistant to one document set and one user group. Here is what quality actually consists of.

What is RAG, in plain words?

A large language model on its own knows the public internet up to its training date and nothing about your business. Retrieval-augmented generation fixes that with a two-step answer:

  1. Retrieve. Your documents - manuals, policies, contracts, product sheets, past tickets - are split into passages and indexed. When a user asks a question, the system finds the most relevant passages.
  2. Generate. The model receives the question and those passages, and writes an answer grounded in them - ideally citing which document each claim came from.

The knowledge lives in an index you can update daily, not inside the model. That is why RAG became the default architecture for the enterprise knowledge assistant - the concrete AI project companies actually commission in 2026: an assistant that answers staff or customer questions from internal documentation, accurately, with sources.

Why is the demo easy and the product hard?

Anyone can build a RAG demo in a week: ten clean PDFs, a vector database, an API call. The gap between that and something your staff can trust is where real projects succeed or fail:

  • Document preparation. Real corpora are messy: scanned PDFs, tables that lose meaning when flattened, three contradictory versions of the same policy, documents that expired in 2022. Cleaning, structuring and choosing what goes in the index is routinely a third or more of the project - and the single biggest quality lever. An assistant indexed on contradictory documents will contradict itself with perfect confidence.
  • Chunking and retrieval design. How documents are split, embedded and searched determines whether the right passage is found at all. If retrieval misses, the model answers from general knowledge - fluently and wrongly.
  • Retrieval evaluation. The professional habit that separates engineering from vibes: a test set of real questions with known correct sources, scored automatically on every change. Without it, you are tuning by anecdote and every “improvement” is a guess.
  • Hallucination control and guardrails. Instructing and verifying that answers stay grounded in retrieved passages; citing sources so users can check; and - critically - making the assistant say “I don’t have this in the documentation” instead of inventing. Add topic guardrails (what it must refuse: legal advice, HR judgments, competitor speculation) and logging for review.
  • Permissions. If your documents have access levels, retrieval must filter by the asking user’s rights before the model sees anything. Skipping this turns the assistant into a data-leak machine with a friendly tone.

Build on APIs or fine-tune a model?

For knowledge assistants, the answer is unambiguous: build on commercial model APIs, almost always.

RAG on model APIsFine-tuning your own model
Teaches the model your factsYes - facts live in the indexUnreliably - fine-tuning shifts style more than knowledge
Updating knowledgeRe-index a document, minutesRetrain, days and cost
Wrong-answer traceabilityCited sources, inspectableOpaque
Model improvementsSwap the API model, keep the indexRestart on every generation
Where it earns its keepKnowledge assistants, Q&A, draftingNarrow style/format tasks at very high volume

Fine-tuning has legitimate niches - enforcing a rigid output format, a specific voice at scale - but “make the model know our business” is not one of them. RAG keeps your knowledge in an asset you own and control, and lets you ride model improvements for free.

What does a realistic first project look like?

Scope it like any well-run first engagement - bounded, real, measurable:

  1. One corpus, one audience. For example: the internal policy and procedures library, for employees; or the product documentation, for support agents. Not “all company knowledge for everyone”.
  2. A written question set before development starts. Fifty to a hundred real questions from the target users, with the correct answers’ sources. This becomes both the specification and the evaluation harness - our advice on specifications vendors can quote applies fully.
  3. A pilot with real users and a feedback loop. Thumbs up/down with comments, weekly review of failures, index and prompt corrections. This is where trust is built or lost.
  4. A team of two to four: an engineer or two on retrieval and integration, someone on document preparation and evaluation, part-time UX. A production-grade first assistant is typically a two-to-four-month effort - anyone quoting two weeks is quoting the demo.

Treat it as a low-risk pilot of both the technology and the development partner: same contract terms as at scale, deliverables in your repositories, evaluation results you can see.

How do you keep company data confidential?

The question every buyer asks, with a clearer answer than the anxiety suggests:

  • Model provider terms. Under the business/enterprise API terms of the major providers, inputs are not used for model training by default - but verify the current terms of your provider and tier, and prefer contracts with explicit data-use and retention commitments. Regional endpoint options exist where data residency matters.
  • Your accounts, your index. The model API account, the vector database and the document store should be in your name, with the development partner working inside them under scoped access - same principle as any outsourced system.
  • Anonymized development data. Retrieval pipelines can be built and evaluated on a masked or synthetic corpus before the real one is indexed.
  • The partner side. NDA, a data processing agreement naming what may be accessed, and full IP assignment of everything built - index schemas, prompts, evaluation sets included. A partner with an established, verifiable legal entity - AD System Asia operates from Bangkok as a BOI-promoted company, with clients owning all IP as standard - handles this in the contract, not in reassurances.

In short

  • RAG = retrieve from your documents, then generate a grounded, cited answer - knowledge lives in an updatable index, not the model.
  • The demo is easy; the product is document preparation, retrieval evaluation, guardrails and permissions - budget most of the effort there.
  • Build on model APIs; fine-tuning is the exception, not the path to an assistant that knows your business.
  • First project: one corpus, one audience, a real question set, two to four months with a small team.
  • Confidentiality is solved with provider terms, your own accounts, anonymized development data and a proper contract - not by avoiding the technology.

AD System Asia builds RAG assistants and AI-powered features from Bangkok - custom software development with European-style management, your accounts, your IP. Request a call and bring ten real questions your assistant should answer.

A few more answers

Frequently asked questions

What is RAG in simple terms?

Retrieval-augmented generation: instead of answering from its general training, the AI model first retrieves the relevant passages from your own documents - manuals, policies, contracts, tickets - and then writes an answer grounded in those passages, ideally with citations. It is how you get an assistant that knows your business, not the internet's average.

Why do RAG projects fail after a good demo?

Because the demo ran on ten clean documents and friendly questions. Production quality depends on unglamorous work: preparing and chunking messy documents, evaluating retrieval against a real question set, handling permissions, and guardrails for questions the system should refuse. Teams that skip evaluation ship an assistant that answers confidently and wrongly.

Should we fine-tune a model or build on APIs?

For a knowledge assistant, build on commercial model APIs with RAG - almost always. Fine-tuning does not reliably teach a model your facts; it changes style and behaviour, costs more to maintain, and restarts with every model generation. RAG keeps knowledge in an updatable index and lets you swap models as they improve.

How long does a first RAG assistant take to build?

A production-grade first assistant on a bounded document set is typically a project of two to four months with a small team, including document preparation, an evaluation set, guardrails and a pilot with real users. Anyone promising an enterprise-ready assistant in two weeks is describing the demo, not the product.

Is our data used to train the AI provider's models?

Under the business and enterprise terms of the major model providers, API inputs are not used for training by default - but verify the current terms of your chosen provider and tier rather than assuming. Contractually, your development partner should access data under NDA, and the model account should be yours.

Can a RAG assistant respect our internal permissions?

Yes, and it must: retrieval should filter by the asking user's rights before the model sees anything, so the assistant never quotes a document to someone who could not open it. This permission-aware retrieval is a build requirement to state explicitly in the specification, not an add-on to bolt on later.