How to Build a Business Knowledge Assistant with RAG

Retrieval-augmented generation, or RAG, lets an AI application retrieve company-approved information and use it as context for an answer. It can make a knowledge assistant more useful and traceable, but it does not make stale, conflicting, restricted, or badly retrieved information trustworthy.

Image disclosure: The images are AI-generated editorial illustrations, not photographs of a ChoiceRidge implementation or a vendor's customer environment.

Short answer

Treat RAG as a knowledge-management and access-control system, not only a model feature. Start with a narrow audience and owned source collection. Clean and version documents, preserve permissions, test retrieval separately from answer generation, require citations or abstention, monitor unanswered questions, and assign owners for updates and deletion.

When RAG is appropriate

RAG is useful when answers should depend on a changing, private, or bounded body of information: policies, product specifications, support procedures, contracts, technical manuals, or approved research. It is less suitable when the task requires deterministic database transactions, complete legal discovery, or facts that must be calculated directly from a system of record.

Before building, define:

• users and decisions the assistant may support;
• sources it may and may not use;
• whether citations are mandatory;
• when it must abstain or escalate;
• freshness requirement and update owner;
• information each user is authorized to retrieve;
• how feedback, logs, and deleted content are handled.

Fix the source collection first

Create an inventory with owner, audience, approval status, effective date, expiry, jurisdiction, sensitivity, and canonical location. Remove duplicates and drafts, resolve conflicting versions, and split mixed-access documents. Scanning thousands of files into a vector index can amplify knowledge debt rather than solve it.

Do not assume a document is safe for AI because employees can open it somewhere. The retrieval layer must enforce the same or stronger access rules as the source system. Consider personal data, confidential contracts, trade secrets, customer content, and licensed material. Record the lawful and contractual basis for processing and whether a vendor may retain inputs or use them for service improvement.

Understand the pipeline

A typical RAG system:

1. ingests approved documents;
2. parses and divides them into retrievable chunks;
3. creates an index or embeddings plus metadata;
4. receives a user query and applies identity and filters;
5. retrieves candidate passages;
6. places selected context in the model prompt;
7. generates an answer, citations, or abstention;
8. logs enough evidence to evaluate and investigate.

Every step can fail. Tables may parse incorrectly, chunks may lose headings, metadata may omit access rules, retrieval may favor a popular but obsolete document, or the model may answer beyond the supplied context.

Evaluate retrieval and answers separately

Microsoft's RAG evaluation guidance distinguishes process evaluation from system evaluation. Retrieval measures whether the right documents or chunks were found. Answer evaluation checks groundedness, relevance, and completeness. A fluent final answer cannot reveal which layer failed.

Build a test set of real questions with expected source documents and acceptable outcomes. Include ambiguous questions, outdated terminology, access differences, conflicting policies, unsupported requests, and malicious instructions embedded in documents. Measure:

Use domain experts for the final rubric. Automated groundedness checks can assist, but inspect their failures and calibrate them against human review.

Design for uncertainty and correction

Show source title, date, and link when appropriate. Do not hide conflicting evidence. Make “I cannot answer from the approved sources” an acceptable response. Provide an escalation route and a way to report outdated or incorrect material.

Feedback should enter a managed queue, not automatically rewrite the knowledge base. A subject owner should decide whether the source, retrieval configuration, prompt, or answer policy needs correction. Keep document and system versions so an incident can be reproduced.

Security and prompt injection

Retrieved content is untrusted input even when it came from an internal repository. A compromised or casually written document can contain instructions that attempt to override the application. Separate system instructions from retrieved text, restrict tools, sanitize ingestion where appropriate, and test indirect prompt injection. OWASP's GenAI guidance treats prompt injection and excessive agency as application risks, not problems solved by a better model alone.

Operating checklist

• Defined audience, task, source boundary, and accountable owner
• Canonical documents have approval, date, sensitivity, and expiry metadata
• Retrieval enforces user and document permissions
• Representative questions have expected sources and abstentions
• Retrieval and answer quality are measured separately
• Citations are validated, not merely displayed
• Updates, deletions, and index rebuilds have service targets
• Injection, leakage, outage, and fallback behavior are tested
• Feedback is reviewed by a source owner
• Logs support audit without retaining unnecessary sensitive data

This guide supports the knowledge and analytics scope of the ChoiceRidge AI for Business & Commerce category. If the requirement is to move records between systems after an approved decision, continue to the Automation & Integration library.

Method and limitations

This product-neutral guide uses official Microsoft evaluation documentation, NIST risk guidance, ICO privacy guidance, and OWASP security material. RAG does not guarantee factual accuracy, full corpus coverage, or legal compliance. Architecture must be adapted to the data, users, and jurisdiction.

References

• Microsoft Foundry: RAG evaluators
• NIST Generative AI Profile
• ICO: AI and data protection guidance
• OWASP GenAI Security Project