Enterprise RAG Agent — Production Blueprint

Executive Overview

RAG succeeds in enterprises when it is treated as a governed knowledge system, not as a one-time embedding job. The production difference is operational: document lifecycle, delta ingestion, freshness ranking, access control, observability, and evaluation gates.

Vector DB options (prototype → enterprise)

Use Case: AI Knowledge Agent for Policies, SOPs & Project Delivery

Problem

Enterprise knowledge is fragmented across PDFs, SharePoint/Confluence, internal portals, and ticket systems. People lose time searching, and critical decisions get made using outdated guidance.

Solution

A RAG-based agent that answers questions using only approved internal evidence, returns citations, respects access control, and abstains when evidence is insufficient.

Figure — Enterprise RAG Architecture Overview

A system view (not a prompt): ingestion → registry → chunking → embeddings → vector store → retrieval with filters → grounded answer with citations. The “ops layer” includes logging, evaluation, and lifecycle controls.

Reference Architecture (Enterprise-Grade)

Core layers

Metadata contract (mandatory)

Production RAG requires a strict metadata contract. If content cannot be governed (owner/version/status/ACL), it should not be indexed.

Figure — Ingestion & Normalization Pipeline

Connectors produce a document manifest (IDs, timestamps, ACL tags). Normalization outputs structured text preserving headings. Registry compares checksums to determine delta ingestion.

End-to-End Control Model (Production): Originals → Chunking → Embeddings → Vector DB → Updates/Deletions

1) What is chunk.text (and what it is not)

2) Chunking: how chunks are created (tools + profiles + rules)

Chunking is one of the strongest quality levers. Production chunking preserves structure (headings, steps, tables), uses controlled chunking profiles, and assigns stable identifiers for updates and auditability.

2.1 Practical tool stack

2.2 Chunking profiles (controlled evolution)

Profiles prevent random tweaks. You only change chunking through versioned profiles, then measure impact via evaluation gates.

Figure — Chunking Blueprint

Normalized structure → chunk boundaries → stable chunk IDs + metadata contract required for governance and updates.

3) Embeddings & Vector DB: who stores what (and where)

3.1 What lives where

Figure — Storage Responsibilities

Separation of concerns: originals in source systems; registry in relational DB; vectors in vector store; traces for audit and evaluation.

4) Updates & deletions: delta ingestion, tombstones, and hard deletes

4.1 Delta ingestion (healthy updates without reindex-all)

• Connectors collect a manifest: doc_id, last_modified_at, source_uri, access_tags.
• Compute a checksum (hash) of normalized content (optionally per section).
• Compare to the registry; process only: new, changed, deleted.
• Upsert new vectors; mark old chunks tombstoned when versions change.
• Record index_state and errors for automated retries.

4.2 Versioned IDs (find, update, and rollback)

4.3 What happens when a physical file is deleted?

The correct approach is automated and governed. People decide lifecycle and ownership; systems execute the pipeline. Deletion is detected via webhooks or polling, then applied as a soft delete (tombstone) followed by controlled hard delete.

Figure — Delta Ingestion + Deletion Handling

Manifest → checksum → registry compare → NEW/CHANGED/DELETED with upsert, tombstone, and scheduled hard delete.

5) Who triggers updates? Webhooks vs polling vs governance actions

6) Governance & Operating Model (mapped control + RACI)

6.1 Governance mapping (who owns what)

6.2 RACI (minimum viable)

6.3 Operating cadence (what runs automatically)

Figure — Governance Operating Model Mapped to the Pipeline

Decision rights (owners/security) over lifecycle and access, plus automated execution (ops) across ingestion, indexing, retrieval, and auditing.

7) Architecture description (end-to-end control)

Build Phases (Prototype → Production-Ready)

Phase 0 — Scope & “Golden Set” (Day 1–2)

• Select 10–50 “golden” documents (policies, SOPs, templates) with high business value.
• Define the first 30–50 benchmark questions (real user queries) and expected sources.
• Define “Definition of Done” for answers: citations required, abstain rules, freshness rules.

Phase 1 — Ingestion & Registry (Day 2–4)

Build a registry first. Your registry is your operational truth: what is indexed, with what configuration, and when.

Figure — RAG Document Registry

The registry is the system of record for indexing configuration and lifecycle state. It enables delta ingestion, auditability, controlled re-chunking, and rollback.

Phase 2 — Chunking (Day 4–5)

Chunking is a primary quality lever. Use “profiles” instead of ad-hoc changes.

Figure — Chunking Profiles & Stability

Shows how chunking profiles create controlled evolution: you do not “randomly tweak chunk sizes” in production; you version and measure changes.

Phase 3 — Embeddings & Upsert (Day 5–6)

• Embed chunks in batches; store embed_model + embed_version in registry.
• Upsert vectors with doc_version in IDs for rollback.
• Apply tombstones for retired content (soft delete) before hard delete.

Phase 4 — Retrieval (Day 6–7)

Production retrieval is not “top-k cosine similarity”. It is similarity + ACL filtering + freshness + lifecycle penalties.

Figure — Retrieval Ranking with Freshness & Lifecycle

A view of how semantic relevance is adjusted by operational metadata so the agent prefers current, active documents, and reduces the risk of citing outdated content.

Phase 5 — Grounded Answering (Day 7–8)

• Use an “evidence-only” system instruction and include citations.
• Define abstention rules: if evidence score below threshold → ask clarifying question or say “not found”.
• Return structured citations (doc_id, section, version, source_uri) to support click-through UX.

From RAG to an Agent (Tool-Using, Policy-Driven)

A “RAG chatbot” answers questions. A RAG agent can decide when to retrieve, when to abstain, and when to execute controlled actions (tickets, approvals, workflow triggers).

Figure — Agent Loop with Retrieval, Policy Gates, and Tools

The loop shows how retrieval is invoked before response, and how governance gates decide whether to answer, ask clarifying questions, or escalate to a human workflow.

Reference pseudo-implementation

Operations & RAG Health Framework (Enterprise)

Document lifecycle (governance)

Delta ingestion (healthy updates without reindex-all)

• Connectors collect a manifest: doc_id, last_modified_at, source_uri, access_tags.
• Compute checksum (hash) of normalized content (or per section).
• Compare to registry. Only process: new, changed, deleted.
• Upsert new vectors; mark old chunks tombstoned if version changes.
• Record index_state and errors for retries.

Figure — Daily / Weekly / Monthly RAG Ops

Daily: delta ingestion + retries + anomaly alerts. Weekly: analyze unanswered queries, duplicates, and missing owners. Monthly: retire docs, run regression evaluation, consider controlled chunking revisions.

Operational routines

Observability & audit (what to log)

Continuous Evaluation (RAG Never Sleeps)

Golden set (minimum viable)

Start with 30–50 real questions per domain (Security, HR, PMO, etc.). Each question should have expected “correct source documents” (doc_id + section).

Metrics (practical + operational)

Operational gates (alerts / stop conditions)

Figure — RAG Evaluation Dashboard

A simple “trust dashboard”: retrieval recall, citation accuracy, abstentions, top unanswered queries, most-cited docs, and indexing health signals (failures, lag, costs).

Security & Access Control (Enterprise Baseline)

Principle: retrieval must enforce least privilege

If a user cannot access a document in the source system, the agent must not retrieve its chunks. ACL enforcement happens at retrieval time via metadata filters.

Figure — ACL Enforcement at Retrieval

Shows how identity (tenant/group membership) becomes retrieval filters, preventing cross-team or cross-tenant leakage, while preserving audit trails for compliance.

Abuse-resistant behaviors

• Abstain by default when no evidence is found.
• Refuse if the user requests restricted content outside their scope.
• Never execute writes without confirmation and explicit authorization.
• Log suspicious prompts (exfiltration attempts, injection patterns).

Security & Access Control (Enterprise Baseline)

Principle: retrieval must enforce least privilege

If a user cannot access a document in the source system, the agent must not retrieve its chunks. ACL enforcement happens at retrieval time via metadata filters.

Figure — ACL Enforcement at Retrieval

Shows how identity (tenant/group membership) becomes retrieval filters, preventing cross-team or cross-tenant leakage, while preserving audit trails for compliance.

Abuse-resistant behaviors

• Abstain by default when no evidence is found.
• Refuse if the user requests restricted content outside their scope.
• Never execute writes without confirmation and explicit authorization.
• Log suspicious prompts (exfiltration attempts, injection patterns).