Retrieval-augmented generation sounds like a simple sequence: search for relevant content, place it in the prompt, and ask a model to answer. Production RAG systems are harder because every step can fail independently. The wrong documents can be ingested, chunks can lose context, retrieval can miss the answer, ranking can favor irrelevant passages, permissions can be ignored, or the model can still produce an unsupported response.<\/p>\n
Microsoft Foundry gives teams models, agents, evaluation, and integration with retrieval services such as Azure AI Search. The architectural goal is not merely to \u201cadd RAG,\u201d but to build a retrieval system whose sources, relevance, security, citations, and failure behavior can be measured.<\/p>\n
A RAG system needs an explicit answer to what knowledge it is allowed to use. Internal policies, product manuals, support tickets, contracts, code, customer data, and public documentation have different owners and risks. Mixing them into one index without metadata or access boundaries makes retrieval easier to build and harder to govern.<\/p>\n
Each source should have ownership, freshness expectations, sensitivity, and retirement rules. The system also needs to know which source wins when documents conflict. If a new policy replaces an old one, both should not remain equally retrievable.<\/p>\n
The fundamentals of embeddings, vector retrieval, and RAG<\/a> explain the core mechanism, but production architecture begins with source authority.<\/p>\n Text extraction is not enough. Useful ingestion can preserve headings, page numbers, document identifiers, timestamps, product versions, security metadata, and other context needed for ranking and citation. A chunk without provenance may help the model answer, but it makes the answer difficult to validate.<\/p>\n Chunk size is a trade-off. Very small chunks can lose context. Very large chunks can dilute relevance and consume prompt space. Different content types need different strategies: legal clauses, code functions, FAQ entries, and long narrative documents should not necessarily be chunked the same way.<\/p>\n Ingestion pipelines should also detect malformed content, duplicate documents, unsupported formats, and unexpectedly empty extraction. Otherwise retrieval quality degrades silently.<\/p>\n Vector search is useful for semantic similarity, but exact terms still matter. Product codes, error messages, policy numbers, names, and identifiers may be better served by lexical matching. Hybrid retrieval combines semantic and keyword signals, often with reranking, to improve robustness.<\/p>\n Metadata filters can narrow the candidate set by product, region, date, language, document type, or user access. That improves both relevance and security. Query rewriting can translate a conversational request into a retrieval-friendly form, but rewritten queries should be observable because a bad rewrite can hide the user\u2019s actual intent.<\/p>\n The Azure AI Search retrieval concepts<\/a> are useful because production RAG depends as much on search quality as on model quality.<\/p>\n A model cannot reliably \u201cpromise not to reveal\u201d content it was given. Retrieval should exclude material the caller is not authorized to access before chunks are placed into the prompt. Permission-aware filters, separate indexes, document ACL metadata, or other access patterns should be selected based on the data source and risk.<\/p>\n Service identities used by ingestion or retrieval should follow least privilege. An indexing pipeline may need broad read access, while an interactive query should still filter results to the user\u2019s authorized subset. Logging should avoid turning sensitive retrieved text into a second uncontrolled data store.<\/p>\n Citations help users verify answers and help operators debug retrieval. A useful citation should point to a source the user can access and identify the relevant passage or location. A link to a document root with no indication of which section supported the answer is less useful.<\/p>\n The architecture should preserve document and chunk identifiers throughout retrieval and generation so citations are not reconstructed heuristically after the fact. If a source is removed or access changes, the citation path should fail safely.<\/p>\n Agentic retrieval can improve hard queries but adds failure modes. An agent can decide to run multiple searches, reformulate a query, use structured data, or call domain-specific tools. This can help with multi-part questions and tasks that require reasoning across sources. It also creates more paths to test.<\/p>\n Agents need limits on search iterations, tools, cost, and time. They should know when retrieved evidence is insufficient and when to ask the user for clarification. A system that searches indefinitely or repeatedly retrieves the same weak evidence is not more intelligent; it is less controlled.<\/p>\n A final answer can look good even when retrieval was poor, and good retrieval can still produce a bad answer. Microsoft Foundry\u2019s RAG evaluation model distinguishes process quality from system quality. Retrieval evaluation asks whether the retrieved context is relevant to the query. System evaluation can examine groundedness, relevance, completeness, and other response characteristics.<\/p>\n This separation is essential for debugging. If retrieval misses the authoritative document, prompt tuning will not fix the root problem. If retrieval is excellent but the model ignores the evidence, the generation stage needs attention.<\/p>\nIngestion should preserve structure and provenance<\/h2>\n
Hybrid retrieval is often stronger than one technique<\/h2>\n
Security must be enforced before context reaches the model<\/h2>\n
Citations should be designed as a product feature<\/h2>\n
RAG evaluation must separate retrieval from answer quality<\/h2>\n