What is RAG?

Retrieval-augmented generation—commonly asked about as “what is RAG”—is an architecture in which a language model queries an external knowledge store at inference time, retrieves the most relevant passages, and uses those passages as grounded context before composing its response. Unlike purely parametric models that rely solely on weights baked in during training, a RAG system can surface accurate, up-to-date, or proprietary information without retraining the underlying model.

The retrieval stage depends heavily on how source documents are prepared. Chunking—splitting documents into manageable segments—is a foundational design decision: chunks that are too large dilute relevance signals, whilst chunks that are too small lose the surrounding context a model needs to reason coherently. Common strategies include fixed-size chunking with overlap, sentence-boundary chunking, and semantic chunking that groups text by meaning rather than character count. Each chunk is typically encoded into a vector embedding and stored in a vector database, where approximate nearest-neighbour search identifies the passages most semantically similar to the user’s query.

Grounding and citations are what make RAG practically valuable in enterprise settings. When a retrieved chunk is injected into the model’s prompt, the model can attribute its answer to a specific document, section, or URL. Well-designed RAG pipelines surface these citations in the final response, allowing users and auditors to verify claims directly against the source material. This traceability is especially important in regulated industries where an unsupported hallucination carries real risk.

Evaluation of a RAG system covers at least three distinct dimensions: retrieval quality, faithfulness, and answer relevance. Retrieval quality asks whether the right chunks are being fetched; faithfulness checks whether the generated answer stays true to the retrieved context; answer relevance checks whether the response actually addresses the user’s question. Retrieval quality can be measured with metrics such as recall at k or mean reciprocal rank, whilst faithfulness can be assessed by checking whether factual claims are entailed by the retrieved passages.

Frequent failure modes include context-window overflow, stale indexes, and query–chunk mismatch, where a user’s natural-language question embeds differently from technical source prose. Hybrid retrieval—combining dense vector search with traditional keyword-based search such as BM25—can address mismatch by ensuring that exact-term matches are not overlooked when semantic similarity alone is insufficient.

In agentic workflows, RAG functions as the memory layer alongside tools and actions. An AI agent might call a retrieval tool to ground its reasoning in an internal knowledge base, then invoke a separate API tool to act on what it has learnt—a pattern taught across Amro Academy’s Learning Agents courses. Practitioners can validate their ability to design, evaluate, and troubleshoot RAG pipelines through OnlineTestPlus assessments aligned with retrieval-aware and multi-agent skills.