GraphRAG: When Your Answers Live Between the Documents

Try this with any standard RAG system: load a thousand internal incident reports and ask, "What are the recurring root causes across our outages this year?"

You will get an answer built from the five chunks that happened to be most similar to the words "recurring root causes". It will sound confident. It will miss almost everything.

This is not a tuning problem. It is a shape problem. Vector RAG answers local questions — the answer lives inside a passage. Some questions are global — the answer lives across the whole corpus, in how things connect. Microsoft's GraphRAG ("From Local to Global", 2024) is the best-known attempt to handle both.

Vector RAG is a search engine. GraphRAG is someone who read everything first and took structured notes.

The Analogy: Detective's Corkboard

Picture the corkboard in every detective film: photos of people, pinned index cards, red string connecting who met whom and where. The detective does not re-read every witness statement for each new question. They look at the board.

GraphRAG builds that board from your documents, then — this is the key part — writes summaries of each cluster of connected cards. "These seven people are all linked to the shipping company." Those summaries are what make global questions answerable.

Indexing: From Text to a Graph of Summaries

documents → chunk into text units → LLM extracts entities + relationships + claims → merge into one knowledge graph → community detection (Leiden) → LLM writes a report per community — The GraphRAG indexing pipeline. Almost every step is an LLM call — which is why it is powerful and expensive.

Step by Step

Chunk the corpus into text units, just like normal RAG.
For every chunk, prompt an LLM to extract entities (people, systems, orgs, concepts), relationships between them, and a short description of each.
Merge duplicates across chunks. "Payments API" in document 12 and document 400 become the same node, with descriptions combined.
Run a community detection algorithm (Leiden) on the graph. It finds densely connected groups — hierarchically, from broad communities down to small ones.
For each community, the LLM writes a community report: what the group is about, key entities, key relationships, important claims.

You end up with three layers of knowledge: raw text chunks, a graph of entities and relationships, and a hierarchy of human-readable summaries.

text chunks → entities + relationships → small communities → large communities → top-level themes — The hierarchy. Global questions are answered from the top; local ones from the bottom.

Querying: Local, Global and in Between

Local Search — "Tell Me About X"

Find the entities that match the question, then walk their neighbourhood: connected entities, relationships, the community reports they belong to, and the original text chunks. It is vector RAG with context enriched by structure.

question → match entities → expand to neighbours + relationships → add community reports + source chunks → LLM answer — Local search: start from specific entities and walk outward.

Global Search — "What Are the Main Themes?"

This is the part vector RAG cannot do. GraphRAG runs map-reduce over the community reports:

question → map: each community report → partial answer + importance score → drop low-score answers → reduce: combine the best partial answers → final answer — Global search: every community gets a say, then the answers are merged.

Analogy: instead of asking one librarian to skim five pages, you ask every department head for a one-paragraph briefing, then have an editor merge the best ones.

DRIFT Search

A later addition that mixes the two: start with a global-style view using community reports, then generate follow-up questions and drill down locally. Think of it as "zoom out, then zoom in".

Where GraphRAG Shines

Sense-making over a whole corpus: themes, trends, "what is going on here?"
Multi-hop questions: "which teams depend on services owned by people who left last quarter?"
Domains that are naturally relational: incidents, legal cases, research literature, codebases, investigations.
Explainability: you can show the entities and relationships the answer used.

Where It Hurts

Indexing cost: an LLM call per chunk for extraction, plus one per community for reports. On large corpora that is a real bill.
Updates are painful: new documents can change communities, which means regenerating reports.
Extraction quality caps everything: if the LLM misses an entity or invents a relationship, the graph is wrong forever (until re-indexed).
Overkill for FAQ-style questions where plain hybrid RAG is faster and just as accurate.

Rule of thumb: if your users ask "what does the doc say about X?", use vector RAG. If they ask "what is the big picture?" or "how are these connected?", consider a graph.

How I Would Adopt It

Start with hybrid vector RAG and a golden set of questions. Label which ones are local vs. global.
If global/multi-hop questions are a real share of traffic and failing, prototype GraphRAG on a slice of the corpus.
Use a cheaper model for extraction and a stronger one for community reports and answers.
Route: local questions → vector RAG, global questions → GraphRAG global search. Modular RAG again.

The cost and update problems are real enough that a whole family of lighter graph-RAG systems appeared. The one I have used most is LightRAG — which is the next post.