Knowledge Graph RAG

Level: Advanced · Reading time: 8 min · Phase: 2 — Advanced RAG

In one line: Knowledge-graph RAG stores facts as connected entities and relations, and retrieves by following those connections instead of (or as well as) matching similar text.

Before you start: What is RAG and Ensemble Retrieval. Graph retrieval is often used alongside normal vector search.

Explain it like I'm new

Think of a family tree. "Who is my mother's brother's daughter?" is hard to answer by searching a pile of letters for similar sentences. With a tree, you just follow lines: me → mother → brother → daughter. Each line is a relationship.

A knowledge graph (KG) stores facts as triples: (subject, relation, object), for example (billing-service, owned_by, Payments Team). Subjects and objects are entities (nodes). Relations are edges. Together they form a graph you can walk.

Vector RAG finds passages that look like the question. It struggles with multi-hop questions, where the answer requires chaining facts that live in different documents. "Who manages the team that owns the billing service?" needs two hops: billing-service → owned_by → Payments Team → managed_by → Priya. No single chunk may contain both facts. A graph makes those hops explicit.

Words to know:

• Knowledge graph (KG): facts stored as a network of connected things.
• Triple: one fact written as (subject, relation, object), such as (billing-service, owned_by, Payments Team).
• Entity: a "thing" in the graph (a node), such as a service, team or person.
• Edge: a labelled link between two entities.
• Multi-hop: a question that needs several links followed in a row.
• Entity resolution: treating different spellings of one thing as a single node.

Watch it

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

Build (offline):

1. Extract triples from documents (an LLM or NER/relation-extraction model, or from structured sources such as a service catalogue or HR system).
2. Normalise entity names ("Payments team", "payments-team" → one node). This is entity resolution.
3. Store in a graph database (e.g. Neo4j) or a simple triple store.

Query (online):

1. Entity linking: find entities mentioned in the question.
2. Traverse: walk 1–3 hops from those entities, or run a structured query (Cypher, SPARQL) that an LLM writes from the question.
3. Serialise the found triples into short sentences.
4. Give those facts (optionally plus vector-retrieved chunks) to the LLM.

graph LR
    B[billing-service] -- owned_by --> P[Payments Team]
    P -- managed_by --> Pr[Priya]
    Pr -- reports_to --> M[Marco]
    B -- depends_on --> DB[postgres-main]
    DB -- hosted_in --> R[eu-west-1]

Starting at billing-service with max_hops=2 reaches Payments Team, Priya, postgres-main and eu-west-1, but not Marco (3 hops).

Worked example

Question: "Who manages the team that owns billing-service?" The graph holds these triples: billing-service owned_by Payments Team; Payments Team managed_by Priya; billing-service depends_on postgres-main; postgres-main hosted_in eu-west-1. Real outputs from adv_graph_rag.py:

1. Entity linking: the question names billing-service, so find_entities returns ['billing-service'].
2. Hop 1: follow owned_by from billing-service to Payments Team.
3. Hop 2: follow managed_by from Payments Team to Priya. follow_path returns ['Priya'].
4. Context for the LLM: with max_hops=2 it also gathers postgres-main and its eu-west-1 link, and turns the four edges into plain sentences (see output in the code example).

Neither fact alone answers the question. The graph chains them.

Code example

From adv_graph_rag.py:

from adv_graph_rag import TRIPLES, TripleStore, find_entities, follow_path, graph_context

kg = TripleStore(TRIPLES)
q = "Who manages the team that owns billing-service?"

print(find_entities(q, kg))                                         # ['billing-service']
print(follow_path(kg, "billing-service", ["owned_by", "managed_by"]))  # ['Priya']
for fact in graph_context(q, kg, max_hops=2):
    print("-", fact)
# - billing-service owned by Payments Team.
# - billing-service depends on postgres-main.
# - Payments Team managed by Priya.
# - postgres-main hosted in eu-west-1.

The core traversal is a breadth-first search that records every edge it crosses until it reaches max_hops.

In real projects teams often use Neo4j with Cypher queries, for example:

MATCH (:Service {name: "billing-service"})-[:OWNED_BY]->(t:Team)-[:MANAGED_BY]->(p:Person)
RETURN p.name

LangChain and LlamaIndex both have graph integrations that can ask an LLM to write such queries from text. Always run LLM-written queries with a read-only database user.

When to use it / when not to

• Use it for multi-hop questions over entities: org charts, service dependencies, supply chains, drug interactions, compliance links.
• Use it when you already have structured data (CMDB, CRM). You get triples for free.
• Avoid it for unstructured "explain this concept" questions; vector RAG is simpler and works well.
• Cost: building and maintaining a clean graph is real work, and LLM triple extraction makes errors.

Common mistakes

• Skipping entity resolution, so "Priya S." and "Priya Sharma" are two nodes and paths break.
• Traversing too many hops, which explodes into hundreds of loosely related facts.
• Trusting LLM-extracted triples without spot-checking. Wrong edges produce confident wrong answers.
• Letting an LLM run write queries (CREATE, DELETE) against the graph.

Recap

If you remember only three things:

• A knowledge graph stores facts as (subject, relation, object) triples that you can walk.
• It shines on multi-hop questions where the answer chains facts from different places.
• Entity resolution and a hop limit keep the graph clean and the results small.

Interview questions

1. What is a multi-hop question, and why does vector RAG struggle? <details><summary>Answer</summary> A question whose answer requires chaining two or more facts, often from different documents. Vector search retrieves chunks similar to the question, but the intermediate entity (e.g. the team name) is not in the question, so the second fact is hard to find. </details>

2. How do you build a knowledge graph from documents? <details><summary>Answer</summary> Extract entities and relations (LLM prompts or NER and relation-extraction models), resolve duplicate entities, validate against a schema of allowed relation types, and load into a graph store. Keep source references on each edge for citations. </details>

3. How do you combine KG retrieval with vector retrieval? <details><summary>Answer</summary> Run both: link entities and fetch their neighbourhood from the graph, run vector search for passages, then put graph facts and text chunks together in the prompt (or use graph facts to expand the vector query). </details>

Go deeper

• Unifying Large Language Models and Knowledge Graphs: A Roadmap (Pan et al.)
• Neo4j Cypher manual

Prev: Ensemble Retrieval · Next: GraphRAG · Back to phase: Phase 2