GraphRAG Multi-Hop Reasoning 2026: Why Basic RAG Fails (and How to Fix It)

Basic RAG works perfectly in the demo — until you deploy it. The core assumption is that semantic similarity implies relevance, but the multi-hop questions users actually ask — "Who leads the company Acme Corp acquired?" — require structured connections from A to B to C, and those concepts rarely live in the same text chunk. In 2026 the answer is increasingly clear: combine the structural rigor of knowledge graphs with the semantic power of vector search. That is GraphRAG. This post builds the full pipeline from scratch in Python.

1. Why Basic RAG Falls Apart on Multi-Hop Questions

Imagine a vector database full of corporate contracts. A user asks, "Who leads the company that Acme Corp acquired?" The answer lives in Chunk 2: "Sarah Connor was appointed CEO of BetaTech." But vector search only understands semantic similarity: it embeds the query, finds the chunks closest to "Acme" and "leadership," and Chunk 2 never surfaces because "Sarah Connor" and "CEO of BetaTech" share zero semantic overlap with "Acme Corp." The naive approach assumes that if two pieces of text are about the same topic, they will be close in embedding space — but multi-hop questions are precisely the case where that assumption breaks. The concepts are connected through a chain of relationships, not through shared vocabulary. Basic RAG also struggles with global summarization: when the answer spans dozens of chunks, top-k retrieval is structurally guaranteed to miss pieces, because every chunk is retrieved in isolation and the aggregator never sees the whole picture. These two failure modes — multi-hop connection and cross-document synthesis — are exactly what enterprise users hit within weeks of deploying a naive RAG pipeline.

// Why naive vector search fails on multi-hop questions.
// Query: "Who leads the company that Acme Corp acquired?"
// Chunk 2 holds the answer but never gets retrieved,
// because "Sarah Connor" has no semantic overlap with "Acme".

const query = "Who leads the company that Acme Corp acquired?";

const chunks = [
  { id: 1, text: "Acme Corp acquired BetaTech in 2025 for $2.1B..." },
  { id: 2, text: "Sarah Connor was appointed CEO of BetaTech..." }, // <- answer
  { id: 3, text: "BetaTech builds enterprise scheduling software..." },
];

const topK = await vectorSearch(query, chunks, { k: 2 });
// Returns chunk 1 and chunk 3 — chunk 2 is invisible to
// semantic similarity, yet it holds the answer.

2. The Core Idea: Build the Graph First, Retrieve Second

GraphRAG makes the LLM process documents during ingestion, extracting nodes (entities) and edges (relationships) to create a knowledge graph from your chunks. Now "Acme Corp -[ACQUIRED]-> BetaTech" and "Sarah Connor -[LEADS]-> BetaTech" are explicit structured facts. Retrieval no longer relies on vector similarity alone: locate a starting entity by vector, then traverse the graph hop by hop — each hop is a deterministic relationship query, not a fuzzy semantic guess. This is the key architectural shift: you stop asking the retriever to understand the question and start asking it to find the starting point, then let the graph do the reasoning. The graph captures the structure that embeddings cannot, and the embeddings capture the semantic recall that exact graph matching cannot. Together they cover the two failure modes of naive RAG.

// GraphRAG ingestion: let the LLM extract nodes (entities)
// and edges (relationships) from each chunk to build a
// knowledge graph. This is the key architectural shift.

import { ChatOpenAI } from "@langchain/openai";

const EXTRACT_PROMPT = `Extract all entities and relationships
from the text as JSON. Use the schema:
{ "nodes": [{"name": string, "type": string}],
  "edges": [{"from": string, "to": string, "rel": string}] }`;

async function extractGraph(text: string) {
  const llm = new ChatOpenAI({ model: "gpt-4o", temperature: 0 });
  const res = await llm.invoke(EXTRACT_PROMPT + "\n\n" + text);
  return JSON.parse(res.content as string);
}

// Chunk 2 -> { nodes: [{name: "Sarah Connor", type: "person"},
//                       {name: "BetaTech", type: "company"}],
//              edges: [{from: "Sarah Connor", to: "BetaTech",
//                       rel: "LEADS"}] }

3. Entity Extraction: The LLM as Graph Builder

The heart of the ingestion pipeline is a structured extraction prompt: ask the LLM to output nodes and edges as JSON, with temperature at zero for reproducibility. Extraction quality dictates graph quality, so the prompt should pin down entity types (person/company) and relationship types (ACQUIRED/LEADS). You also want to normalize entity names — "Acme Corp," "Acme Corporation," and "ACME" should all merge into one node, otherwise the graph fragments into near-duplicates that defeat the whole purpose. In practice, a small post-processing pass with a canonical-name map or an embedding-based dedupe step pays for itself. This step essentially compiles unstructured text into structured knowledge — every downstream query stands on this graph, so it is worth investing in the extraction prompt, the dedupe logic, and the schema design before you scale ingestion to thousands of documents.

// Store the graph in Neo4j (or any graph DB) alongside
// the vector index. Entities get both a node and an embedding,
// so you can start from either semantic or structural search.

from neo4j import GraphDatabase
import openai

driver = GraphDatabase.driver("bolt://localhost:7687")

def upsert_entity(tx, name, etype, embedding):
    tx.run(
        """MERGE (e:Entity {name: $name})
           SET e.type = $etype, e.embedding = $embedding""",
        name=name, etype=etype, embedding=embedding,
    )

def upsert_relationship(tx, src, dst, rel):
    tx.run(
        """MATCH (a:Entity {name: $src}), (b:Entity {name: $dst})
           MERGE (a)-[:$rel]->(b)""",
        src=src, dst=dst, rel=rel,
    )

# Ingest pipeline: for each chunk -> extractGraph() -> upsert
with driver.session() as s:
    s.execute_write(upsert_entity, "Acme Corp", "company", emb1)
    s.execute_write(upsert_entity, "BetaTech", "company", emb2)
    s.execute_write(upsert_relationship, "Acme Corp", "BetaTech", "ACQUIRED")

4. Storage: Graph Database + Vector Index in Parallel

The knowledge graph lives in Neo4j (or any graph DB), and entities keep vector embeddings alongside. You can enter semantically (vector search finds the seed entity) or structurally (query relationships directly). This dual-track design is where GraphRAG gets its resilience: vectors handle fuzzy recall, the graph handles exact connections. When you upsert an entity, you MERGE on its normalized name and set both the type and the embedding in one statement; relationships are created with MERGE too, so re-ingesting a document is idempotent. Keep the two stores consistent by running ingestion as a single transaction per chunk — extract, upsert nodes, upsert edges — and you avoid the classic drift where the vector index knows about an entity the graph does not, or vice versa.

5. Multi-Hop Retrieval: Vectors Find the Start, the Graph Walks the Rest

Answering "who leads the company Acme acquired" takes three steps: vector retrieval locates Acme Corp; traverse the ACQUIRED edge to BetaTech; traverse LEADS backward to Sarah Connor. Every step is a deterministic Cypher query, so answers are stable and reproducible — exactly what enterprise workloads demand. A hybrid retriever combines vector recall with k-hop graph traversal and hands the context to the LLM for the final answer. The hop count is a tunable parameter: too few hops and you miss the answer, too many and you drown the context window in irrelevant neighbors. Start with three hops, inspect the retrieval quality on a dev set, and expand only where the evaluation shows it helps. Because the graph traversal is deterministic, you can unit-test the retrieval path itself — something that is nearly impossible with pure vector search.

// Retrieval: first hop via vector search, then traverse
// the graph for the remaining hops. This answers the query
// that defeated naive RAG.

def answer_multihop(query: str) -> str:
    # Hop 1: find "Acme Corp" in the vector index
    seed = vector_search(query, top_k=5)          # -> Acme Corp
    # Hop 2: traverse ACQUIRED edge
    acquired = graph.query(
        "MATCH (a:Entity {name:$name})-[:ACQUIRED]->(t) RETURN t.name",
        name=seed[0],
    )                                            # -> BetaTech
    # Hop 3: traverse LEADS edge
    leader = graph.query(
        "MATCH (c:Entity {name:$name})<-[:LEADS]-(p:Person) RETURN p.name",
        name=acquired[0],
    )                                            # -> Sarah Connor
    return leader[0]

print(answer_multihop("Who leads the company Acme acquired?"))
# Sarah Connor

6. When to Upgrade to GraphRAG

If user questions cluster around single facts ("what's the contract number?"), basic RAG is enough. The moment you see cross-entity relationship questions, summarization over many documents, or "why" questions that need a reasoning path, it's time for GraphRAG. Migration isn't cheap — you pay for entity extraction and a graph database — but you get enterprise-grade accuracy and explainability. Start with a small pilot domain, say a contract relationship graph, and expand once it's proven. Measure before and after on the same question set: if the multi-hop pass rate does not improve meaningfully, the graph schema probably needs work rather than the retrieval code. In practice, teams that start with a well-normalized graph and a three-hop retriever see the biggest gains on exactly the questions that used to fail — the ones where the answer lived two chunks away from the question.

// Full pipeline wiring in Python: chunk -> extract ->
// graph + vector -> hybrid retrieval -> LLM answer.

import asyncio
from graphrag import GraphIndex, HybridRetriever

async def main():
    docs = load_contracts("contracts/")          # corporate contracts

    index = GraphIndex(llm="gpt-4o", neo4j_uri="bolt://localhost:7687")
    await index.ingest(docs)                     # extraction + upsert

    retriever = HybridRetriever(index)
    ctx = retriever.retrieve(
        "Who leads the company that Acme Corp acquired?",
        vector_k=5, graph_hops=3,
    )
    answer = await llm_complete(ctx.context_text())
    print(answer)

asyncio.run(main())