Research project · 2026
Comparing RAG approaches for banking questions
I tested three ways of retrieving answers from banking documents (standard vector RAG, GraphRAG and PageIndex) with the same documents, the same 40 questions and the same scoring rubric.
- Role
- Research and implementation
- Year
- 2026
- Type
- Research project
- Built with
- Python, LlamaIndex, Chroma, NetworkX, PageIndex, Ollama, Streamlit
The context
Retrieval-augmented generation (RAG) is the usual way to make a language model answer from an organisation's own documents. There are several ways to do the retrieval step, and comparisons often change more than one thing at a time. I wanted a fair test for banking content, where answers have to be accurate and traceable to a source.
What I built
- A fixed corpus of 18 banking documents, including three synthetic procedures that are clearly labelled as synthetic.
- 40 evaluation questions with gold answers, split evenly across five types: fact lookup, multi-hop reasoning, policy navigation, relationships, and cross-document comparison.
- Three pipelines: vector RAG (LlamaIndex and Chroma, 1,018 vectors), GraphRAG (a NetworkX knowledge graph), and PageIndex (a vectorless tree of 1,031 document sections).
- The same local model (qwen3:4b via Ollama), prompt and six-part rubric for all three, with citation checks and a Streamlit demo.
Try it
Answer accuracy, out of 5
- Vector RAG
- 3.58
- GraphRAG
- 3.20
- PageIndex
- 3.30
Decisions
Change one thing only
Only the retrieval method varies. The corpus, questions, answer model, prompt and scoring are identical across all three systems, so the differences in the results come from retrieval.
Answers must cite their evidence
Every answer has to cite the document it came from, and the system has to say so when the documents don't contain the answer, rather than guess. That matters more in banking than a fluent but unsupported answer.
Result
| System | Retrieval coverage | Answer accuracy | Relationship questions | Mean time per answer |
|---|---|---|---|---|
| Vector RAG | 4.53 | 3.58 | 3.38 | 42 s |
| GraphRAG | 4.45 | 3.20 | 4.00 | 37 s |
| PageIndex | 4.90 | 3.30 | 3.25 | 154 s |
Scores are out of 5, averaged over 40 questions. Vector RAG gave the most accurate answers overall (3.58), and GraphRAG was fastest (37 s). PageIndex had the best retrieval coverage (4.90) and the best answers on multi-hop and cross-document questions, but took 154 s per answer on average. GraphRAG was clearly best on relationship questions (4.00 vs 3.38). My recommendation is a router: vector RAG by default, GraphRAG for relationship questions, and PageIndex for complex research questions where speed matters less.