AI Engineer Roadmap All lessons Interactive Quiz MernStackDev GitHub

Multimodal RAG

Advanced · 11 min read · Updated · By Saurabh Pathak

In short: Multimodal RAG makes images (and charts, slides, scans) searchable and then shows the best ones to a vision-language model, usually by indexing both a caption and an image embedding and fusing the two result lists.

Play the interactive lesson: Multimodal RAGRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Advanced · Reading time: 11 min · Phase: 9 · Multimodal AI

Before you start: Multi Modal RAG gives the overview; this lesson goes deeper with working retrieval code. You should also know Multimodal Embeddings and CLIP, Vision Language Models and BM25.

Words to know:

Explain it like I'm new

You are looking for "last year's revenue chart" in a shared folder of 5,000 images. Two ways to look:

  1. Read the labels. Each file has a caption someone typed. You search the captions for "revenue" and "chart". Fast, but a chart captioned "Figure 4" never turns up.
  2. Look at the pictures. A helper who understands both pictures and words can say, "this image looks like a revenue bar chart". It finds "Figure 4", but it can also pull up a similar-looking chart from the wrong year.

Using both and merging the results is more reliable than either one. Then you hand the real pictures, not their labels, to a model that can look at them and answer.

That is the whole idea. The details are in how you index, how you merge and how you check the result.

Watch it

Multimodal RAG: Find the right images and text, then show them to the model.
Multimodal RAG: Find the right images and text, then show them to the model.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

Index time:   for each image -> caption (person or VLM), embedding, id, path, page, source doc
Query time:   caption list  = BM25(question, captions and text chunks)
              vector list   = cosine(query vector, image vectors)
              fused list    = RRF(caption list, vector list)      score = sum 1 / (k + rank)
Answer time:  prompt = instructions + sources named by id (+ the actual images attached)

Three indexing designs

DesignHowStrengthWeak spot
Captions onlyDescribe each image in text (by hand or with a VLM), index textReuses your text stack; easy to debugCaption quality caps recall. "Figure 4." is a dead end
Multimodal embeddings onlyEmbed images with a CLIP-style model; embed the query text with its text sideFinds images with no useful captionWeak on exact words, numbers, names and fine text; similar-looking decoys
Both, fusedRun both searches and merge with RRFEach covers the other's gapsTwo indexes to maintain

Passing images to the model. Retrieval finds ids. The generation step attaches the original images (or crops) to the prompt and names each one by id so the model can cite it. Do not send only the caption: the point of a VLM is that it looks at the pixels. Mind the image token cost: send the top few, resized.

Worked example (from mm_rag.py)

Corpus: two text chunks (t1, t2) and four images (img1 bar chart of quarterly revenue, img2 pump diagram, img3 captioned only "Figure 4." but really a chart, img4 team photo). Question: "how did revenue change by quarter".

  1. Caption search (BM25) returns only img1 (score 2.4538) and t1 (2.0389). The chart img3 is missing, because its caption shares no words with the question.
  2. Vector search (cosine on toy features) returns img1 0.9997, img3 0.9787, img2 0.2035, img4 0.0000. Here the uncaptioned chart is found.
  3. Reciprocal rank fusion with k = 60: img1 is rank 1 in both lists, so it gets 1/61 + 1/61 = 0.0328. img3 (rank 2 in one list only) gets 1/62 = 0.0161. t1 (rank 2 in the caption list) also gets 0.0161. img2 (rank 3 in vectors) gets 1/63 = 0.0159.
  4. Top 3 after fusion: img1, img3, t1. Two images are attached to the prompt.
  5. The prompt names each source, so the model can cite [img1]:
[img1] (image attached) caption: Bar chart of quarterly revenue by region.
[img3] (image attached) caption: Figure 4.
[t1] Revenue grew steadily in the third quarter across all regions.

Question: how did revenue change by quarter

The agreement bonus is the heart of RRF: an item two searches both like beats an item only one likes. Ties (such as img3 and t1 at 0.0161) are broken by id in this demo; real systems often use a reranker.

Evaluation. Test retrieval separately from the answer, as in RAG evaluation:

Failure modes

flowchart LR
    Q[Question] --> C[Caption search BM25]
    Q --> V[Image vector search]
    C --> F[Fuse with RRF]
    V --> F
    F --> P[Prompt with ids + attached images]
    P --> M[Vision-language model]
    M --> A[Answer with cited ids]

Code example

From mm_rag.py (run python 11-multimodal-ai/code/mm_rag.py):

from mm_rag import CORPUS, QUERY_TEXT, QUERY_FEATURES, rank_by_caption, rank_by_vector, rrf, build_prompt

fused = rrf([rank_by_caption(QUERY_TEXT, CORPUS), rank_by_vector(QUERY_FEATURES, CORPUS)])
top = [doc_id for doc_id, _ in fused[:3]]
print(top)                                   # ['img1', 'img3', 't1']
print(build_prompt(QUERY_TEXT, top, CORPUS))

In real projects the toy feature vectors become embeddings from a CLIP-style model (see the Transformers documentation), the lists come from a vector database and a keyword index, and the final prompt sends the actual image files to a VLM. The vector math is the same as in rag_vector_math.py.

When to use it / when not to

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Why index both captions and image embeddings? <details><summary>Answer</summary> Captions give exact-word matching and are easy to debug, but a poor caption hides an image. Embeddings find images by content even without good captions, but are weaker on exact words and can return look-alikes. Fusing both covers each side's gaps. </details>

2. How does reciprocal rank fusion work, and why use it? <details><summary>Answer</summary> Each list gives an item 1 / (k + rank), the scores are summed across lists, and items are sorted by the total. It needs only ranks, not comparable scores, and rewards items that several searches agree on. With k = 60, an item ranked first in two lists scores 2/61, about 0.033. </details>

3. Your multimodal RAG answers chart questions wrongly. How do you find the cause? <details><summary>Answer</summary> Check retrieval first: was the right chart in the top k (recall@k on labelled questions)? If not, improve captions, add embeddings, OCR the text and use filters. If it was retrieved, check that the actual image, not just its caption, was attached at suitable resolution, and that the prompt asked for cited values you can verify. </details>

Go deeper

Prev: Multimodal Embeddings and CLIP · Next: Phase 9 checkpoint quiz · Back to phase: Phase 9