Phase 9: Multimodal AI
Teach AI to see, hear and search pictures. Real data is not only text. Photos, charts, screenshots, speech and video all matter, yet a transformer only reads sequences of vectors. This phase shows how each kind of data becomes tokens, what that costs, and how to search and answer across images and text together.
After this phase you can
- Count the tokens an image or a video costs and explain why doubling resolution quadruples them
- Explain how speech becomes a spectrogram and score a transcript with word error rate
- Work out a CLIP-style contrastive loss and fuse caption and image search into a multimodal RAG prompt
Before you start: Phase 3 (transformers, attention) and the embeddings lesson from Phase 1. Basic Python and school maths.
Lessons in this phase
- Vision Language Models Intermediate · 10 min
A vision-language model (VLM) is a language model that can read pictures: an image encoder turns the image into vectors, a small projector reshapes them into "image tokens".
- How Models See Images Intermediate · 10 min
A vision transformer cuts an image into small square patches, turns each patch into one vector, adds position information, and then runs the same self-attention a text model uses, so the number of patches.
- Audio and Speech to Text Intermediate · 10 min
Speech recognition turns a sound wave into a list of numbers, cuts it into short frames, measures which frequencies each frame holds (a spectrogram), and lets a transformer encoder-decoder write the words.
- Text to Speech Intermediate · 9 min
Text to speech (TTS) cleans the text, predicts a spectrogram that describes how it should sound, and then a vocoder turns that spectrogram into an audio waveform, with latency.
- Video Understanding Advanced · 10 min
Most video models treat a video as a short list of sampled frames, each encoded like an image, so the real engineering work is choosing which frames to keep, paying the token bill, keeping time order.
- Multimodal Embeddings and CLIP Intermediate · 11 min
CLIP trains one encoder for images and one for text so that a picture and its caption land close together in the same vector space, using a contrastive loss that rewards the right pairs and punishes the wrong ones.
- Multimodal RAG Advanced · 11 min
Multimodal RAG makes images (and charts, slides, scans) searchable and then shows the best ones to a vision-language model, usually by indexing both a caption and an image embedding and fusing the two result lists.