How Multimodal Models Relate Text, Images and Audio in One Request

Send a model a photo of a cracked bracket with the question "what is wrong with this part?" and a modern assistant can answer by inspecting the image itself, rather than relying on someone's written description of it. Systems that can take in several kinds of input within a single request are called multimodal models.

This explainer covers what "modality" means in this context, how such models connect words to regions of an image or moments in a recording, why that connection is the difficult part, and what limitations to plan for when you build features on top of them.

Why text-only models hit a wall

The first generation of large language models worked only with text. That is a serious constraint, because much of what people need help with does not start out as text: a photo of a skin rash, a screenshot of a stack trace, a voice memo, a chart buried in a PDF. To use a text-only model, someone first has to turn that material into a written description. Every such translation adds effort and throws away detail the describer did not think was important.

What a modality is

A modality is simply a kind of input: text, images, audio or video. A useful analogy is a very capable colleague who can only communicate through typed notes. To get their opinion on a photo, you would have to describe it in words first, and they would only ever know what you chose to mention. A multimodal model resembles the same colleague with eyes and ears: able to glance at the photo in your hand, or listen to the recording you forward.

The important point is that a multimodal model does more than accept several input types next to each other. It learns how they relate: which words refer to which part of an image, and which sounds belong to which visual event. Without that alignment, you would just have separate systems glued together.

What happens inside a combined request

At a high level, a request that mixes an image with a text question goes through these stages:

1. The caller sends mixed input, for example an image plus a question about it.
2. Each input is processed by a component designed for its type. Text is split into tokens; an image is typically cut into small patches that a vision encoder turns into vectors.
3. Both streams are projected into a shared numerical representation, so that a piece of text and a region of the image can be compared directly.
4. The model reasons over the combined sequence as one problem, not as two unrelated tasks whose results are merged afterwards.
5. The answer draws on everything that was provided and can refer to specific parts of the image in its text.

The shared representation is the hard part

Recognising objects in images and understanding text were each handled reasonably well before multimodal models arrived. The real challenge is building one internal space in which the word "hairline crack" ends up close to the pixels that actually show a hairline crack. Models learn this alignment from very large collections of paired data, such as images with captions or video with transcripts, which teaches them which descriptions go with which visual or audio patterns. When that alignment is weak for a particular domain, the model may see the image clearly in a technical sense and still connect it to the wrong words.

What changes for the user

The practical difference is who does the translation. With a text-only model, the person asking is the go-between, converting the real evidence into words. With a multimodal model, the model examines the evidence itself, and it often notices details the person would not have thought to mention.

Examples across domains

• Healthcare: a clinician can provide a scan together with a written patient history and receive an analysis that takes both into account.
• Education: a student can photograph a handwritten maths problem and get help without retyping every symbol.
• Customer support: a support tool can read a screenshot of an error dialog, capturing the exact message and its layout instead of depending on the user's paraphrase of it.

Why the approach matters

• It removes the describe-it-first bottleneck. People can hand over the actual material instead of translating it.
• It keeps information that words tend to lose. Layout, colour, tone of voice and surrounding visual context all carry meaning that a summary easily drops.
• It matches how questions really arrive. Real requests often arrive with an attachment such as a picture, screen capture or audio clip rather than as clean text.

Limitations to design around

Multimodal models can misread images. An ambiguous diagram, a blurry photo or a chart with tiny, dense labels can produce an answer that sounds confident and is simply wrong. The model has no built-in signal that it could not really see what it described.

When you build on these capabilities, treat visual analysis like any other model output:

• Ask for the model to quote or point to what it relied on, such as the exact text it read from a screenshot, so a person can check it quickly.
• Provide the highest-quality input you can; cropping to the relevant region often helps more than sending a large, busy image.
• Keep a human review step wherever the output drives a significant decision, especially in medical, legal or safety contexts.

Key takeaways

• A modality is a type of input; multimodal models accept several and, crucially, learn how they relate.
• Each input is encoded by a component suited to it and then mapped into a shared representation the model reasons over as a whole.
• The main benefit is removing the human translation step, which preserves detail that written descriptions lose.
• Visual reading errors remain common and can sound authoritative, so verification belongs in the design, not as an afterthought.