Inference Beyond Text: Tuning VLMs, Embeddings, Speech, Image and Video

Speeding up a text-only language model is a solved-enough problem that people treat its tricks as universal. They aren't. Once you ship a real product, text generation turns into one stage among several, and each stage has a different bottleneck. A tuning move that doubles LLM serving speed can be useless for an embedding model, and image or video generation brings compute and attention problems of its own. Here you'll meet vision-language models, embedders, ASR, TTS, speech-to-speech, image generators and video generators, starting from two basic execution patterns and then looking at metrics, hardware choices, caching, quantization, batching and quality trade-offs for each one. The thread running through it: inference engineering depends on the workload.

Why one playbook fails

Real applications chain models together. A voice assistant might take audio, run speech recognition, reason over the text, then synthesize speech. A document system might run a PDF through OCR, a vision model and embeddings before retrieval and an LLM. A video tool might go from text to a generated clip through iterative denoising and finally media decoding.

Each hop has its own cost profile, so the thing you optimize changes with it. What makes an autoregressive language model quick can't just be pasted onto an image or video pipeline.

Two patterns cover most of it

Nearly every generative system follows one of two shapes.

The first is autoregressive generation. The model produces output one step at a time, each token conditioned on what came before:

Input
  ↓
Token 1
  ↓
Token 2
  ↓
Token 3
  ↓
...

That's the familiar LLM loop. Vision-language models, embedding systems built on language-model backbones, ASR decoders and many TTS models can borrow plenty of transformer serving ideas from it.

The second is iterative denoising, the diffusion approach, where a noisy starting point gets refined step after step:

Random / noisy latent
        ↓
      Step 1
        ↓
      Step 2
        ↓
      Step 3
        ↓
       ...
        ↓
Final image/video

Image and video generation mostly live here.

This matters because you can't use one definition of latency, throughput or efficiency across all of them. For a chatbot, time to first token is often the headline number. For speech synthesis, what counts is time to first meaningful audio. For long transcription jobs, real-time factor tends to dominate. For image generation, cutting denoising steps may beat shaving kernel latency. For video, attention and enormous tensors can swallow the whole workload.

Vision-language models turn pictures into tokens

A VLM pairs a vision encoder with a language model. The encoder converts pixels into representations the language model can read alongside text:

Image / Video
      │
      ▼
Vision Encoder
      │
      ▼
Visual Tokens
      │
      ▼
Language Model
      │
      ▼
Text Response

The interesting consequence is that an image lengthens the input sequence. A single high-resolution image can contribute on the order of a thousand visual tokens. More visual detail means more tokens, which means more context and a bigger KV cache.

So tuning a VLM looks a lot like tuning a long-context LLM. Reach for quantized KV caches, speculative decoding, prefix caching, tensor parallelism, offloading the cache, or splitting prefill from decode. Prefix reuse pays off especially for repeated images and multi-turn chats, and as inputs get longer, memory management matters more and more.

Video understanding gets expensive fast

Images cost something. Video multiplies it. At 24 frames per second, even a brief clip yields a huge pile of visual tokens. A four-second high-definition clip could in principle approach 100,000 input tokens before any practical downsampling.

That makes downsampling your main dial for trading quality against speed. You can lower resolution, frame rate, frame count or visual token count, though each cut removes information. So skip the question of how to make video inference faster. Ask what the smallest visual representation is that still meets the application's quality bar. That's the real inference engineering mindset: tune the workload, not merely the kernel.

Omni-modal models aren't always the answer

The industry is heading toward models that accept and emit several modalities at once:

Text ─────┐
Image ────┤
Audio ────┼──► Omni Model ──► Text / Audio / Image
Video ────┘

That sounds ideal, but generality has a price. A dedicated OCR model can beat the OCR ability buried inside a large VLM, and a purpose-built speech model can run more efficiently than a general multimodal one. Production systems therefore tend to become pipelines of specialists:

┌──► OCR
PDF/Image ───┤
             ├──► Vision Model
             └──► Embedding Model
                       │
                       ▼
                     LLM

The engineering rule that follows is to make each component independently optimizable and scalable.

Embedding models want throughput

Embedding models differ fundamentally from generative LLMs. They turn content into a fixed-size vector:

Document / Image
       ↓
Embedding Model
       ↓
[0.12, -0.43, 0.87, ...]

Those vectors power RAG, semantic search, recommendations, agent memory and document retrieval. What matters most is the shape of the workload, and two shapes look very different.

Bulk embedding is one, such as pushing ten million documents into a vector database:

10 million documents
       ↓
Embedding pipeline
       ↓
Vector database

There, throughput and cost win. Online retrieval is the other, where a user query gets embedded and searched immediately:

User query
    ↓
Embedding
    ↓
Vector search

There, milliseconds win. When both are big enough, separate deployments often come out ahead, since each can be tuned for its own goal.

Dimensionality and compatibility

Embeddings add one more quality-versus-speed dial: vector size. Vectors may hold hundreds or thousands of values. Bigger ones keep more information but raise storage, retrieval and similarity-computation costs. Some modern embedding methods support variable effective dimensionality, designed with exactly this trade-off in mind.

There's also a common misunderstanding. Two vectors with the same length aren't automatically comparable, because each embedding model builds its own semantic space. Swapping the model isn't like swapping one API for another. You usually have to rethink the retrieval system around it.

Batching is the main lever for embeddings

Unlike autoregressive decoding, embedding models process tokens largely in parallel, which changes the strategy. Prefix caching and prefill/decode disaggregation aren't central here. Concentrate on large batches, fast tokenization, queueing, horizontal scaling, optimized kernels and quantization. Embedding models can take much bigger batches than generative ones, sometimes dozens or hundreds of inputs together.

A sensible production layout puts a queue and a batch manager in front of several replicas:

Requests
   ↓
Queue
   ↓
Batch Manager
   ↓
GPU Replica 1
GPU Replica 2
GPU Replica 3
GPU Replica 4

For high-volume embedding, one GPU per horizontally scaled replica often beats a complicated multi-GPU model-parallel setup.

Speech recognition: the decoder dominates

ASR converts audio to text. A typical pipeline runs preprocessing, an encoder and then a decoder:

Audio
  ↓
Audio preprocessing
  ↓
Encoder
  ↓
Autoregressive Decoder
  ↓
Transcript

ASR isn't an LLM, but its decoder acts like one and eats most of the inference time. That makes LLM-style optimizations especially effective. For live transcription, raw throughput isn't the objective. The system has to feel instant, and a reasonable target is around 200 ms round trip for a live audio chunk.

Streaming transforms the experience

One of the largest ASR gains needs no model change at all, only a different serving design. Rather than the upload, wait and receive pattern:

Upload audio
      ↓
Wait
      ↓
Receive transcript

switch to a continuous flow:

Continuous Audio
       ↓
     VAD
       ↓
Audio Chunks
       ↓
    ASR GPU
       ↓
Streaming Transcript

A WebSocket holds the stream open while a voice activity detection model picks out the stretches that contain meaningful audio. The lesson is that inference performance isn't always a runtime problem. Sometimes orchestration delivers the biggest win.

Long recordings need parallelism

Long files are a different animal from interactive dictation. Instead of working through them in order:

Chunk 1 → Chunk 2 → Chunk 3 → Chunk 4

you can build a parallel pipeline:

┌──► GPU 1 ──► Chunk 1
Audio → VAD ─────┼──► GPU 2 ──► Chunk 2
                 ├──► GPU 3 ──► Chunk 3
                 └──► GPU 4 ──► Chunk 4
                              ↓
                         Stitch by time

You lose some sequential context reuse but gain a great deal of throughput. Real-time factor is the useful metric for long-form transcription, and parallel chunk processing suits high-throughput jobs. To protect quality, add post-processing that spots suspicious repetition and reruns just the problem chunks.

TTS: more tokens per second isn't the goal

Text-to-speech runs the opposite direction from ASR:

Text
 ↓
TTS Model
 ↓
Audio Tokens
 ↓
Audio Decoder
 ↓
Waveform

Modern TTS models often use LLM-like architectures, so optimized kernels, batching and low-precision inference carry over from LLM serving. The metric differs in a subtle way, though. Generating 500 audio tokens per second isn't better than 100 if the audio decoder only needs roughly real-time output. Once generation keeps up with real time, the question becomes how many concurrent users one GPU can support. That's optimizing for the product requirement, not the benchmark number.

Image generation: drop the LLM instincts

Image generators are mostly iterative denoisers, not token-by-token writers:

Noise
 ↓
Denoising Step 1
 ↓
Denoising Step 2
 ↓
...
 ↓
Image

Unlike LLM decoding, this workload is typically compute-bound, so memory bandwidth stops being the main enemy. The toolbox shifts to kernel selection, kernel fusion, efficient attention, GEMM tuning, Torch compilation, lower precision and doing less denoising work.

Kernels matter a lot

Attention kernels are especially significant in image generation. Depending on GPU architecture and implementation, you can evaluate optimized options such as FlashAttention, DeepGEMM, CuTe and CUTLASS. Normalization operations like RMSNorm are also good fusion candidates.

The bigger lesson is that the best inference engine depends on both workload and hardware. PyTorch gives flexibility, specialized implementations ship tuned kernels, and compilation can fuse operations into a GPU-specific path. The right answer comes from benchmarking your actual model on your actual hardware.

An effective trick for diffusion

Diffusion models run many steps. Early ones set the broad structure and later ones refine detail. That opens an opportunity: cut expensive guidance work in the later steps. One approach skips classifier-free guidance late in the process while keeping the step count unchanged. In an example with 50 steps, model passes drop from 100 to 80, and quality is generally preserved.

The specific trick matters less than the principle. Not every iteration contributes equally to the final result, and inference optimization can exploit that imbalance.

Video generation is another scale

Video generation is image generation with an added time dimension. Where an image latent is:

Height × Width

a video latent becomes:

Height × Width × Time

That makes the compute problem far bigger. Video is compute-bound, and systems often occupy whole multi-GPU nodes with tiny batch sizes, even a batch of one, because all the GPUs cooperate on a single clip.

That draws a sharp line from text generation. With text, a larger batch can raise throughput. With video, batching may help little, since one generation can already consume most of the compute. So the quickest route to cheaper video is often a faster model.

Attention dominates video

In video generation, attention can eat a huge portion of the compute, an estimated 70 to 80 percent in some workloads. That shifts priorities. Rather than mainly quantizing weights, engineers work on attention kernels, attention quantization, caching, context parallelism and GPU-level parallel execution.

Caching for video works differently

Video generation doesn't use the autoregressive KV-cache approach. Instead, systems reuse parts of earlier computation, in two broad ways. Timestep caching reuses outputs from selected denoising steps. Transformer caching reuses hidden states and skips work inside parts of the transformer. Both can deliver large speedups, and both can seriously degrade quality depending on the algorithm. So treat every optimization as a loop:

Optimization
     ↓
Speed ↑
     ↓
Quality ?
     ↓
Evaluate
     ↓
Production

Never assume a caching optimization is safe just because the video still looks plausible.

Quantization follows the bottleneck

For LLMs, quantization usually targets big linear layers and weights, since LLM inference is often limited by memory bandwidth. Video generation is limited by compute, especially attention, so attention itself becomes a tempting quantization target. But attention is also very sensitive to numerical error.

Two options exist. You can quantize in blocks using microscaling formats like MXFP8, or you can quantize selectively, varying precision by step:

Early steps   → FP16
Middle steps  → Lower precision
Late steps    → Lower precision

or by layer:

Input layer   → FP16
Hidden layers → Lower precision
Output layer  → FP16

Preserve precision where the model is fragile, and optimize hard where it isn't.

No universal optimization exists

Across these modalities the winning strategies look nothing alike:

| Modality         | Dominant Concern            | High-Value Techniques                                 |
| ---------------- | --------------------------- | ----------------------------------------------------- |
| VLM              | Long visual context         | KV optimization, caching, parallelism                 |
| Embeddings       | High request volume         | Batching, queueing, horizontal scaling                |
| ASR              | Decoder latency             | Optimized decoder, streaming, parallel chunks         |
| TTS              | Real-time concurrency       | Streaming, batching, efficient decoding               |
| Image generation | Compute                     | Kernel fusion, compilation, fewer expensive passes    |
| Video generation | Massive compute + attention | Attention kernels, caching, quantization, parallelism |

That's why inference engineering is closer to systems work than to simply speeding up an LLM.

Takeaways

1. Multimodal inference depends on the workload. The best architecture follows the execution pattern and the product requirements.
2. Autoregressive and diffusion models have fundamentally different bottlenecks.
3. VLMs convert visual input into extra context, so KV-cache and long-context tuning matter more.
4. Embedding systems are mostly batching and throughput problems, not decode-latency problems.
5. ASR gains come largely from orchestration and streaming, not only faster kernels.
6. For TTS, once generation outpaces real time, tuning turns into a question of concurrent users.
7. Image generation is compute-bound, which makes kernels, fusion, compilation and denoising-step reduction critical.
8. Video generation pushes all of this further, with huge attention workloads and multi-GPU execution.
9. Caching is modality-specific. KV caching drives autoregressive generation, while diffusion systems reuse computation in other ways.
10. Quantize where the bottleneck is. Weights suit memory-bound LLMs, while attention can matter more for compute-bound video.

Where this leads

AI products are becoming multimodal by default. Text, images, audio, video and embeddings are increasingly wired into the same application, so the job grows from tuning one model to tuning an entire pipeline. The question to ask is no longer which optimization technique to use. It's where the bottleneck sits in this particular workload, and what trade-off you can safely make to remove it.

That mindset holds from one GPU serving an embedding model to a multi-GPU cluster generating real-time video. Know the workload, locate the bottleneck, test the trade-off you're willing to make, and tune the whole system rather than only the model.