Multimodal AI

Vision-Language Models

Vision-language models (VLMs) are the most commercially mature category of multimodal AI, with multiple production-quality systems available through APIs and for local deployment. They enable a wide range of applications: image captioning, visual question answering, document understanding, chart analysis, screenshot parsing, medical image interpretation, and visual grounding.

CLIP & Contrastive Learning

CLIP (Contrastive Language-Image Pretraining), introduced by Radford et al. (OpenAI, 2021), is the foundational vision-language model. CLIP jointly trains a vision encoder and a text encoder using contrastive learning on 400 million (image, text) pairs scraped from the internet. The training objective maximises the cosine similarity of matched image-text pairs while minimising it for mismatched pairs within each batch. The result is a shared embedding space where semantically similar images and text descriptions are geometrically proximate.

The key capability that CLIP unlocks is zero-shot classification: given candidate text labels like "a photo of a cat" and "a photo of a dog", CLIP can classify an image without any image-specific training, purely by comparing the image embedding to the text embeddings. On ImageNet zero-shot classification, ViT-L/14 CLIP achieves 76.2% top-1 accuracy — matching ResNet-50 trained on 1.2 million labelled ImageNet examples. This is extraordinary: CLIP was never shown an ImageNet label during training.

Beyond classification, CLIP embeddings are used for: image-text retrieval (find images semantically matching a text query), text-guided image editing (adjust latents in CLIP embedding space), aesthetic scoring (rank images by aesthetic quality using CLIP similarity to aesthetic descriptors), and as the conditioning signal for text-to-image diffusion models (as covered in Part 11).

The CLIP architecture family has expanded significantly. OpenCLIP trained larger models on LAION-5B (5 billion image-text pairs), with ViT-G/14 achieving 80.1% ImageNet zero-shot. SigLIP (Google) replaces the contrastive loss with a sigmoid binary cross-entropy loss, enabling better performance without large batch sizes. CLIP Interrogator reverses the flow: given an image, it generates the text prompt that best characterises it — used extensively in reverse-engineering prompts from Stable Diffusion outputs.

GPT-4V/4o & Multimodal LLMs

GPT-4V (and its successor GPT-4o) extended the GPT-4 language model with a vision encoder, enabling simultaneous reasoning over text and images. The vision integration uses a dynamic tiling approach: high-resolution images are divided into 512×512 tiles, each processed by the vision encoder and compressed into 256 tokens (for "high" detail mode; 85 tokens for "low" detail mode). The resulting token sequence is concatenated with the text tokens and processed by the LLM in a single forward pass.

GPT-4o's capabilities across visual tasks represent a qualitative step beyond earlier VLMs: it can read and interpret handwritten notes, extract data from complex charts and graphs, reason about spatial relationships in diagrams, analyse multi-panel scientific figures, describe screenshots with UI element context, and interpret medical imaging with appropriate clinical framing (though not cleared for clinical use). The model handles up to 20 images per API call in the current implementation.

Claude 3.5 Sonnet is the primary competitor, with particularly strong performance on document understanding, code from screenshots, and complex diagram interpretation. Claude's multimodal capabilities extend to PDFs natively, removing the need to convert to images before processing. The 200K token context window allows processing very long documents with embedded images in a single call.

Gemini 1.5 Pro takes the most aggressive multimodal approach: a 1M-token context window that accepts interleaved text, images, audio, and video in a single prompt. This enables use cases like analysing an entire long video with accompanying transcript, or processing a book-length document with embedded figures. Google's approach of training all modalities jointly from scratch (rather than connecting pretrained modules) achieves tighter integration at the cost of significantly higher pre-training compute.

Open-Source VLMs: LLaVA & Idefics

LLaVA (Large Language and Vision Assistant) demonstrated that high-quality visual instruction following can be achieved by simply connecting a CLIP vision encoder to a powerful LLM (originally LLaMA) with a small learnable projection layer, trained on GPT-4-generated visual instruction data. LLaVA-1.6 (NeXT) extended this with dynamic high-resolution processing (up to 672×672), achieving competitive performance with GPT-4V on multiple benchmarks while running locally on a single A100.

InstructBLIP (Salesforce) introduced the Q-Former (Querying Transformer) architecture: a bottleneck transformer that extracts a fixed number of query embeddings from the image encoder output, regardless of image resolution. This fixed-length visual representation is then passed to the LLM, decoupling image resolution from LLM sequence length. The Q-Former architecture was adopted in BLIP-2 and later models.

Idefics (HuggingFace) is an open reproduction of Flamingo's architecture, enabling interleaved image-text inputs — the model can alternate between text passages and images within a single prompt, enabling few-shot visual learning. Idefics-2 extended this with higher resolution support and improved instruction following, making it the most capable openly available model for interleaved multimodal tasks.

Audio-Text Integration

Audio-text multimodal systems bridge the gap between spoken language, environmental sound, and the text-based reasoning capabilities of large language models. The two primary directions are speech-to-text (transcription and translation) and audio understanding (classifying, describing, and reasoning about audio content beyond speech).

Speech-Language Models

Whisper (OpenAI) is the dominant speech recognition model for production use. Trained on 680,000 hours of multilingual audio with weak supervision (internet audio paired with subtitles/transcripts), it achieves near-human word error rates across 99 languages without language-specific fine-tuning. Key architectural choices: a simple encoder-decoder transformer where the audio encoder processes 30-second mel-spectrogram windows and the decoder generates transcription tokens autoregressively. Whisper large-v3 achieves 2.7% WER on LibriSpeech clean — below the 5.8% human baseline.

Seamless-M4T (Meta, 2023) represents a more ambitious integration: a single model handling speech-to-text, text-to-speech, speech-to-speech, and text-to-text translation across 100 languages. The unified architecture learns shared representations across modalities and language pairs, enabling transfer learning effects that improve low-resource languages via cross-lingual transfer. Seamless-M4T is particularly valuable for real-time translation workflows where multi-step pipelines (speech recognition → translation → synthesis) introduce latency and compounding errors.

GPT-4o Audio introduced end-to-end speech understanding directly within a multimodal LLM, without an intermediate transcription step. This enables the model to reason about prosody, emotion, speaker identity, and background audio — information that is lost when audio is first converted to text. Real-time speech conversation with sub-300ms latency, voice emotion detection, and audio-grounded reasoning are capabilities that require this tighter audio-language integration.

Audio Understanding

Beyond speech, audio understanding covers environmental sound recognition, music analysis, and acoustic event detection. CLAP (Contrastive Language-Audio Pretraining) applies the CLIP contrastive learning paradigm to audio: a text encoder and an audio encoder (typically a transformer processing mel-spectrograms) are trained to align matching audio-text pairs. CLAP enables zero-shot audio classification, audio-text retrieval, and text-conditioned audio generation (as used in AudioLDM and Udio).

Production audio understanding applications include: content moderation (detecting gunshots, screaming, explicit language in user-generated content), industrial monitoring (detecting abnormal machine sounds indicating equipment failure), meeting analytics (identifying speakers, detecting sentiment, flagging action items), and accessibility (providing audio descriptions of visual content for visually impaired users). Deployment typically requires real-time streaming inference with latency under 100ms, favouring smaller, quantised models over the largest transformer architectures.

Cross-Modal Retrieval

Cross-modal retrieval — finding relevant items in one modality given a query in another — is one of the most commercially valuable applications of multimodal AI. Text-to-image search, image-to-product matching, and audio-to-video alignment are all cross-modal retrieval problems. The common infrastructure is a shared embedding space where similar content from different modalities is geometrically close.

Shared Embedding Spaces

The architecture for cross-modal retrieval typically follows one of two patterns. The dual encoder pattern (as in CLIP) trains separate encoders for each modality, connected by a contrastive objective. At retrieval time, queries and items are encoded independently and compared via cosine similarity — enabling fast approximate nearest-neighbour search (ANN) over pre-computed item embeddings. This scales to billions of items with millisecond retrieval latency using FAISS or ScaNN.

The fusion encoder pattern (as in FLAMingo, CoCa) applies cross-attention between modality representations, enabling richer modelling of cross-modal interactions at the cost of requiring joint processing at query time (no pre-computed embeddings). Fusion encoders achieve higher accuracy on complex VQA and cross-modal reasoning tasks but do not scale to large retrieval indices.

Production cross-modal retrieval systems at companies like Pinterest, Google Images, and Etsy use hybrid approaches: a dual encoder for efficient candidate retrieval (top-k candidates from an ANN index), followed by a fusion encoder re-ranker that scores the candidates with cross-modal attention for higher accuracy. This two-stage architecture achieves both scale and quality.

Multimodal RAG

Retrieval-Augmented Generation (RAG) extended to multimodal inputs is a pattern of growing importance for enterprise AI. Standard text RAG retrieves text chunks from a knowledge base and provides them as context to an LLM. Multimodal RAG adds the ability to retrieve and reason over images, charts, diagrams, and audio alongside text.

Two architectures are prevalent: late fusion multimodal RAG converts all modalities to text (OCR for images, transcription for audio) before retrieval, preserving compatibility with standard text embedding and LLM infrastructure at the cost of losing visual information. Native multimodal RAG indexes CLIP or CLAP embeddings of all modalities and retrieves across modalities; the retrieved images are then passed directly to a VLM. The latter is more powerful but requires VLM-capable deployment infrastructure.

Document Understanding

Document understanding — extracting structured information from PDFs, scanned forms, invoices, contracts, and scientific papers — is one of the highest-value applications of multimodal AI in the enterprise. Traditional approaches relied on OCR (optical character recognition) pipelines followed by rule-based extraction or NLP post-processing. Multimodal AI replaces this brittle pipeline with a single model that jointly understands layout, visual formatting, and text semantics.

From OCR to Visual Document QA

The evolution of document understanding spans three generations. Traditional OCR + NLP (Tesseract, ABBYY + spaCy/regex) is highly accurate for clean, typed documents but struggles with handwriting, complex layouts, tables, and mixed-modality documents. Maintenance burden is high as document formats evolve. Layout-aware language models (LayoutLM, LayoutLMv3) incorporate bounding box coordinates as additional input features, enabling the model to understand spatial relationships between text elements. LayoutLMv3 achieves state-of-the-art on the FUNSD form understanding benchmark.

Multimodal document foundation models (GPT-4V, Claude 3.5, Gemini 1.5 Pro, Donut) treat the document as an image and apply vision-language reasoning directly, without an OCR step. Donut (Document Understanding Transformer) was the first end-to-end model in this category, achieving competitive document information extraction without OCR by training a vision encoder-decoder on document image-text pairs. Its inference is simple: feed the document image, receive structured text output. GPT-4V and Claude have superseded Donut in quality but at much higher compute and API cost.

Structured Data Extraction

Practical document extraction pipelines benefit from explicit output structure prompting. Rather than asking the model to "describe the document", production systems prompt for specific fields in JSON format, enabling downstream processing without natural language parsing. Claude 3.5 Sonnet and GPT-4o are particularly strong at following complex extraction schemas and maintaining field type consistency (dates as ISO 8601, amounts as numeric floats, etc.) across varied document formats.

The most challenging document understanding tasks involve: scanned handwritten forms (variable legibility, inconsistent layout); multi-table financial statements (cross-referencing values across tables, detecting calculation errors); scientific figures (interpreting data from charts, extracting numerical values from graphs); and legal contracts (identifying obligations, conditions, defined terms, and cross-references across long documents). For these tasks, the combination of multimodal VLMs with retrieval-augmented document navigation consistently outperforms either approach alone.

Production Multimodal Systems

Deploying multimodal AI in production introduces specific engineering challenges beyond standard LLM deployment. The most significant are: (1) image tokenisation overhead — high-res images consume hundreds to thousands of tokens, dramatically increasing inference cost and latency; (2) multimodal context management — images in long conversation contexts must be cached or re-encoded efficiently; (3) content moderation — images require separate safety classifiers in addition to text safety; (4) evaluation — assessing generation quality across modalities is significantly harder than text-only evaluation.

Image token budgeting is the primary cost lever for production VLM deployment. GPT-4o's "low" detail mode (85 tokens/image) costs roughly 4× less than "high" detail mode (≥255 tokens) and is sufficient for most classification and high-level description tasks. "High" detail mode is necessary for reading fine print, analysing dense charts, and interpreting medical imaging. Production systems should route based on task type rather than always using maximum detail.

Hallucination in multimodal systems manifests in specific patterns: VLMs may confabulate text that appears to be in an image, misidentify objects in cluttered scenes, over-describe obvious visual elements while missing subtle ones, or provide confidently incorrect numerical readings from charts. Mitigation strategies: prompt for explicit uncertainty acknowledgement, request multiple independent readings for critical extractions, use model ensembling for high-stakes decisions, and evaluate with held-out multimodal benchmarks (MMMU, ScienceQA, DocVQA) before deployment.

For latency-sensitive applications, smaller open-source VLMs running locally often outperform API calls to large models. LLaVA-1.6-7B runs at 15–30 tokens/second on a single A100 with ~10GB VRAM, suitable for real-time visual QA. The quality-latency-cost trade-off for multimodal tasks roughly follows: GPT-4o (best quality, highest cost, 2–5s latency) > Claude 3.5 Sonnet (excellent quality, high cost) > Gemini 1.5 Flash (good quality, lower cost, fast) > LLaVA-1.6-7B (good quality, free local, variable latency).

Multimodal AI Comparison Tables

Multimodal Model Comparison

Cross-Modal Tasks Reference

Code: Vision-Language Analysis with GPT-4V

The following demonstrates a production-ready GPT-4o vision analysis function with base64 image encoding. This pattern is applicable to invoice extraction, chart analysis, medical image description, and any task requiring structured reasoning about image content.

from openai import OpenAI
import base64
from pathlib import Path

client = OpenAI()

def analyze_image(image_path: str, question: str) -> str:
    """Use GPT-4V to answer questions about an image."""
    # Encode image as base64
    image_data = base64.standard_b64encode(Path(image_path).read_bytes()).decode("utf-8")

    response = client.chat.completions.create(
        model="gpt-4o",  # gpt-4o natively handles vision
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": question},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}",
                        "detail": "high"  # high = 2048 token image budget; low = 85 tokens
                    }
                }
            ]
        }],
        max_tokens=500
    )
    return response.choices[0].message.content

# Production use cases:
print(analyze_image("invoice.jpg", "Extract all line items, quantities, and total cost as JSON"))
print(analyze_image("chart.png",  "What trend does this chart show? List 3 key observations."))
print(analyze_image("xray.jpg",   "Describe any abnormalities visible in this chest X-ray."))
# Note: Medical use requires validation against radiologist ground truth before deployment

Code: CLIP Zero-Shot Image Classification

CLIP enables zero-shot classification across any set of candidate labels — no image-specific training required. The function below demonstrates production-ready CLIP classification with cosine similarity scoring, applicable to product categorisation, content moderation, or any visual classification task.

import torch
import clip
from PIL import Image
import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-L/14", device=device)  # larger model, higher accuracy

def zero_shot_classify(image_path: str, candidate_labels: list[str]) -> dict:
    """Classify image without any image-specific training."""
    image = preprocess(Image.open(image_path)).unsqueeze(0).to(device)

    # Text embeddings for each candidate label
    text_inputs = clip.tokenize([f"a photo of {label}" for label in candidate_labels]).to(device)

    with torch.no_grad():
        image_features = model.encode_image(image)
        text_features = model.encode_text(text_inputs)

        # Normalized dot product = cosine similarity
        image_features /= image_features.norm(dim=-1, keepdim=True)
        text_features /= text_features.norm(dim=-1, keepdim=True)

        logits = (100.0 * image_features @ text_features.T).softmax(dim=-1)

    return dict(zip(candidate_labels, logits[0].tolist()))

# Zero-shot classification — no training required!
result = zero_shot_classify("product.jpg",
    ["sneakers", "dress shoes", "sandals", "boots", "loafers"])
best = max(result, key=result.get)
print(f"Classification: {best} ({result[best]:.1%} confidence)")
# Works for any label — no ImageNet pretraining bias

Code: Multimodal Document Understanding

The following demonstrates using Claude's vision API for structured extraction from multi-page document images. This pattern handles scanned PDFs by treating each page as an image and sending them as a batch to Claude with a structured extraction prompt.

from anthropic import Anthropic
import base64

client = Anthropic()

def extract_from_document(pdf_pages: list[bytes], extraction_prompt: str) -> dict:
    """Claude's vision to extract structured data from document pages."""
    content = []
    for i, page_bytes in enumerate(pdf_pages):
        encoded = base64.standard_b64encode(page_bytes).decode("utf-8")
        content.extend([
            {"type": "text", "text": f"Page {i+1}:"},
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": encoded}}
        ])
    content.append({"type": "text", "text": extraction_prompt})

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        messages=[{"role": "user", "content": content}]
    )
    return response.content[0].text

# Example: extract contract terms from a scanned PDF
contract_data = extract_from_document(
    pdf_pages=scanned_pages,
    extraction_prompt="Extract: parties, contract value, start date, end date, key obligations. Return as JSON."
)

Exercises

These exercises progress from applying existing multimodal APIs to building production-grade multimodal pipelines. Each exercise develops a skill directly applicable to real engineering work.

Evaluating Multimodal Systems & Failure Modes

Evaluating multimodal AI systems is significantly harder than evaluating unimodal text models. The output quality depends on multiple interacting factors: the accuracy of each modality's understanding, the fidelity of cross-modal reasoning, and the coherence of the final output. Standard NLP benchmarks are insufficient; multimodal systems require task-specific and human evaluation protocols.

Multimodal Benchmarks

MMMU (Massive Multidisciplinary Multimodal Understanding) is the most comprehensive general VLM benchmark: 11,500 questions across 30 university-level subjects (science, engineering, medicine, art, humanities), each requiring reasoning over college-level domain knowledge combined with images, diagrams, tables, and charts. GPT-4V scores ~55% on MMMU; Claude 3.5 Sonnet ~65%; human expert performance is ~88%.

DocVQA tests document visual question answering on scanned business documents. Evaluation metric: ANLS (Average Normalised Levenshtein Similarity) between predicted and ground truth answers. Leading models achieve ANLS >0.92 on single-page documents; performance drops significantly on multi-page and handwritten documents.

ScienceQA (Science Question Answering) evaluates multimodal reasoning on school science questions paired with images. It is particularly useful for testing educational AI applications. MMSTAR is a manually curated benchmark specifically designed to prevent models from answering correctly based on language priors without actually perceiving the image — a critical issue in earlier VLM benchmarks where many questions could be answered from text alone.

VCR (Visual Commonsense Reasoning) tests not just visual recognition but commonsense reasoning about visual scenes: given an image and a question, the model must provide an answer and a rationale. This tests the integration of perception and language reasoning, which is the core capability of multimodal LLMs.

Hallucination in Vision-Language Models

Multimodal hallucination takes several forms distinct from text-only hallucination:

• Object hallucination: The model mentions objects, text, or people that are not present in the image. Evaluated by CHAIR (Caption Hallucination Assessment with Image Relevance), which measures what fraction of mentioned objects do not appear in ground truth annotations. Early VLMs hallucinate 30–50% of described objects; leading models hallucinate <5%.
• Attribute error: The model correctly identifies an object but mis-describes its colour, count, position, or relationship to other objects. Particularly common for fine-grained visual attributes and spatial reasoning tasks.
• Text OCR hallucination: When reading text from images, models may confabulate plausible-sounding but incorrect text, especially for stylised fonts, partially occluded text, or handwriting. Critical for document processing applications.
• Cross-modal inconsistency: The model's text response contradicts information visible in the image — the most dangerous failure mode for production applications like medical imaging analysis.

Mitigation strategies: prompt for explicit uncertainty acknowledgement ("if you are not certain about any element, say so"); request structured output with a confidence field per extracted item; implement a secondary verification pass using a different VLM or a discriminative classifier; and evaluate on representative samples from your production distribution before deployment.

Latency and Cost Optimisation for VLMs

Image tokens are expensive. The cost and latency breakdown for a typical GPT-4o API call with a 1024×1024 image in "high" detail mode: the image is tiled into approximately 765 tokens (9 tiles × 85 tokens each), adding ~$0.0023 in image token cost per call at current pricing. Across 100,000 daily API calls, this is $230/day in image token costs alone. Optimisation strategies:

• Use low detail mode for classification tasks: 85 fixed tokens regardless of image size — sufficient for classification, sentiment, and coarse description. Only use high detail for tasks requiring fine text reading, detailed chart analysis, or medical imaging.
• Resize images before encoding: The API tiles images based on the 512px tile grid. A 512×512 image uses 1 tile (85 tokens in high detail). Downsizing to 512×512 before sending can reduce token usage by 4–9× for large images without quality loss for many tasks.
• Cache image embeddings: For repeated processing of the same images (e.g., product catalogue analysis), cache the base64-encoded image string and even the API response where appropriate. Context caching (supported by Anthropic's API) caches the full prompt including images, providing up to 90% cost reduction for repeated image calls.
• Route to smaller models for simple tasks: Use CLIP or a fine-tuned ResNet for classification/retrieval tasks; reserve VLM calls for tasks requiring language reasoning.

Multimodal RAG Architecture in Practice

A production multimodal RAG system for an enterprise knowledge base typically follows this architecture:

Privacy and Ethics in Multimodal AI

Multimodal AI introduces privacy risks beyond those of text-only systems. Images and audio contain biometric data (faces, voices, fingerprints) that is subject to strict data protection regulations under GDPR, CCPA, and biometric privacy laws in Illinois (BIPA), Texas, and Washington. Processing images of individuals through commercial VLM APIs may constitute biometric data processing requiring explicit consent and data processing agreements.

Key privacy engineering practices for multimodal systems: implement face detection and blurring before sending images containing individuals to third-party APIs; audit VLM API terms of service for data retention and training data usage policies (critical for medical and legal document processing); implement data residency controls for regulated industries; and evaluate whether on-premise VLM deployment (LLaVA-1.6, Idefics-2) is required to comply with data sovereignty requirements.

Multimodal AI System Card Generator

System cards document the inputs, outputs, capabilities, limitations, and ethical considerations for multimodal AI systems deployed in production. Given the complexity of multimodal systems — multiple input types, varied output formats, potential for cross-modal hallucination — thorough documentation is critical for responsible deployment.

Conclusion & Next Steps

Multimodal AI has crossed from research frontier to production infrastructure. CLIP's contrastive learning established the shared embedding space paradigm that underlies text-to-image search, zero-shot classification, and generative conditioning. GPT-4o, Claude 3.5, and Gemini 1.5 Pro have demonstrated that language models can reason over images, audio, and video with capabilities that meaningfully augment human workflows across document processing, scientific analysis, medical imaging, and customer service.

The convergence thesis — that different modalities share underlying representational structure — has profound implications: it suggests that each new modality added to a multimodal model benefits from and contributes to all other modalities' representations, creating positive transfer effects. This is the architectural reason why the largest and most capable models are all multimodal, and why purely unimodal architectures are increasingly rare in research frontiers.

The engineering challenges of multimodal production deployment are significant: token cost management, cross-modal hallucination mitigation, content safety across modalities, and evaluation methodology for multimodal outputs all require active attention. The legal landscape around privacy in visual/audio data and the ethical questions around non-consensual facial recognition and voice cloning are evolving rapidly and will shape deployment constraints for years to come.

Looking ahead, the next frontier is real-time multi-sensory AI: models that process live video, audio, and text simultaneously with sub-second latency, enabling truly interactive AI systems that can see, hear, and speak in real time. GPT-4o's Advanced Voice Mode is a preview of this direction; the convergence of ultra-fast inference hardware and multimodal architectures will bring this to broad deployment within the next two to three years.