AI/ML6 min read

Multimodal AI

Build applications that understand images, documents, and audio using multimodal AI - vision models, document extraction, and combining modalities with language models.

Multimodal AIVisionImage UnderstandingClaude APIGPT-4V

skill: Multimodal AI category: AI/ML tags: [Multimodal AI, Vision, Image Understanding, Claude API, GPT-4V] readingTime: 6

Multimodal AI makes a class of problems trivially solvable that previously required expensive custom computer vision pipelines. Receipt scanning, document extraction, screenshot analysis, chart reading - all solved with a prompt. Start with the simplest approach (direct image + text prompt) before building complex pipelines.

Workflow

Send images to vision modelsModern LLMs accept images directly alongside text - no separate computer vision API needed for most use cases.
Extract structured data from imagesImages of receipts, invoices, forms, and tables can be converted to structured JSON - eliminating manual data entry.
Process documents and PDFsPDF documents can be converted to images page-by-page and processed with vision models - handling complex layouts, tables, and charts that pure text extraction misses.
Write effective prompts for visionVision models respond to specific, structured prompts differently than text-only models. Describing what you want in detail improves accuracy.
Handle multiple images and comparisonComparing two images (before/after, two designs, multiple screenshots) enables powerful use cases like visual diff, A/B test analysis, and change detection.

Output Format

Multimodal AI: image + text messages, base64 encoding, structured JSON extraction, and multi-image comparison.

messages: [
  { role: 'user', content: [
    { type: 'image', source: { type: 'base64', media_type: 'image/jpeg', data: b64 } },
    { type: 'text', text: 'Extract data as JSON: { field: type, ... }' }
  ]}
]

Parse: JSON.parse(response.content[0].text)

Examples

Input

Output

Edge Cases & Gotchas

Image size limits - APIs have per-image size limits (Claude: ~5MB per image). For large images, resize before sending: sharp(buffer).resize(2048, 2048, { fit: 'inside' }).jpeg({ quality: 85 }).toBuffer().
Privacy-sensitive images - Sending images to cloud APIs exposes them to the model provider. For HIPAA, PII, or confidential images, use on-premise vision models (LLaVA, Moondream, or Claude with private deployment).
Handwriting recognition accuracy - Messy handwriting has significantly lower accuracy than printed text. Pre-process with contrast enhancement and set expectations: include confidence scores or flag uncertain extractions for human review.
Rate limits for batch processing - Processing 1000 images quickly hits rate limits. Use exponential backoff, process in batches of 10-20, and check the rate limit headers in responses.

When to Break These Rules

Use Tesseract OCR or AWS Textract instead of a general vision LLM for high-volume, simple text extraction - much cheaper and faster for structured forms at scale.
Use specialized models (YOLO, SAM, CLIP) for real-time object detection, image segmentation, or embedding-based image search - general vision LLMs aren't optimized for these.

Reference Files

https://docs.anthropic.com/en/docs/build-with-claude/vision
https://platform.openai.com/docs/guides/vision