Sending images and documents to a model

What changes when the input is not text

Multimodal input means a request holds images or documents as well as words. The model reads them, but your service now carries large files, checks their type and pays by the image or the page. It matters because a request that was a few hundred bytes can become many megabytes.

A user photographs a receipt and uploads a ten-page contract. One request now has three parts and three ways to be refused.

By the end of this lesson

Note

I can say what changes when the input is an image or a document.

Key words

Word	What it means	Example
Part	one piece of a request: text, image or document
Media type	the label for a file's format	image/png
Payload size	the bytes a request carries
Page limit	the most pages a document may have
Unit price	the cost of one image or page

The cells students fill in are left blank in this preview.

The lab takes text only

Warning

The lab's model endpoint accepts text. A Run here cannot send an image or a document. This lesson teaches from a recorded request shape, invented for the lesson, and checks it with code.

A request becomes a list of parts

Text input is one string. Multimodal input is a list of parts, each with a type and its data. The data of a file travels as bytes, often encoded as text, which makes it about a third bigger.

Below is a recorded shape, invented for this lesson. A real provider has its own field names, so always read its documentation.

A recorded request shape

Code: node

{ "messages": [{ "role": "user", "content": [
    { "type": "text", "text": "What is on this receipt?" },
    { "type": "image", "media": "image/png", "data": "<base64, 412000 bytes>" },
    { "type": "document", "media": "application/pdf", "data": "<base64, 880000 bytes>" }
] }] }

Check every part before sending

Check each part against a rule you set: the media type, the byte size and the page count. Check the totals too: how many images, how many bytes in all.

Refuse early with a clear reason. A request that fails at the provider has already cost you the upload and maybe the money.

Worked example: limits and a price

Part	What it is	Check	Cost in units
1	text, 1800 characters	no limit here	2
2	image, png, 412000 bytes	allowed, under 5 MB	4
3	image, gif, 90000 bytes	gif is not on the list	4
4	document, 24 pages	over the 20 page limit	48

The cells students fill in are left blank in this preview.

Cost is per image and per page

Text is priced by the token. An image or a page is usually priced by a fixed amount or a number of tokens set by its size. Add the parts up before the call so a budget rule can refuse a costly one.

Where possible, send less: reduce the image size or send only the pages that matter.

A common mistake

Warning

Many people trust the file name or the extension. A name can say png on a file that is not one. Check the type from the bytes on the server, and also keep the limits on size. The extension is a hint, not a fact.