跳转到内容
Wetel Docs

Image Understanding for an Agent

此内容尚不支持你的语言。

An agent can look at an image a user sends — a screenshot, a photo, a scanned document — and answer questions that genuinely depend on what’s in it, not just its filename or a caption. This page is the end-to-end picture: how an image reaches your agent’s workflow, how you attach it to a model call, and the one thing you can’t skip (a vision-capable model).

This feature is new (shipped 2026-09-17). If you’re building against it right after reading this, confirm with your Wetel contact that it’s live on your environment — a change can merge to the platform’s source before it’s rolled out to every deployed environment.

The three pieces, and who configures each

Section titled “The three pieces, and who configures each”
PieceWhat it doesWho sets it up
The [[WETEL_ATTACHMENT ...]] markerGets an image’s URL into the workflow’s context, from any channelYour own backend, if you’re bridging a channel yourself — see below
A condition → action (fetchAsBase64Var) branchFetches the image’s bytes server-side once one is presentConfigured on the workflow graph
An llm node with imageVars, on a vision-capable providerActually sends the image to the model as an image, not just a filenameConfigured on the workflow graph, and the agent’s model provider

The first piece is something your own integration can act on directly with no help from Wetel. The second and third require a workflow/agent configuration change — today that means either you or Wetel’s team calling updateWorkflow/publishWorkflow/updateAgent (all JWT-authenticated GraphQL mutations, not gated behind an internal-only endpoint), but read Where you stand today before assuming you should just flip this on yourself.

Step 1: get the image’s URL into the conversation

Section titled “Step 1: get the image’s URL into the conversation”

If you already have a Channel Connector (Telegram today) forwarding messages into a Wetel agent, this is already done for you — skip to Step 2. The gateway prepends a marker line to the message text automatically whenever the incoming message carries a file, and it’s parsed server-side before your workflow ever runs.

If you’re bridging your own channel into sdkSendMessage directly (a headless integration), you construct that marker yourself:

[[WETEL_ATTACHMENT url="https://your-cdn.example.com/uploads/abc123.png" mime="image/png" filename="screenshot.png"]]

Prepend it — on its own line, at the very start of the message text — before whatever text the user actually typed:

[[WETEL_ATTACHMENT url="https://your-cdn.example.com/uploads/abc123.png" mime="image/png" filename="screenshot.png"]]
What's wrong with this error message?

Send that whole string as SdkSendMessageInput.text, same as any other turn.

If you forward any end-user-typed text yourself, neutralize [[ in it before prepending your own marker (text.replace(/\[\[/g, '[') is sufficient) — otherwise a user can type a [[...]]-shaped line of their own that gets parsed as if your platform had sent it. See Channels: attachment markers for the full spec, including the exact context variables this seeds (attachment, attachmentUrl, attachmentMime, attachmentFilename).

mime is optional if you can’t determine it — the platform will synthesize one from the filename or URL extension. url must be a real, fetchable http(s) link; it doesn’t need to be pre-authorized for Wetel specifically, since the fetch that follows is a plain unauthenticated GET.

Step 2: branch on whether an attachment is present

Section titled “Step 2: branch on whether an attachment is present”

Not every turn carries an image, so the graph needs to check first. Use the flat attachmentUrl mirror, not the nested attachment object — a condition expression that evaluates attachment.url throws when there’s no attachment at all, instead of just evaluating false:

{
  "id": "has_attachment",
  "type": "condition",
  "label": "Has Attachment?",
  "config": { "expression": "attachmentUrl != \"\"" },
  "position": { "x": 150, "y": 0 }
}

Step 3: fetch the image’s bytes

Section titled “Step 3: fetch the image’s bytes”

On the true branch, an action node fetches the URL server-side and base64-encodes it — this is the same fetchAsBase64Var mechanism already used to feed a partner API a file, just landing in a context variable an llm node will read instead of an outgoing request:

{
  "id": "fetch_image",
  "type": "action",
  "label": "Fetch Attachment Bytes",
  "config": {
    "fetchAsBase64Var": "attachmentUrl",
    "outputVar": "attachmentImageBase64"
  },
  "position": { "x": 300, "y": -150 }
}

Give this node a success/failure edge pair, both pointed at the same downstream llm node — a broken or oversized image URL should degrade the turn to a text-only reply, not fail the whole run:

{ "id": "e_fetch_success", "source": "fetch_image", "target": "reply", "label": "success" },
{ "id": "e_fetch_failure", "source": "fetch_image", "target": "reply", "label": "failure" }

See Action Node: Success/failure edges for how that routing works.

Step 4: attach it to the model call with imageVars

Section titled “Step 4: attach it to the model call with imageVars”

Both branches — attachment fetched, and no attachment at all — converge on the same llm node. Its imageVars field names the context variable holding the image; when that variable is empty (the no-attachment branch, or a fetch that failed), imageVars benignly no-ops and the node just answers from text:

{
  "id": "reply",
  "type": "llm",
  "label": "Generate Reply",
  "config": {
    "promptTemplate": "The user said: \"{{userMessage}}\"{{#if attachmentUrl}} and attached an image — look at it and reference what you actually see while answering, never guess.{{/if}} Reply helpfully and concisely.",
    "outputVar": "replyText",
    "imageVars": ["attachmentImageBase64"]
  },
  "position": { "x": 450, "y": 0 }
}

Shorter alternative: let the llm node fetch the URL

Section titled “Shorter alternative: let the llm node fetch the URL”

Added 2026-10-06. Steps 3 and 4 can be one node: set imageUrlsVar to the variable that holds the URL, and the llm node fetches the image itself. No condition or action node is needed, because an empty variable just means no image this turn:

{
  "id": "reply",
  "type": "llm",
  "label": "Generate Reply",
  "config": {
    "promptTemplate": "The user said: \"{{userMessage}}\"{{#if attachmentUrl}} and attached an image — look at it and reference what you actually see while answering, never guess.{{/if}} Reply helpfully and concisely.",
    "outputVar": "replyText",
    "imageUrlsVar": "attachmentUrl"
  }
}

The URL path also takes several URLs at once, and with imageMarkerHistory it can collect images named by [image:<url>] markers in the user’s messages. The image bytes go straight to the model and are never stored in the workflow context or run record. Differences from Step 3 to keep in mind: only https:// and s3:// URLs are fetched, and an image that cannot be loaded is skipped (with a note to the model) rather than routed down a failure edge. attachmentUrl is set for any attached file, so a PDF or document sent on this path is skipped as not an image, and the model gets the “could not be loaded” note. Signed URLs in [image:<url>] markers are stored as sent but shown as [image:https://host/path?[redacted]] when the text is read back through the API. See LLM Node: Fetching images by URL for the full rules.

Which to use: the URL options when you have one or more image URLs and just want the model to see them; imageVars when the image is already in a context variable as base64 or a data URL, or when you need the fetched bytes for something else in the graph too. Existing imageVars graphs keep working unchanged.

The one thing you cannot skip: a vision-capable model

Section titled “The one thing you cannot skip: a vision-capable model”

Since 2026-10-01 the default provider (Claude Haiku) reads images, so this section only matters for agents pinned to a provider that cannot read images. Before calling the model, the runner checks whether the resolved provider can actually accept images — and if a turn carries one and the provider can’t, it fails the entire workflow run, not just this node, unless that llm node has success/failure edges (available on llm nodes since 2026-10-15; see LLM Node: Success/failure edges), in which case the run continues down failure.

You get around this with the exact same llmProviderOverride field described in Agents: LLM provider override — set it on the agent (or ask Wetel to) to a provider whose capabilities.images is true.

Vision-capable providers:

  • WEBBYXAI_HAIKU (the default; Claude Haiku 4.5, plus Sonnet 4.6 and Opus 4.6 where enabled) — added 2026-10-01. Up to 100 images per request, each under 5 MB as base64. On a quick digit-reading check Haiku scored 9 of 10 and Gemini 10 of 10, so use a Gemini model on the node when exactness matters.
  • GEMINI_DIRECT — Gemini’s native vision support. Recommended for general-purpose image understanding.
  • BEDROCK_MANTLE (with vision-capable modelId) — Six models currently support image input on Bedrock Mantle. See LLM API: Vision-capable models for the current model list, accuracy metrics, and latency expectations.
mutation EnableVision($id: ID!) {
  updateAgent(id: $id, input: { llmProviderOverride: GEMINI_DIRECT }) {
    id
    llmProviderOverride
  }
}

If you use BEDROCK_MANTLE, you must also specify a vision-capable model via llmModelOverride on the same agent:

mutation EnableVisionWithModelOverride($id: ID!) {
  updateAgent(
    id: $id
    input: {
      llmProviderOverride: BEDROCK_MANTLE
      llmModelOverride: "mistral.mistral-large-3-675b-instruct"
    }
  ) {
    id
    llmProviderOverride
    llmModelOverride
  }
}

Where you stand today: what needs Wetel vs. what doesn’t

Section titled “Where you stand today: what needs Wetel vs. what doesn’t”
  • Getting an image’s URL into the conversation (Step 1) is entirely yours to control — no Wetel involvement needed, whether you’re using a packaged Channel Connector (automatic) or bridging your own channel (construct the marker yourself).
  • The workflow graph shape (Steps 2–4) is a normal updateWorkflow/publishWorkflow call — the same JWT-authenticated management-plane mutations used to create and edit any workflow. If your own backend already provisions/edits agents and workflows programmatically (rather than only calling the SDK surface — sdkStart/sdkSendMessage/sessionEvents), you already have everything needed to build this graph shape yourselves.
  • llmProviderOverride is documented as a pilot field — Wetel asks that you loop in your Wetel contact before turning it on for a production, customer-facing agent. This isn’t a hard technical gate (the field isn’t feature-flagged behind anything today), it’s a rollout/support courtesy: image-bearing turns cost more (they bill by real image token usage, not a flat rate) and Wetel wants visibility into which agents are using it while it’s new.
  • Practically: the fastest path today is to tell your Wetel contact which agent/workflow you want this enabled on. They can either configure it directly, or confirm you’re clear to call updateWorkflow/updateAgent yourselves for that specific agent.

Limits to design around

Section titled “Limits to design around”
  • imageVars: up to 20 images per llm node call, each under about 5 MB, 18 MB combined. Extra entries are dropped, not rejected. imageVars images are never resized: an oversized image fails the node.
  • URL images (imageUrlsVar, imageMarkerHistory): up to 40 images per node by default. Originals up to 20 MB are fetched, decoded once, rotated upright (EXIF) and shrunk to a long edge of about 1568 px as JPEG (PNG kept when small or transparent), typically 200 to 600 KB each; an image already within the limits is passed through untouched. After shrinking, each image must fit 5 MB (about 3.9 MB on the Claude route) and the call 18 MB in total (24 MB when the node pins a Gemini model with modelOverride). 40 phone photos normally total 10 to 15 MB. Originals over 20 MB are skipped (too-large) and so are images over 36 megapixels (too-many-pixels). A resized copy is still welcome because it is faster, but it is no longer required.
  • Time for URL images: the whole fetch-and-shrink phase has a hard 90-second deadline, and each server shrinks three images at a time. Measured with 40 large photos: alone, 40 of 40 attached in about 54 s end to end; 2 overlapping runs attached 40 and 40 in about 113 s each; 4 overlapping runs attached 38 to 40 in 109 to 129 s each (the losses are deadline skips). Keep imageMarkerHistory as small as the question needs: every user message fetches the last N images again.
  • JPEG, PNG, GIF, and WEBP are recognized — by the actual decoded bytes, not by file extension or declared MIME type. With imageVars, an unrecognized format fails the node (and, per the caution above, the whole run on an image-bearing turn); with the URL options it is skipped unless imageUrlsFailPolicy is "fail".
  • URL images: only https:// and s3:// (granted buckets only), no redirects, no private addresses, and a 90-second deadline for the whole fetch step.
  • These are platform defaults and may be tuned; confirm with your Wetel contact if a number matters to your design.
  • Office documents, PDFs, and spreadsheets are not read as images by this mechanism — for those, see Document Extraction below.
Section titled “Related: document extraction”

Images aren’t the only attachment type worth extracting something from mid-conversation. A workflow can also pull plain text out of a PDF, DOCX, Markdown, CSV, or XLSX file the same way — via the same fetchAsBase64Var fetch, redirected to text extraction instead of base64 with the extractTextVar field. This works on any LLM provider, including the platform default — no vision model, no llmProviderOverride, needed. See Action Node: Extracting text instead of raw bytes.

If you want a document to be retrievable across future sessions too (not just read once, mid-conversation), that’s a separate mechanism — Knowledge Base (RAG) ingestion, which accepts the same five formats.

See also

Section titled “See also”