Skip to content
Wetel Docs

Image Understanding for an Agent

An agent can look at an image a user sends — a screenshot, a photo, a scanned document — and answer questions that genuinely depend on what’s in it, not just its filename or a caption. This page is the end-to-end picture: how an image reaches your agent’s workflow, how you attach it to a model call, and the one thing you can’t skip (a vision-capable model).

This feature is new (shipped 2026-09-17). If you’re building against it right after reading this, confirm with your Wetel contact that it’s live on your environment — a change can merge to the platform’s source before it’s rolled out to every deployed environment.

The three pieces, and who configures each

Section titled “The three pieces, and who configures each”
PieceWhat it doesWho sets it up
The [[WETEL_ATTACHMENT ...]] markerGets an image’s URL into the workflow’s context, from any channelYour own backend, if you’re bridging a channel yourself — see below
A condition → action (fetchAsBase64Var) branchFetches the image’s bytes server-side once one is presentConfigured on the workflow graph
An llm node with imageVars, on a vision-capable providerActually sends the image to the model as an image, not just a filenameConfigured on the workflow graph, and the agent’s model provider

The first piece is something your own integration can act on directly with no help from Wetel. The second and third require a workflow/agent configuration change — today that means either you or Wetel’s team calling updateWorkflow/publishWorkflow/updateAgent (all JWT-authenticated GraphQL mutations, not gated behind an internal-only endpoint), but read Where you stand today before assuming you should just flip this on yourself.

Step 1: get the image’s URL into the conversation

Section titled “Step 1: get the image’s URL into the conversation”

If you already have a Channel Connector (Telegram today) forwarding messages into a Wetel agent, this is already done for you — skip to Step 2. The gateway prepends a marker line to the message text automatically whenever the incoming message carries a file, and it’s parsed server-side before your workflow ever runs.

If you’re bridging your own channel into sdkSendMessage directly (a headless integration), you construct that marker yourself:

[[WETEL_ATTACHMENT url="https://your-cdn.example.com/uploads/abc123.png" mime="image/png" filename="screenshot.png"]]

Prepend it — on its own line, at the very start of the message text — before whatever text the user actually typed:

[[WETEL_ATTACHMENT url="https://your-cdn.example.com/uploads/abc123.png" mime="image/png" filename="screenshot.png"]]
What's wrong with this error message?

Send that whole string as SdkSendMessageInput.text, same as any other turn.

If you forward any end-user-typed text yourself, neutralize [[ in it before prepending your own marker (text.replace(/\[\[/g, '[') is sufficient) — otherwise a user can type a [[...]]-shaped line of their own that gets parsed as if your platform had sent it. See Channels: attachment markers for the full spec, including the exact context variables this seeds (attachment, attachmentUrl, attachmentMime, attachmentFilename).

mime is optional if you can’t determine it — the platform will synthesize one from the filename or URL extension. url must be a real, fetchable http(s) link; it doesn’t need to be pre-authorized for Wetel specifically, since the fetch that follows is a plain unauthenticated GET.

Step 2: branch on whether an attachment is present

Section titled “Step 2: branch on whether an attachment is present”

Not every turn carries an image, so the graph needs to check first. Use the flat attachmentUrl mirror, not the nested attachment object — a condition expression that evaluates attachment.url throws when there’s no attachment at all, instead of just evaluating false:

{
  "id": "has_attachment",
  "type": "condition",
  "label": "Has Attachment?",
  "config": { "expression": "attachmentUrl != \"\"" },
  "position": { "x": 150, "y": 0 }
}

Step 3: fetch the image’s bytes

Section titled “Step 3: fetch the image’s bytes”

On the true branch, an action node fetches the URL server-side and base64-encodes it — this is the same fetchAsBase64Var mechanism already used to feed a partner API a file, just landing in a context variable an llm node will read instead of an outgoing request:

{
  "id": "fetch_image",
  "type": "action",
  "label": "Fetch Attachment Bytes",
  "config": {
    "fetchAsBase64Var": "attachmentUrl",
    "outputVar": "attachmentImageBase64"
  },
  "position": { "x": 300, "y": -150 }
}

Give this node a success/failure edge pair, both pointed at the same downstream llm node — a broken or oversized image URL should degrade the turn to a text-only reply, not fail the whole run:

{ "id": "e_fetch_success", "source": "fetch_image", "target": "reply", "label": "success" },
{ "id": "e_fetch_failure", "source": "fetch_image", "target": "reply", "label": "failure" }

See Action Node: Success/failure edges for how that routing works.

Step 4: attach it to the model call with imageVars

Section titled “Step 4: attach it to the model call with imageVars”

Both branches — attachment fetched, and no attachment at all — converge on the same llm node. Its imageVars field names the context variable holding the image; when that variable is empty (the no-attachment branch, or a fetch that failed), imageVars benignly no-ops and the node just answers from text:

{
  "id": "reply",
  "type": "llm",
  "label": "Generate Reply",
  "config": {
    "promptTemplate": "The user said: \"{{userMessage}}\"{{#if attachmentUrl}} and attached an image — look at it and reference what you actually see while answering, never guess.{{/if}} Reply helpfully and concisely.",
    "outputVar": "replyText",
    "imageVars": ["attachmentImageBase64"]
  },
  "position": { "x": 450, "y": 0 }
}

The one thing you cannot skip: a vision-capable model

Section titled “The one thing you cannot skip: a vision-capable model”

The platform’s default LLM provider does not support image input. Before calling the model, the runner checks whether the resolved provider can actually accept images — and if a turn carries one and the provider can’t, it fails the entire workflow run, not just this node. There’s no success/failure edge to fall back on for an llm node the way there is for action/tool nodes.

You get around this with the exact same llmProviderOverride field described in Agents: LLM provider override — set it on the agent (or ask Wetel to) to a provider whose capabilities.images is true. As of this writing that’s Gemini (GEMINI_DIRECT); ask your Wetel contact which provider(s) currently support images before building against a specific value, since this list can grow.

mutation EnableVision($id: ID!) {
  updateAgent(id: $id, input: { llmProviderOverride: GEMINI_DIRECT }) {
    id
    llmProviderOverride
  }
}

Where you stand today: what needs Wetel vs. what doesn’t

Section titled “Where you stand today: what needs Wetel vs. what doesn’t”
  • Getting an image’s URL into the conversation (Step 1) is entirely yours to control — no Wetel involvement needed, whether you’re using a packaged Channel Connector (automatic) or bridging your own channel (construct the marker yourself).
  • The workflow graph shape (Steps 2–4) is a normal updateWorkflow/publishWorkflow call — the same JWT-authenticated management-plane mutations used to create and edit any workflow. If your own backend already provisions/edits agents and workflows programmatically (rather than only calling the SDK surface — sdkStart/sdkSendMessage/sessionEvents), you already have everything needed to build this graph shape yourselves.
  • llmProviderOverride is documented as a pilot field — Wetel asks that you loop in your Wetel contact before turning it on for a production, customer-facing agent. This isn’t a hard technical gate (the field isn’t feature-flagged behind anything today), it’s a rollout/support courtesy: image-bearing turns cost more (they bill by real image token usage, not a flat rate) and Wetel wants visibility into which agents are using it while it’s new.
  • Practically: the fastest path today is to tell your Wetel contact which agent/workflow you want this enabled on. They can either configure it directly, or confirm you’re clear to call updateWorkflow/updateAgent yourselves for that specific agent.

Limits to design around

Section titled “Limits to design around”
  • Up to 4 images per llm node call, ~5MB each (raw, before base64 encoding). Extra entries in imageVars beyond 4 are dropped, not rejected.
  • JPEG, PNG, GIF, and WEBP are recognized — by the actual decoded bytes, not by file extension or declared MIME type. An unrecognized format fails the node (and, per the caution above, the whole run on an image-bearing turn).
  • Office documents, PDFs, and spreadsheets are not read as images by this mechanism — for those, see Document Extraction below.
Section titled “Related: document extraction”

Images aren’t the only attachment type worth extracting something from mid-conversation. A workflow can also pull plain text out of a PDF, DOCX, Markdown, CSV, or XLSX file the same way — via the same fetchAsBase64Var fetch, redirected to text extraction instead of base64 with the extractTextVar field. This works on any LLM provider, including the platform default — no vision model, no llmProviderOverride, needed. See Action Node: Extracting text instead of raw bytes.

If you want a document to be retrievable across future sessions too (not just read once, mid-conversation), that’s a separate mechanism — Knowledge Base (RAG) ingestion, which accepts the same five formats.

See also

Section titled “See also”