Recipe: Build a Reliable Workflow

This content is not available in your language yet.

A workflow that calls other systems will meet slow, busy and broken services. This recipe puts together the five tools Wetel gives you for that, in the order a call meets them: a retry policy for short, safe-to-repeat failures, failure edges for the failures that remain, safe error text for the user, workflow settings for anything you did not route, and mock-mode tests plus versions so you can prove the branches before release and undo a bad release in one call.

Availability: available on production from 2026-10-24 (the retry policy, mock mode and the mock-mode default for regression replays). Failure edges, the safe error variables, set_variable, switch, workflow versions and the workflow settings were already in production before that date.

Prerequisites

An agent with a workflow you can edit, and a dashboard sign-in (Authorization: Bearer <token> plus x-huat-platform: customer). The version operations and updateWorkflow do not accept an API key.
A Custom Action for the service you call. This recipe uses id 9, an order-status lookup that answers JSON such as { "status": "shipped", "eta": "24 October" }.
The Fail gracefully recipe is a shorter introduction to failure edges alone; this one adds retries, testing and release.

The layers, in order

flowchart TD
    C[node calls a service] --> R{failed?}
    R -->|no| S[success edge]
    R -->|yes| P{retry policy allows another attempt?}
    P -->|yes, after a short wait| C
    P -->|no| F{node has a failure edge?}
    F -->|yes| H[failure branch: reply with _lastNodeErrorPublic]
    F -->|no| W[run fails: failureResponse text, optional owner notice]

Layer	Handles	Where it is set
`retry` on the node	Short, transient failures where resending is safe	`config.retry` on a `tool`, `action` or `webhook` node
`success` / `failure` edges	Every failure that is left after the retries	Edge labels on a failure-routable node
`_lastNodeErrorPublic`	What the user is told	A `response` node on the failure branch
`failureResponse` and `notifyOnFailure`	Failures on nodes without a `failure` edge	Workflow settings, through `updateWorkflow`
Mock mode and versions	Proving each branch without side effects, undoing a bad release	`sendMessage` with `testDraft: true`; `publishWorkflow`, `rollbackWorkflow`

Which node types can do what

Node type	`failure` edge	`retry` policy	Mocked in a mock-mode test
`tool`	Yes	Yes	Yes
`action`	Yes	Yes (not with `fetchAsBase64Var`)	Yes
`webhook`	Yes	Yes (not with `async: true`)	Yes
`llm`	Yes	No: the model layer has its own	Only when asked (`draftMockLlm`)
`sub_agent`	Yes	No	When given a mock
`set_variable`	Yes	No	No (no outside effects)
`pdf_export`	Yes	No	Yes

Every other node type ignores success and failure labels as failure routes (the FAILURE_EDGE_ON_UNROUTABLE_NODE diagnostic flags them), and saving a retry field on any other type is refused. The full rules are in Failure routing and Retrying a failed call.

Example 1: a lookup that retries, branches and fails safely

The order number arrives as per-turn data (turnContext: { "orderId": "A-1042" } on sdkSendMessage). The workflow copies it into a variable, looks the order up with a retry policy, reads the status with a second set_variable, and branches on it with a switch. If the lookup still fails after its retries, the failure edge sends a safe sentence.

flowchart TD
    S[start] --> V1[set_variable: orderId]
    V1 --> A[action: look up order, retry up to 3 attempts]
    A -->|success| V2[set_variable: orderStatus]
    A -->|failure| E[response: _lastNodeErrorPublic + try again]
    V2 --> W{switch}
    W -->|shipped| R1[response: shipped, eta]
    W -->|delayed| R2[response: delayed]
    W -->|default| R3[response: being processed]

mutation UpdateWorkflow($input: UpdateWorkflowInput!) {
  updateWorkflow(input: $input) {
    id
    version
  }
}

{
  "input": {
    "id": 74,
    "nodes": [
      {
        "id": "start",
        "type": "start",
        "label": "Start",
        "config": {},
        "position": {
          "x": 0,
          "y": 0
        }
      },
      {
        "id": "read_order_id",
        "type": "set_variable",
        "label": "Read the order number",
        "config": {
          "assignments": [
            {
              "variable": "orderId",
              "kind": "template",
              "source": "{{turn.context.orderId}}"
            }
          ]
        },
        "position": {
          "x": 0,
          "y": 100
        }
      },
      {
        "id": "lookup_order",
        "type": "action",
        "label": "Look up order",
        "config": {
          "customActionId": 9,
          "argsTemplate": "{\"orderId\": \"{{orderId}}\"}",
          "outputVar": "orderResult",
          "timeoutMs": 8000,
          "retry": {
            "maxAttempts": 3,
            "on": ["UNREACHABLE", "RATE_LIMITED", "TIMEOUT", "UPSTREAM_ERROR"],
            "idempotent": true,
            "deadlineMs": 20000
          },
          "mockResponse": {
            "status": "shipped",
            "eta": "24 October"
          }
        },
        "position": {
          "x": 0,
          "y": 200
        }
      },
      {
        "id": "read_status",
        "type": "set_variable",
        "label": "Read the status",
        "config": {
          "assignments": [
            {
              "variable": "orderStatus",
              "kind": "template",
              "source": "{{orderResult.status}}"
            }
          ]
        },
        "position": {
          "x": -200,
          "y": 300
        }
      },
      {
        "id": "by_status",
        "type": "switch",
        "label": "Which status?",
        "config": {
          "cases": [
            {
              "label": "shipped",
              "expression": "orderStatus == \"shipped\""
            },
            {
              "label": "delayed",
              "expression": "orderStatus == \"delayed\""
            }
          ]
        },
        "position": {
          "x": -200,
          "y": 400
        }
      },
      {
        "id": "reply_shipped",
        "type": "response",
        "label": "Shipped",
        "config": {
          "messageTemplate": "Order {{orderId}} has shipped and should arrive by {{orderResult.eta}}."
        },
        "position": {
          "x": -400,
          "y": 500
        }
      },
      {
        "id": "reply_delayed",
        "type": "response",
        "label": "Delayed",
        "config": {
          "messageTemplate": "Order {{orderId}} is delayed. We will message you as soon as it ships."
        },
        "position": {
          "x": -200,
          "y": 500
        }
      },
      {
        "id": "reply_other",
        "type": "response",
        "label": "Any other status",
        "config": {
          "messageTemplate": "Order {{orderId}} is being processed."
        },
        "position": {
          "x": 0,
          "y": 500
        }
      },
      {
        "id": "reply_failed",
        "type": "response",
        "label": "Lookup failed",
        "config": {
          "messageTemplate": "{{_lastNodeErrorPublic}} Please try again in a few minutes."
        },
        "position": {
          "x": 250,
          "y": 300
        }
      }
    ],
    "edges": [
      {
        "id": "e1",
        "source": "start",
        "target": "read_order_id"
      },
      {
        "id": "e2",
        "source": "read_order_id",
        "target": "lookup_order"
      },
      {
        "id": "e3",
        "source": "lookup_order",
        "target": "read_status",
        "label": "success"
      },
      {
        "id": "e4",
        "source": "lookup_order",
        "target": "reply_failed",
        "label": "failure"
      },
      {
        "id": "e5",
        "source": "read_status",
        "target": "by_status"
      },
      {
        "id": "e6",
        "source": "by_status",
        "target": "reply_shipped",
        "label": "shipped"
      },
      {
        "id": "e7",
        "source": "by_status",
        "target": "reply_delayed",
        "label": "delayed"
      },
      {
        "id": "e8",
        "source": "by_status",
        "target": "reply_other",
        "label": "default"
      }
    ]
  }
}

Why it is built this way:

The retry is safe because the lookup is. A lookup can be sent twice without harm, so idempotent: true lets the engine resend after a TIMEOUT or a 5xx (UPSTREAM_ERROR) as well as after a connection failure or a 429. Three attempts, waiting about 0.5 s and then 1 s (the default exponential backoff, spread by up to 20 % either way), all inside deadlineMs: 20000. Each attempt still has its own timeoutMs of 8 s.
set_variable instead of an llm node. Copying a value costs no model call. A template assignment renders text, so a missing status field gives an empty string and the switch takes default, rather than failing the run.
The switch has a default edge, which publishing requires, and case labels that are not reserved edge labels (success, failure, default and the other outcome labels cannot be case names).
Only _lastNodeErrorPublic reaches the user. _lastNodeError can contain host names, ids, status lines and the other service’s own message; keep it for traces or an internal note, never a reply.
mockResponse is what the lookup returns in a mock-mode test (Step 3 below). A live run ignores it.

workflowDiagnostics on this graph reports only MOCK_RESPONSE_SET, an INFO reminder that the mock exists.

Example 2: a create call that must not be sent twice

Creating a ticket is different: if the first request timed out, the helpdesk may already have created the ticket, and sending it again makes a duplicate. Leave the policy at its default classes, which only resend when the request cannot have been acted on (the connection never opened) or the service said 429:

{
  "id": "create_ticket",
  "type": "webhook",
  "label": "Create ticket",
  "config": {
    "url": "https://helpdesk.example.com/api/tickets",
    "method": "POST",
    "headers": {
      "content-type": "application/json"
    },
    "bodyTemplate": "{\"conversationRef\": \"session-{{sessionId}}\", \"summary\": \"{{userMessage}}\"}",
    "outputVar": "ticket",
    "timeoutMs": 10000,
    "retry": {
      "maxAttempts": 3
    }
  },
  "position": {
    "x": 0,
    "y": 100
  }
}

Add TIMEOUT to retry.on without marking the request idempotent and workflowDiagnostics answers with a WARNING:

{
  "code": "RETRY_NOT_IDEMPOTENT",
  "severity": "WARNING",
  "nodeId": "create_ticket"
}

The engine would not resend those failures anyway, so the warning tells you the policy does less than it says. Two honest fixes: drop TIMEOUT and UPSTREAM_ERROR from on, or make the call idempotent on the service’s side (for example an upsert keyed by conversationRef) and set retry.idempotent: true. A GET webhook counts as idempotent without the flag. webhook header values are static, not templated, so an Idempotency-Key header in headers would carry the same value on every request; put a per-request key in the body instead.

Give this node the same success / failure pair as Example 1 so a failure after the last attempt still gets a safe reply.

Step 1: Set the workflow-level safety net

Failures on nodes without a failure edge still end the run. Two settings decide what happens then. Both take effect as soon as updateWorkflow saves them, need no publish, and are not part of any version, so a rollback leaves them as they are.

{
  "input": {
    "id": 74,
    "settings": {
      "failureResponse": "Sorry, I could not finish that just now. A colleague will follow up here.",
      "notifyOnFailure": "OWNERS"
    }
  }
}

failureResponse: plain text, 1 to 500 characters, sent instead of the standard error reply. No {{ }} templates, no HTML, nothing shaped like a credential or signed URL (refused with BAD_USER_INPUT). null clears it; omitting it keeps the stored value.
notifyOnFailure: OWNERS sends the workspace owners an in-app notice, at most one per workflow per 60 minutes; NONE (the default) sends nothing. The notice names the workflow, the run and the failing node type only, never the conversation or the error text. Draft tests, regression replays, interrupted turns and failures handled by a failure edge never notify.

Step 2: Check the graph

Run workflowDiagnostics on the draft before every publish. The codes that matter for reliability:

Code	Severity	Means
`SUCCESS_WITHOUT_FAILURE`	Warning	A `success` edge with no `failure` edge: a failure still fails the run.
`FAILURE_EDGE_ON_UNROUTABLE_NODE`	Warning	A `failure` label on a node type that does not route failures.
`RETRY_NOT_IDEMPOTENT`	Warning	`TIMEOUT` or `UPSTREAM_ERROR` in `retry.on` on a request not known idempotent.
`MOCK_RESPONSE_SET`	Info	A node has a `mockResponse`; nothing to fix.

The full list is on Diagnostics.

Step 3: Test every branch in mock mode

A draft test (sendMessage with testDraft: true) runs in mock mode by default (draftSideEffects: MOCK): the action and webhook nodes build their request exactly as live and then return a mock instead of calling out. Nothing is sent, no credentials are read and nothing is metered. One session, one turn per branch:

mutation TestTurn($input: SendMessageInput!) {
  sendMessage(input: $input) {
    id
  }
}

Branch to prove	`draftMocks` for the turn	Expected reply
Shipped	none (the node’s `mockResponse` is used)	`Order A-1042 has shipped and should arrive by 24 October.`
Delayed	`{ "lookup_order": { "output": { "status": "delayed" } } }`	`Order A-1042 is delayed. ...`
Unknown status	`{ "lookup_order": { "output": { "status": "on_hold" } } }`	`Order A-1042 is being processed.`
Busy service	`{ "lookup_order": { "fail": "Request failed with status code 429" } }`	`The service is busy right now. Please try again in a few minutes.`
Timeout	`{ "lookup_order": { "fail": "timeout of 8000ms exceeded" } }`	`The service took too long to respond. Please try again in a few minutes.`

For example, the busy-service turn:

{
  "input": {
    "sessionId": 123,
    "text": "Where is my order?",
    "testDraft": true,
    "turnContext": { "orderId": "A-1042" },
    "draftMocks": {
      "lookup_order": { "fail": "Request failed with status code 429" }
    }
  }
}

What to know about mocked failures:

The code comes from your message, the same way as from a real error: status code 429 gives RATE_LIMITED, status code 503 gives UPSTREAM_ERROR, timeout gives TIMEOUT. A message the rules do not recognise gives UNKNOWN.
A mocked failure is never retried. The trace row shows retryOutcome: "mock", so a mock test proves the routing, not the retry timing.
Read the result with workflowRuns(sessionId): the run has mocked: true, the lookup’s trace row has "mocked": true, "status": "FAILED_HANDLED" on the failure turns, and the renderedRequest it would have sent.

When the mocks agree with the real service, the workspace owner can send one turn with "draftSideEffects": "LIVE" against a test record (anyone else gets FORBIDDEN). The full rules, including draftMockLlm and the limit of 20 real model calls per test turn, are in Testing a Workflow Safely.

Regression replays default to mock mode too. runRegressionScenario replays a saved conversation against the draft with each outside call returning what it returned in the recorded run, and a recorded handled failure failing the same way. Pass sideEffects: LIVE (owner only) to call the real systems, for example to notice a partner API change. Each result records sideEffectMode.

Step 4: Publish with a note, roll back in one call

mutation Publish($id: Int!, $note: String) {
  publishWorkflow(id: $id, note: $note) {
    id
    currentVersionId
  }
}

{ "id": 74, "note": "Order lookup: retry policy and status switch" }

Every publish that changes the published graph is kept as a numbered version. Before publishing, workflowVersionDiff with fromVersion set to the current version and no toVersion shows exactly what the publish will change (key names only, never values).

If the release misbehaves, publish the previous version again:

mutation Rollback($workflowId: Int!, $versionNumber: Int!, $note: String) {
  rollbackWorkflow(
    workflowId: $workflowId
    versionNumber: $versionNumber
    note: $note
  ) {
    id
    currentVersionId
  }
}

{
  "workflowId": 74,
  "versionNumber": 4,
  "note": "Back to v4 while we check the lookup"
}

A rollback is a new version (source: ROLLBACK); history is never rewritten and the draft is not changed. To edit from an old version, use restoreWorkflowVersionToDraft.
The old graph is checked against today’s rules, so a rollback can be refused with the same errors as a publish.
It takes effect from the next turn. A conversation paused at an await_reply or approval node finishes on the version it paused on.
Settings (failureResponse, notifyOnFailure) are not versioned and do not roll back.

Test checklist

workflowDiagnostics shows no WARNING you have not decided to accept.
Every tool, action and webhook node has a success and a failure edge, or you have decided its failure should end the run.
Every retry policy on a request that changes data either leaves out TIMEOUT and UPSTREAM_ERROR or is truly idempotent.
No reply template contains {{_lastNodeError}}.
Each branch has a mock-mode test turn with the expected reply, and the trace rows are FAILED_HANDLED where expected.
failureResponse is set, and notifyOnFailure is OWNERS if someone should hear about unhandled failures.
You know the version number to roll back to before you publish.

What this does not do

Retries are short. At most 4 attempts, waits of at most 5 s each, about 10 s of waiting per node in total. A Retry-After longer than maxDelayMs makes the node give up instead of waiting. Anything longer belongs in a later turn, not a retry.
llm nodes have no node-level retry. The model layer already retries and falls back between providers; give an llm node a failure edge for what is left.
Mock mode does not test the other system. It proves your graph’s routing and the request it would send; one live test against a test record proves the rest.
A rollback does not undo side effects. Tickets created or messages sent by the bad version stay created and sent.