Avatar Service REST API

Unlike the rest of the platform, avatar/voice operations are exposed over a separate REST service, not the GraphQL API. This service handles text-to-speech (TTS) synthesis, speech-to-text (STT) transcription, and serving the catalog of available 3D avatar models. Its address is returned by the GraphQL API (avatarServiceUrl in the speechToken and sdkStart responses); the examples below write it as $AVATAR_SERVICE_URL.

Hostnames

The GraphQL examples use the sandbox, https://api.wetel.dev/graphql; your production base URL is issued when you go live. A key or token works in only one environment. Set AVATAR_SERVICE_URL to the avatarServiceUrl the GraphQL API returns for your environment rather than hard-coding a hostname.

This page assumes you’re already familiar with Avatar Config and Sessions from the main GraphQL API. See also <vai-avatar> and Voice Interface for the higher-level client components that call these endpoints for you — most integrations should reach for those rather than calling this REST API directly.

Authentication

Every endpoint below (except the public model catalog and health check) requires an avatar token — a short-lived, HMAC-signed token that is distinct from your regular JWT or API key. Send it as a standard bearer token:

Authorization: Bearer <AVATAR_TOKEN>

You obtain an avatar token from the main GraphQL API, not from this service directly:

• the speechToken mutation, for /stt and /tts* only, with no session needed (JWT or API key), or
• automatically, as part of sdkStart’s response, when using the embed SDK entry point.

Tokens are minted server-side

Avatar tokens are minted server-side by the main API — there is no public endpoint on this service for requesting one directly. (Internally, the main API mints tokens via a server-to-server call authenticated with a separate internal key; that mechanism is not part of the public surface and should never be called from client code.)

Rate limiting: two independent ceilings per route

Every authenticated route below (/tts, /tts/google, /tts/gemini, /stt, /stt/google, /stt/gemini, /avatar-session) is governed by two separate, named throttles that must both allow the request:

• A per-client-address ceiling (the number quoted in each route’s own “Rate limit” line below), meant to stop one device or one browser from flooding a route.
• A per-tenant ceiling, shared across every caller currently authenticated with that tenant’s tokens (regardless of which address each caller connects from). This is the one that normally binds for a partner that sends all its users’ audio from one server.

A request without a valid, current token (missing header, malformed token, expired token, bad signature) is governed by the per-address ceiling only; the per-tenant throttle is skipped for it. Exceeding either ceiling returns the same 429, and there is no way from the response alone to tell which of the two was hit.

Changed 2026-10-05: the speech routes’ per-address ceilings are higher, and now really per address

Until 2026-10-05 the per-address ceilings on the speech routes were 10 requests a minute for speech-to-text and 30 for text-to-speech, and in practice every caller on the platform shared them: the service saw only its reverse proxy’s address, so the limit was one bucket for everyone. A burst from one integration could therefore return 429 to unrelated customers. That is fixed: the service now reads the caller’s address from the proxy, and the defaults are 120 a minute for speech-to-text and 300 for text-to-speech per address, with the per-tenant ceilings unchanged (200 and 600). /avatar-session is unchanged.

POST /tts

Synthesizes speech from text or SSML.

Auth required: Avatar token (Bearer).

Rate limit: 300 requests/minute per client address, and 600 requests/minute per tenant (see Rate limiting: two independent ceilings per route above).

Request body:

Field	Type	Required	Notes
text	string	One of text/ssml	Plain text to synthesize
ssml	string	One of text/ssml	SSML markup; takes precedence over text if both are set
voiceName	string	Yes
languageCode	string	Yes	e.g. en-US
speakingRate	number	No	0.25–4.0
audioEncoding	string	No	MP3 or LINEAR16

Response: 200 OK

Field	Type	Notes
audioContent	string	Base64-encoded audio
timepoints	array	[{ markName: string, timeSeconds: number }] — SSML mark timing, for lip-sync

curl -X POST $AVATAR_SERVICE_URL/tts \
  -H "Authorization: Bearer <AVATAR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Hello, how can I help you today?",
    "voiceName": "en-US-Neural2-F",
    "languageCode": "en-US"
  }'

{
  "audioContent": "//uQxAAAAAAAAAAAAAAAAAAAAAAAAAAA...",
  "timepoints": []
}

POST /tts/google

A request/response-shape-compatible variant of /tts, for third-party avatar rendering libraries (such as TalkingHead) that expect the Google Cloud Text-to-Speech request format instead of this service’s native shape.

Auth required: Avatar token (Bearer).

Rate limit: 300 requests/minute per client address, and 600 requests/minute per tenant (see Rate limiting: two independent ceilings per route above).

Correction: this is its own bucket, not shared with /tts

An earlier version of this page said this limit was “shared with /tts.” It isn’t — @nestjs/throttler keys its counters per route handler, so /tts and /tts/google each have their own independent 30/min-per-IP and 600/min-per-tenant counters that just happen to use the same numbers, not one pooled bucket.

Request body (Google Cloud TTS shape):

{
  "input": { "text": "Hello there", "ssml": null },
  "voice": { "languageCode": "en-US", "name": "en-US-Neural2-F" },
  "audioConfig": { "speakingRate": 1.0 }
}

Only input.text/input.ssml, voice.name, voice.languageCode, and audioConfig.speakingRate are read — any other fields a third-party library sends alongside these (e.g. enableTimePointing) are accepted and ignored, not rejected.

Response: Same shape as /tts — { audioContent: string, timepoints: [...] }.

POST /tts/gemini

Added 2026-10-01, live in production. An optional text-to-speech engine for audio that is not played back in real time, such as voice notes. It uses a Gemini speech model; /tts and /tts/google are unchanged and remain the right choice for the lip-synced avatar.

Same authentication and rate limits as the other /tts routes. Request body:

Field	Type	Notes
text	string	Required, up to 5,000 characters. Plain text only.
voiceName	string	Optional Gemini voice such as Kore (the default) or Puck. Any Cloud-style id (it contains -) selects the default voice.

Response 200:

{ "audioContent": "<base64 WAV>", "timepoints": [], "mimeType": "audio/wav" }

What differs from /tts/google:

• Audio is a 24 kHz, 16-bit mono WAV. MP3 and OGG are not produced.
• timepoints is always empty, so it cannot drive lip-sync.
• Slower: about 3 to 4 seconds for a two-sentence reply in our checks, against under a second for Cloud voices.
• The language is detected from the text. languageCode, speakingRate and SSML are not accepted.
• No silent fallback. If the engine is not configured on the environment, the route answers 503; it never returns audio from another engine.
• Metered like /tts, per character.

POST /stt

Transcribes an audio clip to text.

Auth required: Avatar token (Bearer).

Rate limit: 120 requests/minute per client address and 200 requests/minute per tenant (see Rate limiting: two independent ceilings per route above). STT is billed per 15-second audio block upstream, which is why the per-tenant ceiling is the one to plan around.

Request body:

Field	Type	Required	Notes
audioBase64	string	Yes	Base64-encoded audio, max ~4MB of raw audio (~5,600,000 base64 characters)
audioEncoding	string	Yes	MP3, LINEAR16, WEBM_OPUS, or OGG_OPUS
languageCode	string	Yes	The primary language, e.g. en-US. See Language fallback below.
alternativeLanguageCodes	string[]	No	Up to 3 fallback languages (BCP-47). Omit to use the default set; send [] to recognise in languageCode only. More than 3 is rejected with a 400. See below.
sampleRateHertz	number	No	8000–48000. For OGG_OPUS, leave this unset — see below. If omitted for WEBM_OPUS: 48000. If omitted for MP3/LINEAR16: 16000.
normalize	string	No	"numbers" turns spoken number words into digits. Off when omitted; any other value is a 400. See Number normalisation.

OGG_OPUS (OGG container, Opus codec) is the native format WhatsApp and Telegram voice notes are recorded in — send one straight through without transcoding it first. It’s a distinct value from WEBM_OPUS because the two use different container formats despite both carrying Opus audio.

For OGG_OPUS, sampleRateHertz is auto-detected from the file itself when omitted, not guessed. Different source apps record at different real rates — Telegram’s voice notes are 48kHz; a different real device/app is not guaranteed to match. Rather than assume one value, this endpoint reads the real rate directly out of the audio’s own OGG container header (the same field standard tools like ffprobe read). If you do pass an explicit sampleRateHertz for OGG_OPUS, it’s used as given and skips auto-detection entirely — only do this if you know the real rate for certain, since a wrong value produces a clean 200 with an empty transcript, not an error.

Response: 200 OK

Field	Type	Notes
transcript	string
confidence	number	0–1. 0 means “not reported”, not “low” — see below.
languageCode	string	The language the audio was actually recognised in, which can differ from the one you requested. See below.
attempts	number	How many recognition calls were made: 1 normally, or 2 when a fallback language won but returned an empty or mixed-script (Han plus Latin) transcript and was automatically rechecked.
rawTranscript	string	Only when the request sent normalize: "numbers": the transcript exactly as the engine wrote it, before normalisation.

curl -X POST $AVATAR_SERVICE_URL/stt \
  -H "Authorization: Bearer <AVATAR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "audioBase64": "UklGRi...",
    "audioEncoding": "WEBM_OPUS",
    "languageCode": "en-US"
  }'

{
  "transcript": "What time does the library close today?",
  "confidence": 0.94,
  "languageCode": "en-US"
}

transcript covers the whole clip. When the speech provider splits audio into several segments (typically at a pause), they are joined with spaces into one string, and confidence is their average. Before 2026-09-26, audio under ~50 seconds returned only the first segment, which silently dropped everything after the first pause. See Sending Voice Messages to an Agent.

Language fallback and the detected language

Added 2026-09-28. Every /stt call is now sent to the speech provider with fallback languages as well as your primary languageCode, and the provider picks whichever one the audio is actually in. Before this, each call recognised in exactly one language. A voice note in a different language didn’t fail: it came back as a short, confident-looking garbage transcript (a Mandarin clip sent as en-US came back as "for").

The default fallback set, in priority order, is en-US (English) and ms-MY (Malay). Your primary is removed from that list, leaving at most two fallbacks. (Shipping 2026-09-29: Mandarin and Cantonese are being removed from the defaults because accented English clips are being misclassified as Chinese — see below.)

• Override it with alternativeLanguageCodes: up to 3 BCP-47 codes of your choice. The Wetel-owned voice clients fill this in for you from the agent’s stored settings (live in production since 2026-10-01): see Speech-to-text languages for the per-agent sttLanguageCode / sttAlternativeLanguageCodes fields, the Telegram connector, and the <vai-avatar> stt-lang / stt-alternative-langs attributes.
• Turn it off with "alternativeLanguageCodes": [], which recognises in languageCode only (the old behaviour).
• Why the hard cap of 3: that is the speech provider’s real limit. Past it, the provider doesn’t return an error. It silently ignores the whole list, which quietly turns fallback off. We verified this live, so /stt rejects a 4th entry with a 400 rather than let that happen.

languageCode in the response is now the detected language. It used to echo whatever you sent. It now reports the language the provider actually recognised, spelled the way you spelled it in the request (so cmn-Hans-CN comes back as cmn-Hans-CN, Cantonese comes back as yue-Hant-HK if you sent it that way). When a clip has several segments in different languages, the language covering the most transcript text wins. If nothing was recognised, it falls back to your requested languageCode. Use this field to decide, for example, which language the agent should reply in.

Fallback re-check: automatic quality gate (live in production since 2026-10-01). When a fallback language wins the recognition but returns an empty transcript, or returns mixed-script garbage (a small Chinese fragment plus an English clause, like "蓝色in Chinese"), a short clip (under ~50 seconds) is automatically re-checked once in the primary languageCode only, with no fallbacks. If the primary-only result is at least as good as the first, it replaces the fallback’s result. You are billed for the transcript once, whatever attempts says (before 2026-10-21 the re-check was metered as an extra 15-second block): the second call is the platform’s own quality check. Known limits: (1) Cantonese heard on a Mandarin primary (zh/zh-*/cmn-*) counts as the primary — every Han-script winner on a Han-script primary is treated as the primary, so it is never retried. (2) Japanese fallbacks are re-checked only when the transcript is empty, never for mixed-script. (3) A genuinely garbled transcript (Mandarin with short English phrases woven in naturally) can trigger the re-check but the original is kept unless the primary-only result is at least as confident.

confidence: 0 means “not reported”. The provider sometimes returns no confidence at all when it recognised the audio through a fallback language. We have seen this with correct Malay transcripts. /stt reports that as 0, which is the provider’s own “not set” value. If your integration throws away low-confidence transcripts, apply your threshold only when confidence > 0. Otherwise you will discard correct transcripts. Wetel’s own Telegram connector was changed to work this way.

Known limits:

• Mandarin and Cantonese are not in the default fallback set (the default is en-US and ms-MY, live since 2026-10-01). The provider classifies accented English clips as Chinese (cmn-Hans-CN or yue-Hant-HK), even when the transcript is 100% English — a misclassified languageCode can be corrected when the script doesn’t match the content, but preventing the mislabel in the first place is more reliable than correcting it after the fact. See Choosing Languages and Voices. If your application genuinely needs Chinese detection, pass alternativeLanguageCodes: ["cmn-Hans-CN", "yue-Hant-HK"] or similar explicitly.
• Cloud Speech-to-Text (/stt/google) does not handle code-switching within one clip. For mixed speech (English, then Mandarin, then English again), it picks one dominant language per segment and the other language’s words are usually lost or mangled. The Gemini engine (/stt/gemini, the default /stt route today) detects the language itself and did much better on our mixed English and Malay test clips, though not perfectly. See Speech-to-Text and Text-to-Speech Integration Guide for the measurements and how to verify on your own audio.
• Hokkien and Hakka are not supported by the speech provider at any setting.
• Tested on synthetic voices only so far. Real, accented, conversational voice notes have not been measured yet. Check against your own traffic before you depend on this.

Update, 2026-09-29 — the default fallback set already changed (live in production and staging):

The speech provider classifies accented English clips (especially short clips like “Hi testing one two three”) as Mandarin or Cantonese, even though the transcript is pure English. Mandarin and Cantonese have been removed from the defaults as of 2026-09-29 (now live on staging and production) — the default set is now [en-US, ms-MY] only. This false-positive misclassification is costlier than the fix-after-the-fact language-tag correction. If a segment’s languageCode is cmn-Hans-CN or yue-Hant-HK but the transcript contains zero Han characters, the tag is still corrected to the first non-Han language you requested — but this correction only works if the fallback was included in the first place. If your integration needs Chinese detection, pass it explicitly: "alternativeLanguageCodes": ["cmn-Hans-CN", "yue-Hant-HK"] (or whichever Chinese variants you support) in the request. There is also a separate issue, unfixed: a genuinely garbled transcript traces to which language you send as languageCode (primary), not to the fallback list. Testing against real voice notes found the primary carries much more weight in the provider’s recognition than the alternatives do: the same audio produced a clean transcript with en-US as primary, and a corrupted transcript with cmn-Hans-CN as primary — reproduced exactly, byte for byte. Send your best actual guess as the primary languageCode, not just a placeholder — it isn’t “one candidate among equals” the way the alternatives are. We tested narrowing alternativeLanguageCodes as a potential fix and found it does not help (it made results worse in some cases) — the fix is a better primary guess, not a shorter fallback list.

The same detected-language behaviour applies to the LLM Playground’s transcribeSpeech query.

If the underlying speech provider is rate-limited or unavailable, this endpoint returns 503 Service Unavailable rather than a generic 500 — worth handling as a distinct, retryable case in your client.

As of 2026-09-25, this endpoint automatically routes audio over ~50 seconds through an asynchronous long-running transcription path instead of synchronous recognition — you still send the whole file in a single call and get the transcript back in the same response, just with proportionally longer latency for longer audio. (Previously this endpoint had a hard, silently-enforced ~60-second ceiling; audio longer than that failed outright. That limit no longer applies — no client-side chunking workaround is needed, and none is supported.) Confirmed working up to ~5 minutes; a much longer clip could still fail with a 500 (a real, explicit error — not silent) — see Sending Voice Messages to an Agent: constraints if this matters for your integration.

Speech engines: /stt, /stt/google, /stt/gemini

Added 2026-10-01, live in production. There are two speech-to-text engines, and you can pick one per call:

Route	Engine
POST /stt	The deployment’s default engine. Since 2026-10-02 this is Gemini, with Cloud Speech-to-Text answering automatically if Gemini errors.
POST /stt/google	Always Cloud Speech-to-Text (the behaviour described above).
POST /stt/gemini	Always the Gemini speech model (gemini-3.5-flash on Wetel’s own environments since 2026-10-08; it is configured per environment).

All three take the same request body, the same avatar token and the same rate limits, and return the same response shape. Calling an explicit route on an environment where that engine isn’t configured returns 503, never a transcript from the other engine. Plain /stt uses the default engine (see above); confidence is 0 when Gemini answered. Call /stt/google to keep the previous behaviour exactly.

What differs on /stt/gemini, measured on the same audio:

• Faster and cleaner text. About 1.4–3.0 s against 3.2–3.4 s for Cloud STT in our checks, and the text comes back capitalised and punctuated.
• No language hint needed. The model detects the language itself, so languageCode does not steer it and alternativeLanguageCodes is ignored. A Malay clip sent as en-US still came back correct. The languageCode in the response simply echoes the one you sent: this engine does not report a detected language.
• No confidence score. confidence is 0, which means “not reported”, the same as it does for the other engine when it has no score.
• No fallback re-check. attempts is not returned.
• Formats. MP3, WEBM_OPUS, OGG_OPUS and LINEAR16 all work; LINEAR16 can be a WAV file or raw 16-bit mono PCM (set sampleRateHertz for raw PCM, default 16000).

Billing. Each transcript is billed once at the rate of the engine that served it: Cloud Speech-to-Text per started 15-second block, and the Gemini engine at the Gemini model’s token rate (falling back to the block rate when the model reports no token count). On plain /stt that is whichever engine actually answered. Speech usage counts toward your workspace’s monthly spend cap.

Number normalisation (normalize: "numbers")

Live in production since 2026-10-21. Speech engines often write numbers as words (“empat lapan dua satu”, “twenty five ringgit”, “零一二三”), which a workflow that reads an amount, a phone number or an id cannot use. Send "normalize": "numbers" on /stt, /stt/google or /stt/gemini and English, Malay and Mandarin number words in the transcript become digits. It is off by default, makes no model call and does not change what is billed.

• A run of single digits is joined digit by digit and keeps leading zeros: kosong satu dua becomes 012, nine zero zero one becomes 9001, 零一二 becomes 012. English double and triple repeat the next digit (double six becomes 66).
• Anything else is read as one value: dua puluh lima becomes 25, one hundred and five becomes 105, 一千零一 becomes 1001, 三千五 becomes 3500.
• Ringgit after a number becomes a prefix: lima puluh ringgit, 50 ringgit and 五十令吉 all become RM50.
• The response then also carries rawTranscript, the engine’s text before normalisation. Keep it if you need the original wording.

For example, a clip the engine transcribes as nombor saya kosong satu dua, harga lima puluh ringgit comes back as (the confidence and languageCode values are placeholders; they are whatever the engine reported, unchanged by normalisation):

{
  "transcript": "nombor saya 012, harga RM50",
  "rawTranscript": "nombor saya kosong satu dua, harga lima puluh ringgit",
  "confidence": "<confidence as reported by the engine>",
  "languageCode": "<language code, as without normalize>"
}

Left alone on purpose, because each is ambiguous: a lone English one (“no one”, “the red one”), a lone Mandarin numeral inside a word (“一起”, “两个”), oh for zero, fractions, ordinals, decimals said as words, and digits the engine already wrote. A transcript over 20,000 characters is returned unchanged. Opting in still accepts rare false positives: Malay number words are matched whatever the request language (the city “Lima” becomes 5), an idiom such as 三三两两 becomes digits, and a year is read as two values (nineteen ninety nine becomes 19 99). Test it on your own audio before you rely on it.

POST /avatar-session

Negotiates a HOSTED_API (photorealistic video avatar) rendering session. Only relevant if your agent’s avatarBackend is configured to HOSTED_API — see Avatar & 3D Rendering: Avatar rendering options. Not used for the default 3D (CLIENT_3D) rendering path, which is driven entirely by sdkStart’s response instead.

Auth required: Avatar token (Bearer).

Rate limit: 10 requests/minute per client IP, and 200 requests/minute per tenant (added 2026-09-17 — see Rate limiting: two independent ceilings per route above). Lower than TTS/STT because this negotiates a real, per-minute-billed hosted video session with the underlying vendor, not a per-utterance call.

Request body:

Field	Type	Required	Notes
anamAvatarId	string	No	Must match the value already configured on the agent’s avatarBackend config — sent only so a mismatch can be rejected loudly.
anamAvatarModel	string	No	Same as above.

Your own client code never chooses or overrides the vendor persona/model here — the real value comes from the agent’s server-side avatarBackendConfig, embedded in the avatarToken at sdkStart time. These fields exist only so the server can detect and reject a mismatch, not so the browser can request an arbitrary persona.

Response: 200 OK

Field	Type	Notes
sessionToken	string	A vendor session credential. The browser never sees the vendor’s own API key.

curl -X POST $AVATAR_SERVICE_URL/avatar-session \
  -H "Authorization: Bearer <AVATAR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "sessionToken": "eyJhbGciOi..."
}

A negotiation failure (missing vendor credentials, vendor outage) returns a generic 500 with no vendor-specific detail — the underlying error is logged server-side, never surfaced to the browser, since it may reference internal config state.

GET /avatars/models

Returns the catalog of available 3D avatar models.

Auth required: None — this is a public catalog listing.

Response: 200 OK

{
  "models": [
    {
      "id": "avatar-01",
      "label": "Aria",
      "gender": "female",
      "path": "/avatars/models/aria.glb"
    }
  ]
}

Field	Type	Notes
id	string
label	string	Display name
gender	string
path	string	Relative path to the .glb model asset

curl $AVATAR_SERVICE_URL/avatars/models

GET /health

A basic health check endpoint — returns a 200 OK when the service is up. No auth required, no meaningful response body to parse; use it only for uptime/liveness checks, not as a functional API.

---

For the config that determines which voice/avatar a given agent uses by default, see Avatar Config. For how sessions and avatar tokens fit together, see Sessions. For the client-side components that wrap this API, see <vai-avatar> and Voice Interface.