Speech-to-Text and Text-to-Speech Integration Guide

This content is not available in your language yet.

Wetel exposes its speech engines over a small REST service. You can use them to turn a voice note or call recording into text, and to turn text into spoken audio, without using Wetel’s avatar or chat UI at all. This guide takes you from nothing to a verified integration, and it is honest about what works well, what is measured and what is not.

If you only want the endpoint reference, see Avatar Service REST API. If you only want to feed one voice note to an agent, see Sending Voice Messages to an Agent. This page is the longer path that connects them.

Quick start (5 minutes)

You need an API key ($WETEL_API_KEY), curl and jq, and one short voice note, note.ogg (OGG/Opus, for example a WhatsApp or Telegram voice note).

# 1. Get a speech token. The response also tells you which speech service to call.
RESP=$(curl -s https://api.wetel.dev/graphql \
  -H "Content-Type: application/json" -H "x-huat-platform: customer" \
  -H "x-api-key: $WETEL_API_KEY" \
  -d '{ "query": "mutation { speechToken { token avatarServiceUrl } }" }')
TOKEN=$(echo "$RESP" | jq -r .data.speechToken.token)
AVATAR_URL=$(echo "$RESP" | jq -r .data.speechToken.avatarServiceUrl)

# 2. Speech to text. A successful call answers 201.
curl -s "$AVATAR_URL/stt" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"audioBase64\":\"$(base64 < note.ogg | tr -d '\n')\",\"audioEncoding\":\"OGG_OPUS\",\"languageCode\":\"ms-MY\"}"
# -> {"transcript":"Boleh tak you send me the report by Friday lah?","confidence":0,"languageCode":"ms-MY"}

# 3. Text to speech. audioContent is base64 MP3.
curl -s "$AVATAR_URL/tts" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"text":"Selamat pagi. Your appointment is at 3pm.","voiceName":"ms-MY-Wavenet-A","languageCode":"ms-MY"}' \
  | jq -r .audioContent | base64 --decode > reply.mp3

If step 1 returns an error or null, check the key and that the x-huat-platform: customer header is present. If step 2 returns 401, the token has expired (it lasts about an hour): repeat step 1. The rest of this page explains each step, the mixed-language (“rojak”) results, errors, and how to verify on your own audio.

What you get

Capability	Route	Engine	Best for
Speech-to-text	`POST /stt`	The deployment default	Most integrations. Today the default is Gemini, with Cloud Speech-to-Text answering if Gemini errors.
Speech-to-text	`POST /stt/gemini`	Gemini speech model	Mixed-language speech, unknown language, voice notes. No language hint needed.
Speech-to-text	`POST /stt/google`	Google Cloud Speech-to-Text	Predictable single-language audio, when you want a confidence score and a language you chose.
Text-to-speech	`POST /tts`	Google Cloud Text-to-Speech	Natural voices in a specific language and tier, MP3 or other encodings.
Text-to-speech	`POST /tts/google`	Google Cloud (TalkingHead shape)	Lip-synced 3D avatars only. Skip it unless you use `<vai-avatar>`.
Text-to-speech	`POST /tts/gemini`	Gemini speech model	Voice notes and other audio that is not played in real time. Slower, WAV output, language from the text.

How the pieces fit

Your backend calls the GraphQL API’s speechToken with its API key and gets back { token, avatarServiceUrl }, then calls the avatar service’s /stt and /tts with that token.

flowchart TB
    B[Your backend] -->|API key| G["GraphQL API: speechToken"]
    G -->|"{ token, avatarServiceUrl }"| B
    B -->|token| A["Avatar service: /stt, /tts"]

Two services, two credentials. The GraphQL API (sandbox https://api.wetel.dev/graphql; the production base URL is issued when you go live) issues the token with your API key. The speech service does the speech work, and its address comes back in the token response (avatarServiceUrl): use that value, do not hard-code a hostname.

Prerequisites

Before you write code, you need:

A Wetel tenant and an API key. Ask your Wetel contact. The key stays on your server. Never ship it to a browser or phone.
The GraphQL hostname for your environment (the speech service address comes back with the token), and the x-huat-platform: customer header on every GraphQL call.
Audio you can name the format of. The accepted encodings are MP3, LINEAR16, WEBM_OPUS and OGG_OPUS. WhatsApp and Telegram voice notes are already OGG_OPUS, so send them as they are. A wrong audioEncoding fails with a decode error, not a helpful message.
A way to keep the token. A speech token lasts about an hour. Ask for a new one with speechToken when expiresInMs runs low; the same token is returned while it still has more than ten minutes left, so calling it often is cheap.

Step 1: get a token

curl -s https://api.wetel.dev/graphql \
  -H "Content-Type: application/json" \
  -H "x-huat-platform: customer" \
  -H "x-api-key: $WETEL_API_KEY" \
  -d '{ "query": "mutation { speechToken { token expiresInMs avatarServiceUrl } }" }'

The response gives you the token, how long it has left, and the avatarServiceUrl to send speech requests to. Keep the token on your server and reuse it for many clips. It belongs to your tenant, not to a session, and it works on the speech routes only: it cannot send messages or read events. See speechToken.

Step 2: understand the mixed-language (“rojak”) story

Malaysians switch between English, Malay, Mandarin, Cantonese and local slang inside one sentence. Speech engines handle this very differently, so read this before you choose a route. Everything below says how we know it and how well.

What rojak speech looks like

Most mixed sentences have one base language that carries the grammar, with borrowed words from the others and a few particles on top:

Base	Example	What the engine has to cope with
Malay	“Boleh tak you send me the report by Friday lah”	English words and a Malay particle (`lah`) in a Malay frame
English	“Wah lau eh, this one cannot make it lah”	Hokkien and Malay interjections spelled in Latin letters
Mandarin	“昨天我的车被 saman 了，你帮我 confirm 一下”	Malay and English words inside Chinese characters
Cantonese	“你食咗飯未呀？一齊去 mamak 食 roti canai 啦”	Colloquial Cantonese characters plus Malay food words

What each engine can do

Engine	Language handling	Rojak
Cloud Speech-to-Text (`/stt/google`)	Your `languageCode` is the primary; up to 3 alternatives. It commits to one language per segment.	Weak. Words from the minority language are replaced by look-alikes.
Gemini speech model (`/stt/gemini`, the default `/stt`)	Works out the language itself. It ignores `languageCode` and `alternativeLanguageCodes`, and gives no confidence score (`confidence` is `0`).	Much better on every mixed set we measured: English and Malay, Mandarin, Cantonese and slang.

Measured results

All of this is synthetic speech: text-to-speech voices reading mixed sentences, which is a stand-in for a Malaysian speaker, not the same thing. Counts are small and the clips are not independent, so read the direction, not the decimals.

English and Malay (66 clips, 22 sentences, 3 voices). Mean word error:

Engine	Mean word error	Clips with half or more of the words wrong
Gemini (`/stt/gemini`)	about 16%	4 of 66
Cloud, primary `ms-MY`	about 34%	17 of 66
Cloud, primary `en-US`	about 38%	22 of 66

Phone and account numbers came back exact on Gemini for all 9 clips with digits, and on 8 of 9 for Cloud. Gemini took about 2.3 s per clip and Cloud about 2.0 s.

Mandarin, Cantonese and Chinese slang (39 sentences, 113 clips). Mixed error rate, which counts every Chinese character and every Latin word as one unit. Cloud Speech-to-Text was run on all 74 clips made by Cloud voices with three settings (Mandarin, Cantonese or English as the primary); the Gemini speech model that backs /stt/gemini was run on a 30-clip sample (10 per set), because that model has a small daily request limit (see below).

Speech	Clips	Cloud, best setting	Gemini (30-clip sample)
Pure Mandarin	3	0%	not run
Mandarin with English and Malay words	28	15%	5%
Cantonese with English and Malay words	20	33%	12%
English with Chinese and Malay slang	22	44%	13%

On exactly the same 30 clips the average was 10% for Gemini against 23% for the best Cloud setting. What this means in plain terms:

Cloud copes with one language and with mixed Mandarin, but not well. All 3 pure Mandarin sentences came back exact. Mandarin with a few English or Malay words was 15 to 20% wrong; Cantonese with such words 33 to 42%; English with Chinese and Malay slang 44 to 46%.
Gemini was clearly better on every mixed set, on average by 2 to 3 times, and by more where Cloud was weakest.
The Latin words are what Cloud loses. saman came back as 三忘 (“three forgot”), confirm as 配送 (“delivery”), mamak tapau teh tarik as “mama at about 3:30”, Wah lau eh as “while I wait”.
Choosing the right primary language barely helps Cloud. The three settings were within 5 points of each other on Mandarin mixes, and the English primary was the best of the three on Cantonese.
What Gemini still gets wrong is Malay and Hokkien loanwords it spells by sound: Sibeh slow lah became “Sibuk slowlah”, Walao became “walau”. It did not return the wrong script in this run.
Strict scoring understates both on Cantonese. The engines write 今日好塞車 for the spoken 今日 traffic 好塞: the meaning is right, the loanword became a Chinese word, and a strict comparison counts it wrong. An agent reading the text would understand it.

A limit to know about. Google enforces its dedicated transcription model (gemini-3.5-transcribe) at 100 requests a day per project, even for paid accounts (a known, unresolved report on Google’s developer forum). Wetel’s own environments therefore use gemini-3.5-flash for speech-to-text (production since 2026-10-08), which has not shown that limit: a 48-hour test sent about 690 clips a day with no quota error. If an environment is configured with the dedicated model and its volume passes the limit, /stt answers from Cloud Speech-to-Text instead, so a transcript still comes back but with Cloud’s accuracy. If your volume is more than a few hundred clips a day, ask your Wetel contact how your environment is set up.

Real recordings, not synthetic (openly licensed public sets, so real people and human transcripts, though not Malaysian rojak). Mean error, with clips wrong by half or more in brackets:

Real speech	Clips	Cloud	Gemini
Spontaneous Mandarin and English mixed (Hong Kong students)	50	32 to 45% (12 to 21)	14 to 19% (1)
Malay read aloud by Malaysian speakers	25	31% (4)	4 to 6% (0)
Cantonese read aloud	25	36% (9)	7 to 16% (0)

The ranking holds on real speech. Gemini was 2 to 8 times better than Cloud on all three sets. Real spontaneous speech is harder than synthetic (fillers, restarts, names), so expect the 14 to 19% range rather than the 3 to 5% we saw on generated clips.
Cloud can decide Malay is English. With Malay as the primary language and English as an alternative (the default pair), Cloud transcribed 4 of 25 plain Malay sentences as English nonsense. If your users speak one language, set alternativeLanguageCodes: [] on Cloud.
Part of the Gemini figure is formatting, not mishearing: it sometimes writes Traditional characters, translates an English place name into Chinese, or drops “um” and “嗯”. A strict score counts those as errors.
What is still missing is spontaneous speech by Malaysians mixing Malay, English and Chinese. No openly licensed set exists that we could find, which is why the numbers above are a stand-in and why your own recordings matter.

Hokkien and Hakka

Neither engine lists Hokkien or Hakka, and neither text-to-speech engine can speak them, so we could not produce Hokkien audio to test. What we could test is the part Malaysians actually write and say most: Hokkien and Cantonese words inside English and Malay sentences (kiasu, wah lau eh, sibeh, lim kopi). Cloud got 44 to 46% of those wrong, often turning them into English look-alikes; Gemini got 13% wrong on the 10 clips it was run on. Speech that is genuinely Hokkien or Hakka from start to finish is not supported at any setting. If your users speak it, plan to collect real recordings and measure.

Recommended starting point

Situation	Use
Mixed English and Malay, or you do not know the mix	`POST /stt` (the Gemini default) or `POST /stt/gemini`. It ignores language hints. Check the script of the result and retry on `/stt/google` if it is wrong.
Mostly one language, occasional foreign words	`POST /stt/google` with that language as the primary and `alternativeLanguageCodes: []`.
Mandarin or Cantonese mixed with English or Malay	`POST /stt` (the Gemini default) first: it was 2 to 3 times better than Cloud on our samples. If you use Cloud, set the Chinese language as the primary and list the others as alternatives (for example `languageCode: "cmn-Hans-CN"`, `alternativeLanguageCodes: ["en-US", "ms-MY"]`) and expect misses on the foreign words. Measure on your own clips either way.
Chinese is expected at all	Chinese is not in the default fallback set on purpose (it made English with an accent come back as Chinese). Name it yourself.
You need a confidence score	`POST /stt/google`. Gemini returns `confidence: 0`, which means “not reported”, not “low”.

Failure patterns to guard against

Cloud: foreign words replaced by look-alikes. Nothing flags it, and confidence is not a reliable signal on a mixed clip. If a name, an amount or a place matters, confirm it back to the user.
Gemini: the wrong language or script. In 3 of the 66 English and Malay clips, English-accented Malay came back in the wrong language or script (Thai, a Hindi-like romanisation, German). Check the script of the transcript against the languages you expect and retry once on /stt/google with your best primary.
Digits. Some answers spell numbers out in words (“kosong satu dua …”). Normalise before you compare. /stt can do this for you: send normalize: "numbers" and English, Malay and Mandarin number words come back as digits, with the original text in rawTranscript. See Number normalisation.
Spoken instructions. An audio clip containing “answer this question” or “ignore your instructions” can make a speech model answer instead of transcribe. We saw the default Gemini model do this in 2 of 4 probes. Treat a transcript as untrusted text, exactly like typed user input, before it reaches an agent or a tool.

Text-to-speech with mixed text

Untested. A Cloud voice reads in one language, so Malay words inside an English sentence may sound foreign. /tts/gemini detects the language from the text and may do better, but we have not measured it. Have a native speaker listen to your own phrases on each voice you plan to ship.

How we know, and what would change this

The measurements come from an evaluation script that generates mixed-language clips, transcribes them with each engine and scores the result. Ask your Wetel contact for it: it also runs on your own recordings with the same scoring, which is the evidence that should decide your setup. Real recordings of accented, spontaneous Malaysian speech, Hokkien and Hakka speakers especially, are the missing piece, and the numbers above will be replaced when we have them.

Step 3: transcribe

const res = await fetch(`${avatarServiceUrl}/stt/gemini`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: `Bearer ${speechToken}`,
  },
  body: JSON.stringify({
    audioBase64: audioBuffer.toString("base64"),
    audioEncoding: "OGG_OPUS",
    languageCode: "ms-MY", // sent, not used to steer Gemini
  }),
});
const { transcript, confidence, languageCode } = await res.json();

Rules that bite:

A successful /stt call answers 201 Created, not 200. Treat any 2xx as success.
languageCode is required on every call, even when the engine ignores it. The response echoes it back on Gemini and reports the detected language on Cloud.
Leave sampleRateHertz unset for OGG_OPUS. It is read from the file. A wrong explicit value gives a clean but empty transcript with no error.
Audio over about 50 seconds is handled automatically, and clips up to about five minutes are confirmed working. The request stays open until transcription finishes.
The body limit is about 4 MB of raw audio (about 5,600,000 base64 characters).
Throttles on every /stt route: 120 requests per minute per client address and 200 per minute per tenant.

Step 4: synthesise speech

curl -s "$AVATAR_URL/tts" \
  -H "Authorization: Bearer $AVATAR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Selamat pagi. Your appointment is at 3pm.",
    "voiceName": "ms-MY-Wavenet-A",
    "languageCode": "ms-MY"
  }'

The response is { "audioContent": "<base64>", "timepoints": [...] }. Decode audioContent and play or store it.

A Google voice name carries its language and tier, for example en-US-Chirp3-HD-Kore. Keep the voice, languageCode and tier consistent. See Choosing Languages and Voices.
Text is limited to 5,000 characters per call. TTS is billed per character.
Throttles: 300 requests per minute per client address and 600 per minute per tenant.
For a non-real-time voice note, POST /tts/gemini takes { "text", "voiceName" } (a bare Gemini name such as Kore) and returns a 24 kHz WAV. It is slower, about three to four seconds, and ignores SSML and speaking rate.

Step 5: handle errors

Status	Meaning	What to do
`400`	Validation: bad encoding, missing field, more than 3 alternatives	Fix the request. Do not retry unchanged.
`401`	Missing, malformed or expired token	Call `speechToken` for a new one, then retry once.
`429`	Throttled, or provider quota reached (quota errors carry `Retry-After`)	Wait the stated seconds. Back off. Do not hammer.
`503`	Provider unavailable, or an explicit engine is not configured	Retry later. An explicit route never silently uses the other engine.
`500`	Provider error, with a reason in the body	Log the body. A very long clip can cause it.

Never treat an empty transcript as an error by itself. Silence legitimately returns an empty string.

Step 6: test and verify on your own audio

Do not accept the advice on this page on trust. Run this small check before you go live.

Ask your Wetel contact for the evaluation script (speech-eval). It runs steps 3 to 6 for you: you give it a folder of clips and a manifest.json with the expected text, and it paces itself to the rate limits and prints a report per engine.

Collect 20 to 50 real clips from your own users, in the formats you will actually receive, covering the language mixes you expect. Include short clips, noisy clips and a few that are one language only.
Write the correct transcript by hand for each one.
Run every clip through both engines: /stt/gemini and /stt/google (with your best primary, and again with alternativeLanguageCodes: []). Pause between calls so you stay within the per-address limit (120 per minute on every /stt route, see the throttles above).
Score them. Compute word error rate against your hand transcript, or at minimum count clips where a key word or number was wrong or lost. Compare the engines per language mix, not as one average.
Check the numbers that matter to your product. Names, phone numbers, amounts and dates fail first. Test those on purpose.
Check cost and speed. Gemini was faster in our checks. A Gemini transcript is billed at the Gemini model’s token rate, and Cloud is billed per started 15-second block, so a 16-second Cloud clip bills two blocks; see Speech engines.
For TTS, listen. Have a native speaker rate your real phrases, mixed-language ones included, on each voice and tier you plan to ship.

A good result: the engine you choose gets your critical words right on at least the share of clips your product needs, on your audio, not ours.

Step 7: go live

Mint a token on your server, cache it, and refresh it before it expires.
Keep the API key out of every client.
Log the response languageCode and attempts. A transcript is billed once whatever attempts says (before 2026-10-21 in production, attempts: 2 on Cloud meant two billed recognition calls).
Alert on a rising 429 or 503 rate.
Re-run Step 6 whenever you change engine, voice or language settings, and after any Wetel announcement that changes a default. See the changelog.

Who is this for?

Partner platforms that bridge a channel (WhatsApp, Telegram, a call recorder) can transcribe a note and pass the text to an agent. Use this page together with Sending Voice Messages to an Agent.
Teams building their own voice features can use the same routes without any Wetel agent or session.
Teams who want speech inside a Wetel agent do not call these routes at all. Set the agent’s speech languages and voice instead. See Choosing Languages and Voices and <vai-avatar>.