Voice notes for an AI employee: English, Malay, Chinese, and what we have not measured

Picture a colleague listening to a customer’s voicemail in a busy office. They play it, catch most of it, and write down what they heard. If the customer switched from Malay to English halfway through, a colleague who speaks both copes fine; one who only expected English might write down half a sentence. If a phone number came through muffled, a careful colleague calls back to check it rather than guessing.

That is a fair picture of what an AI employee does with a voice note. It hears the recording, turns it into text, and then answers the text the way it would answer a typed message. How well the first step goes depends on the recording, the languages in it, and the settings someone chose. This post explains each part as it works in Wetel today, for a customer-service lead in Malaysia or the region and for the developer who wires it up. It ends with what we have not measured, which matters as much as what we have built.

What happens to a voice note, step by step

A voice note becomes text first, and from then on it is handled exactly like a typed message. The workflow, the persona and the reply are the same ones you already built for text.

flowchart LR
  A["Customer records<br/>a voice note"] --> B["Speech-to-text<br/>turns it into a transcript"]
  B --> C["Transcript sent to the agent<br/>as a normal message"]
  C --> D["Workflow runs"]
  D --> E["Reply as text"]
  D -. "if voice replies are on" .-> F["Reply as a voice note"]

There are two documented ways to get there.

Wetel’s own channel connector for a messaging app transcribes voice notes for you, using the agent’s language settings (see below). It can also answer with a voice note: turn on voiceReplyEnabled and a voice note gets a voice reply while a text message still gets text; add alwaysVoiceReply and every reply is spoken. Both are off by default. See updateChannelConnectorSettings.
Your own bridge (a partner system or your own app) sends the audio to the speech service’s /stt route, takes the transcript and sends it to the agent as an ordinary message. The voice messages walkthrough has the copy-paste version. For a spoken reply, your bridge calls the text-to-speech route itself.

The request your bridge sends is small. Voice notes from the common messaging apps are already in the OGG_OPUS format, so they go straight through without conversion:

{
  "audioBase64": "T2dnUwACAAAAAAAAAAB...",
  "audioEncoding": "OGG_OPUS",
  "languageCode": "en-US"
}

The answer carries the transcript, a confidence and the languageCode. One detail trips people up: a confidence of 0 means “not reported”, not “low”. Some correct transcripts, Malay ones in particular, come back with no score. If you throw away low-confidence transcripts, only apply the threshold when confidence is above zero.

To call the speech routes without starting a conversation, use a speech token from speechToken. It belongs to your workspace and works on the speech routes only.

Which languages work today

Wetel does not keep its own list of supported languages; the speech engines’ published lists are the source of truth, and Choosing Languages and Voices links to them. What we can tell you is how English, Malay and Chinese behave, because that is where the settings matter most.

There are two speech engines, and you can pick one per request (/stt reference):

The default engine works out the language by itself. It ignores any language hint, gives no confidence score, and returns capitalised, punctuated text. If it errors, the second engine answers automatically.
The second engine uses the language you set. You give it a primary language and up to three alternatives it may switch to.

On mixed speech the docs are plain: the second engine picks one dominant language per segment and usually loses the other language’s words, while the default engine copes much better but “is not perfect either”. Hokkien and Hakka are not supported at any setting. Hokkien or Cantonese words dropped into an English or Malay sentence (the way many Malaysians speak) are handled with mixed results, and the speech guide says so.

Setting the languages for each agent

Each agent can carry its own listening languages, and every Wetel voice client reads them, so one agent hears the same languages on every channel. This has been live in production since 2026-10-01.

sttLanguageCode is the primary language the agent expects to hear. Left empty, it follows the agent’s speaking language (ttsLang).
sttAlternativeLanguageCodes are the languages it may switch to. Left unset, the default is English (en-US) and Malay (ms-MY). An empty list ([]) means “primary only”. At most three, with no duplicates and not repeating the primary; anything else is refused when you save, rather than silently ignored.

The documented advice, in short:

Your customers	Set this
Speak one language	That language as the primary, alternatives `[]`
Mix English and Malay	Leave the defaults
Speak Chinese	A Chinese code such as `cmn-Hans-CN` as the primary, others as alternatives
You are not sure	The default engine, which detects the language itself

Chinese is left out of the defaults on purpose: with it included, accented English sometimes came back labelled as Mandarin, with an empty or garbled transcript. And the primary matters far more than the alternatives, so never make a secondary language the primary. A Malay-only agent, straight from the docs:

mutation {
  updateAgent(
    id: 12
    input: {
      ttsLang: "ms-MY"
      sttLanguageCode: "ms-MY"
      sttAlternativeLanguageCodes: []
    }
  ) {
    id
  }
}

Note that these settings steer the second engine. The default engine ignores them and decides the language from the audio.

The documented limits

These limits come straight from the docs; check with us if one matters to your design.

Length: clips up to about five minutes are confirmed working, sent as one request. Anything over about 50 seconds is handled automatically on a longer path, so do not split audio yourself. A clip of ten minutes or more may still fail, with a clear error rather than silently.
Size: about 4 MB of raw audio per request, well above a normal voice note.
Formats: MP3, LINEAR16, WEBM_OPUS and OGG_OPUS. Leave the sample rate unset for OGG_OPUS; it is read from the file, and a wrong explicit value gives an empty transcript with no error.
Pauses: a voice note with pauses comes back as one full transcript. (Before 2026-09-26, short clips lost everything after the first pause; that is fixed.)
Volume: 200 speech-to-text requests a minute per workspace and 120 per caller address. Speech is billed per started 15-second block.

One more rule is about safety, not size: treat a transcript as untrusted text, exactly like typed input. A recording that says “ignore your instructions” can make a speech model answer instead of transcribe, and the docs record seeing this. Whatever the transcript says, your workflow, and the person who owns it, decide what the AI employee may do.

What we have not measured

We have not measured transcription accuracy on real customer recordings, and this post claims no accuracy figure. The speech guide does publish the evaluations we ran, so you can see exactly what they are: synthetic speech (text-to-speech voices reading mixed sentences) and openly licensed public recordings, none of them spontaneous Malaysian speakers mixing Malay, English and Chinese. We could not find such a recording set, which is the gap that matters most.

So the honest position is:

Your audio is the test that counts. The speech guide’s verification step explains how to run 20 to 50 of your own clips through both engines and score them, and we can give you the evaluation script.
Names, phone numbers, amounts and dates fail first. If one of those drives an action, have the AI employee read it back to the customer to confirm before acting.
Spoken replies in mixed language are untested. A voice reads in one language, so Malay words in an English sentence may sound foreign. Have a native speaker listen to your phrases before you ship.

What is not built yet

Phone calls are not built: an AI employee cannot receive or place a real phone call today. That, and these voice items, are on the roadmap as direction, with no dates:

Streaming recognition, where text arrives while the person is still speaking (needed for live calls).
Long recordings with speaker labels, for interviews and call recordings.
Custom vocabulary, for your product names and local words.
Accuracy and fallback monitoring, showing how often each engine answered.

What is live is the voice-note path above: record, transcribe, answer, and optionally reply with a voice note.

Sending Voice Messages to an Agent: transcribe a voice note and send it to an agent, with the constraints
Speech-to-Text and Text-to-Speech Integration Guide: both engines, mixed-language behaviour, the evaluations we ran and how to verify on your own audio
Choosing Languages and Voices: primary and alternative languages, voice tiers
Agents: speech-to-text languages: the per-agent fields and their rules
Channels: voice notes and speech-to-text languages: how the channel connector uses the agent’s settings
Avatar Service REST API: POST /stt: the request, response and engine choice
Roadmap: telephony and the voice items still ahead

If your customers already send voice notes, tell us which work you’d hand over first and we will tell you honestly what works today and what you should test on your own recordings.