How Does Voice AI Work? Speech Recognition, Synthetic Voices and Voice Cloning Explained

Ask your phone for the weather, dictate a text message, or ring a helpline and talk to a machine that talks back: all of that is voice AI. It is the branch of AI that turns speech into something a computer can work with, and turns the computer’s answer back into speech.

For years it sounded robotic and stumbled over names. Today a synthetic voice can be hard to tell from a real one, which is why police forces are now issuing warnings about it.

Why it matters now

On 19 October 2026 INTERPOL launched a public campaign warning that AI tools “can now convincingly reproduce the face or voice of a real person”, and that doing so no longer takes significant technical skills or money. Four days earlier the FBI warned that scammers around the US midterm elections are using AI “to generate highly realistic voices, images, and written messages” that impersonate candidates, campaigns, government bodies and trusted contacts. We covered both warnings here.

The same technology also does a lot of good: live captions, dictation, real-time translation and phone lines that understand you the first time. So it is worth knowing how it works, what it can and cannot do, and why the old advice, “listen for a robotic voice”, no longer holds.

How it works, step by step

Step 1: Sound becomes numbers

A microphone measures tiny changes in air pressure thousands of times a second. The software cuts that recording into very short slices and turns each slice into a snapshot of which pitches are loud and which are quiet. Laid side by side, these snapshots form a spectrogram, a kind of picture of sound. To an AI model, your voice is a long strip of these little pictures.

Step 2: The listener (speech recognition)

Speech recognition, also called speech-to-text, turns that strip into words. Modern systems learn by example rather than by hand-written rules. OpenAI’s Whisper, described in a 2022 paper, was trained on 680,000 hours of audio paired with transcripts in many languages. Training on that much messy, real-world audio is what lets systems like it cope with accents, background noise and people talking fast.

It still guesses. When it hears something unclear, it picks the most likely words, which is why unusual names and technical terms still come out wrong.

Step 3: The thinker

The words then go to a large language model, which works out what you meant and writes a reply. This is where the “intelligence” sits. The ears and the mouth are separate skills, which is why a voice assistant can sound fluent and still give a wrong answer.

Step 4: The speaker (text-to-speech)

Text-to-speech turns the reply back into sound. Older systems stitched together recorded fragments, which is why they sounded choppy. Newer ones chop audio into small codes, like letters in an alphabet of sound, and a model learns to predict the next code the way a chatbot predicts the next word.

Microsoft researchers’ VALL-E, published in January 2023, worked this way and was trained on 60,000 hours of English speech. Its paper reported that it could produce personalised speech from “only a 3-second enrolled recording” of a speaker it had never heard before. That is voice cloning: a model captures the sound of a person’s voice, its pitch, accent and rhythm, and then makes it say anything you type. Think of a voice like handwriting. A skilled forger needs only a few lines to copy it.

Step 5: From relay team to one model

Passing the baton three times takes time. When OpenAI introduced GPT-4o in May 2024, it said its earlier voice mode chained three separate models and took 2.8 seconds (GPT-3.5) to 5.4 seconds (GPT-4) on average to reply. The new model, which handles audio directly, could respond in “an average of 320 milliseconds”, which OpenAI said is similar to human response time in a conversation.

Handling sound directly also keeps information that the relay throws away. OpenAI noted that in the old pipeline the main model “can’t directly observe tone, multiple speakers, or background noises”, and could not laugh or sing. A single speech model can.

A worked example: the call from “your brother”

Here is how a cloned-voice scam works, and why a simple habit defeats it. The names are made up; the steps follow what INTERPOL and the FBI describe.

1. The raw material. A scammer finds a 30-second clip of Sam talking in a video he posted publicly.
2. The clone. They feed the clip to a voice-cloning tool and type a script: “I’ve had an accident, I need money for the hospital, please don’t tell Mum.”
3. The call. Sam’s sister Priya picks up. It sounds exactly like Sam. But the call has the classic signs INTERPOL lists: it is unexpected, it is urgent, and it asks for secrecy.
4. The check. Priya asks for the family’s safe word, the phrase they agreed in advance. The caller cannot give it. She hangs up and rings Sam on the number she already has. He is fine.
5. The lesson. The clone could copy Sam’s voice. It could not copy what only Sam knows, or answer the phone Sam actually owns.

INTERPOL’s advice matches this: “hang up and contact the person through a known, verified channel”, and agree a secret phrase or question with close family. Its campaign slogan sums it up: “Wait. Validate. Don’t take the bait.”

What people get wrong

“I’d know a fake voice if I heard one.” That used to be fair advice. INTERPOL’s own warning is that AI is “erasing the tell-tale signs of a scam”. Check the request and the channel, not the sound.

“Cloning a voice needs hours of recordings.” Research systems such as VALL-E reported convincing results from a few seconds. A short clip from a video or voicemail can be enough raw material.

“The assistant understood me, so it understood what I meant.” Speech recognition and understanding are different steps. A perfect transcript can still lead to a wrong answer if the language model gets the meaning wrong.

“AI voices are illegal.” The technology is not banned; how it is used is regulated. In the US, the Federal Communications Commission ruled in February 2024 that the robocall rules covering “artificial or prerecorded” voices apply to AI-generated voices, naming voice cloning specifically. So an AI-voiced robocall needs the same consent as any other.

Where it is heading

• Real-time voice agents. Big model makers are shipping speech models built for live conversation, such as Google’s Gemini 3.8 Live and NVIDIA’s open speech-to-speech model, which NVIDIA documented weak spots and all.
• Faster, streaming transcription. Microsoft has shipped its first streaming transcription model, which writes down words as you speak rather than after you stop.
• More warnings, and verification habits. INTERPOL and the FBI are now pushing the same message: verify the request through a channel you already trust.
• Rules on use, not on the technology. The FCC’s 2024 ruling is an example: it did not ban AI voices, it brought them under existing consent rules.

Key terms

• Speech recognition (speech-to-text)
• Text-to-speech
• Voice cloning
• Deepfake
• Large language model
• Inference
• Spectrogram

Sources

• INTERPOL, “INTERPOL launches public awareness campaign highlighting risk of AI deepfake scams”, 19 October 2026. interpol.int
• FBI Internet Crime Complaint Center, “Scammers Exploit 2026 U.S. Midterm Election to Perpetrate Multiple Fraud Schemes”, PSA I-101526-3-PSA, 15 October 2026. ic3.gov
• Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision” (Whisper), arXiv:2212.04356, 2022. arxiv.org
• Wang et al., “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers” (VALL-E), arXiv:2301.02111, 2023. arxiv.org
• OpenAI, “Hello GPT-4o”, 13 May 2024. openai.com
• Federal Communications Commission, Declaratory Ruling FCC 24-17 on AI-generated voices under the TCPA, adopted 2 February 2024, released 8 February 2024. fcc.gov