How it works
Cross-lingual voice cloning, and what it costs you
Yes, you can clone from an English clip and generate Spanish. The question nobody answers is what that actually sounds like to a Spanish speaker.
6 min read ·
Cross-lingual cloning is the feature that sounds like magic in a demo and produces complaints in production, because the demo is judged by someone who does not speak the target language.
What is actually happening
A zero-shot cloning model encodes your reference clip into a compact description of your voice, then conditions generation on it. That description captures voice *identity* — roughly, what your vocal tract does — reasonably separately from what language you were speaking.
Reasonably separately, not completely. And the leakage is the whole story.
What transfers well
- Timbre. Whether you sound bright or dark, thin or full. This is the largest component of what people mean by "that sounds like him", and it crosses languages nearly intact.
- Pitch range. Where your voice sits and how far it moves.
- Perceived age and gender, which follow from the above.
- Speaking rate, approximately, though it gets pulled toward the target language's natural pace.
What does not transfer
- Accent, in the wrong direction. A clone from an English reference generating Spanish tends to sound like a Spanish speaker with an English-influenced accent. Sometimes subtly; sometimes obviously.
- Language-specific phonation. Sounds your reference language does not contain have no example in your clip. The model produces them from its general training rather than from you, so they sound like someone else briefly.
- Prosodic habits. Your particular way of landing a sentence is partly a fact about your language, and it does not survive the crossing.
When it is genuinely the right call
- Dubbing your own content into a language you do not speak, where a same-voice-across-languages effect is the goal and a slight accent is acceptable or even appropriate.
- Related languages. Spanish to Italian, Hindi to Marathi — closer phonologies leak less.
- Internal, draft or scratch audio where speed matters more than polish.
- Where the accent is the point. A deliberately foreign-accented read is a legitimate creative choice.
When to record a second reference instead
If you speak the target language at all, record a reference in it. A single extra clip of a few seconds removes the entire accent problem, and voice slots exist precisely so one person can hold several.
This is the recommendation we give by default: same person, one reference per language you publish in. It is more work than checking a box, and it is the difference between output that passes and output that gets comments.
Where it does not work at all
On Mayantar, cross-lingual generation is a property of the multilingual generation path — the twenty-one non-Indian, non-English languages. The Indian-language path is a different model with a different constraint: it conditions on the reference transcript as well as the audio, which makes a reference in the wrong language actively harmful rather than merely accented.
So for Hindi, Tamil, Bengali and the rest: record in the language you intend to publish in. Not a preference — a requirement for output that works.