transformers.js
Add support for VITS (multilingual TTS)
#466
Merged
Overview
Commits
15
Changes
View On GitHub

Add support for VITS (multilingual TTS) #466

xenova merged 15 commits into main from add-vits
xenova
xenova1 year ago

This PR adds support for VITS (text-to-speech). In particular, it allows us to run Meta's large collection of MMS models, covering over 1000 different languages!

closes #429

Example usage:

  • English:

    import { pipeline } from "@xenova/transformers";

const synthesizer = await pipeline('text-to-speech', 'Xenova/mms-tts-eng');
const out = await synthesizer('I love transformers');
// {
//   audio: Float32Array(26112) [...],
//   sampling_rate: 16000
// }
    english.mp4

  • French:

    import { pipeline } from "@xenova/transformers";

const synthesizer = await pipeline('text-to-speech', 'Xenova/mms-tts-fra');
const out = await synthesizer('Bonjour');
// {
//   audio: Float32Array(23808) [...],
//   sampling_rate: 16000
// }
    french.mp4

Surprisingly, the quantized versions of the models still work quite well 👍.

See here for the full list of supported languages.

xenova Add custom VITS tokenizer converter
6c7ed967
xenova Do not decode if expected input_ids is empty
53e78a03
xenova Update vits tokenizer tests
21ed1f1c
xenova Implement `VitsTokenizer`
aa66b4f7
xenova Add support for VITS model
f1ca1d90
xenova Support VITS through pipeline API
d78cf16a
xenova Update JSDoc
019ed2b5
xenova Add TTS unit test
381b5eb1
xenova Add speecht5 unit test
5da17e0a
HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev1 year ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

xenova Fix typo
df39fa4e
xenova Fix typo
83d33720
xenova Update speecht5 model id
27873298
xenova Add note about using quantized speecht5 in unit tests
9f0ccfeb
xenova Monkey-patch `BigInt64Array` and `BigUint64Array`
fb4850a6
arpu
xenova Merge branch 'main' into add-vits
b4a78bfb
xenova xenova merged 1394f731 into main 1 year ago
xenova xenova deleted the add-vits branch 42 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum