transformers
c5c92c8b - Add GradiumPhononTTS (#49791)

Commit
30 days ago
Add GradiumPhononTTS (#49791) * [pocket_tts] generate and tests: hooks for derived models - PocketTTSGenerationMixin._placeholder_token_ids: the ids excluded from the text count of the default frame budget, overridable by a model without a voice BOS. - The voice-as-cache check looks for the voice placeholder instead of the voice BOS (a voice always comes as both in Pocket TTS). - The tester's training and voice-cache helpers, and the test class, use the tester's model_class and audio_prompt_dim instead of PocketTTSForConditionalGeneration and latent_dim. * Add Gradium Phonon TTS (modular over Pocket TTS) GradiumPhononTTS: Pocket TTS trained by Gradium. Differences, all in the modular file: the voice prompt is made of 512-dim speaker-codec latents (`audio_prompt_dim`), a learnt speaker vector is added to every audio latent (`embed_audio_latents`), no voice BOS (config and processor). The generation mixin subclasses Pocket TTS's (nemotron3_5_asr precedent, TRF009 allowlist). Conversion, the normalising chat template and the processor tests follow. Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg * [gradium_phonon_tts] normalising chat template, processor tests The original runtime's `normalize_text` (5 languages, `lang` template variable, "none" skips it) and `prepare_text_prompt`, ported to the chat template: matches the Rust runtime's fixture (gradium-ai/xn-ptts@preview-0.4.0, 13 normalisation and 5 preparation rows) and the Python port on 72k random strings, every language. Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg * [gradium_phonon_tts] converter, Gradium's timestep norm, integration tests - Converter: the model package (GGUF, tokenizer, voices) to the TTS checkpoint and a decoder-only audio tokenizer (sliding window 251, exact GELU); the chat template; the runtime's generation config. Sanity: 51 frames, first end-of-speech logit -8.1250, as the Rust runtime. - GradiumPhononTTSTimestepNorm: Gradium's runtime normalises the time embeddings with the biased variance rescaled by sqrt((n - 1) / n) (Pocket TTS: the unbiased one), which only differs in where eps sits but moved the free-running latents by 1e-3. - No fade-in by default (Gradium's runtime applies none). - Integration tests (Rust runtime, f32): single, batch, sampled, tokenization. Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg * [gradium_phonon_tts] follow Pocket TTS: first audio position, voice items, training - `<|audio_bos|>` closes the prompt (`audio_bos_token_id`, vocab 4003), as Pocket TTS. - Chat template: the voice placeholder only with an `audio` item (empty for precomputed latents), the training branch (assistant target audio); the normalisation unchanged. - Processor: no voice BOS, voices and targets told apart by `audio_bos_token` alone. - Config docstrings carry Pocket TTS's training fields; converter: the new token, the training-only weights ignored, generation config returns the dict. - Converter sanity unchanged (51 frames, first end-of-speech logit -8.1250, as Gradium's runtime); integration 4/4 on CPU and GPU. * [gradium_phonon_tts] follow Pocket TTS: flat audio tokenizer config, decoder_seanet * [gradium_phonon_tts] conversion: latent_proj in the audio tokenizer * [gradium_phonon_tts] follow Pocket TTS: loss_function, loss weighting on the model * tmp * tmp * use pocket tts proc * use pocket tts proc * wip: uncommitted changes * [gradium_phonon_tts] GradiumPhononTTSTokenizer (modular over PocketTTSTokenizer): text preparation out of the chat template * [gradium_phonon_tts] tokenizer: normalizers for the character pass, data-driven number rewrite * [gradium_phonon_tts] drop GradiumPhononTTSTokenizer: the runtime's text preparation as the normalizer of a versioned tokenizer file * [gradium_phonon_tts] PocketTTSTokenizer does the capitalization: drop the per-letter normalizer steps * [gradium_phonon_tts] back to TokenizersBackend: the capitalization as per-letter normalizer steps * [gradium_phonon_tts] conversion: the runtime's tokenizer.json as it is, the placeholder names in tokenizer.5.0.0.json * [gradium_phonon_tts] follow Pocket TTS: latent statistics in fp32 * [gradium_phonon_tts] follow Pocket TTS: return_tensors default in tokenizer_config.json * [gradium_phonon_tts] num_speakers as a long tensor * [gradium_phonon_tts] config: plain defaults for num_speakers and duration_delta_max_period The modular converter cannot parse an `interval(...)(default=...)` override of an inherited config class. * [gradium_phonon_tts] tests: no fixtures dir Integration metadata from the bucket (as Pocket TTS), the chat template from the converted checkpoint. The Rust runtime goldens (text normalisation, prepare_text_prompt) are dropped for now, to re-add from the bucket. * [gradium_phonon_tts] no processor tests: it reuses PocketTTSProcessor, tested with Pocket TTS * [gradium_phonon_tts] fix CI: ruff format, allow config attributes read by generate and the LSD loss * [pocket_tts] cast the inputs of audio_prompt_proj / embed_audio_latents to the projection's dtype Voices and generated latents are fp32 whatever the model's dtype, so a bf16 load (e.g. dtype="auto" on a bf16 config) crashed in the projections. * [gradium_phonon_tts] config dtype from the stored flow_lm weights (bf16) - conversion: dtype="auto" now loads the TTS weights in the dtype they are stored in - embed_audio_latents override: cast to the projection's dtype, as Pocket TTS - integration tests: pin dtype=torch.float32 (Rust runtime f32 references) * [processor] forward only the Hub access and placement kwargs to the audio tokenizer The audio tokenizer lives in another repository, but `from_pretrained` forwarded every kwarg to it: `revision="refs/pr/1"` looked for that ref in the audio tokenizer's repo, and `tokenizer=...` or another sub-processor override reached the audio tokenizer model's `__init__`. It now gets `cache_dir`, `force_download`, `proxies`, `token`, `local_files_only`, `device_map` and `dtype` only. - Pocket TTS doc: no `encoder_config=None` on the processor any more (no longer forwarded) - Pocket TTS processor test: the common `test_save_load_pretrained_additional_features` passes as is, a test that `tokenizer=` and `revision` are not forwarded * [gradium_phonon_tts] tokenizer normalizer: only the runtime's language-independent text preparation The normalizer followed the runtime's English normalization as of 1 October, whatever the checkpoint's language. It now does only what does not depend on the language, as ptts v0.4.0 does it: fold the characters the vocabulary would read as raw bytes (typographic quotes and dashes, bullets, arrows, emoji, control characters, letters it lacks, subscripts), the pause punctuation, then `prepare_text_prompt`. Numbers, amounts, symbols read aloud, addresses and the like are left to the runtime's `ptts.normalize`, whose output these steps leave unchanged (checked on ~46k texts in five languages); on text without such content they match the runtime exactly. * [gradium_phonon_tts] doc: the released checkpoints, voices, batches and long text, text normalization - Overview: the seven gradium/phonon-* checkpoints, voices as precomputed latents (each checkpoint's default voice and the gradium/phonon-voices catalog) - Usage tips, as Pocket TTS's: chat template and voice latents, decoding, the runtime's sampling defaults, bf16 (float32 on CPU to match the runtime), split_text - Usage: a catalog voice; a batch of a long text split into chunks and another voice - Text normalization: what the tokenizer does and does not, ptts for the rest * [gradium_phonon_tts] conversion: no training fields in the saved config (training is not supported) * [gradium_phonon_tts] integration tests: gradium/phonon-31m-en, goldens from Gradium's runtime (ptts 0.4.0) The integration tests ran the pre-release checkpoint (eustlb/gradium-phonon-tts) against goldens from a pre-release runtime. They now run gradium/phonon-31m-en (the commit that adds the transformers files) with voices of gradium/phonon-voices, against goldens computed by ptts 0.4.0 from PyPI on the checkpoint and voice revisions it pins (f32, temperature 0, one thread): https://gist.github.com/eustlb/b54c2ef3b6d0e92b20b23a974fbefad6 - single and batch: frame counts, then waveforms (AUDIO_ATOL 2e-3: up to 5.3e-4 on the CI A10G, 7.9e-4 on CPU) - no sampled case and no per-step latents: the released runtime neither replays given noise nor returns latents
Author
Parents
Loading