Add GradiumPhononTTS (#49791)
* [pocket_tts] generate and tests: hooks for derived models
- PocketTTSGenerationMixin._placeholder_token_ids: the ids excluded from the text count
of the default frame budget, overridable by a model without a voice BOS.
- The voice-as-cache check looks for the voice placeholder instead of the voice BOS
(a voice always comes as both in Pocket TTS).
- The tester's training and voice-cache helpers, and the test class, use the tester's
model_class and audio_prompt_dim instead of PocketTTSForConditionalGeneration and
latent_dim.
* Add Gradium Phonon TTS (modular over Pocket TTS)
GradiumPhononTTS: Pocket TTS trained by Gradium. Differences, all in the modular file:
the voice prompt is made of 512-dim speaker-codec latents (`audio_prompt_dim`), a
learnt speaker vector is added to every audio latent (`embed_audio_latents`), no voice
BOS (config and processor). The generation mixin subclasses Pocket TTS's
(nemotron3_5_asr precedent, TRF009 allowlist). Conversion, the normalising chat
template and the processor tests follow.
Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg
* [gradium_phonon_tts] normalising chat template, processor tests
The original runtime's `normalize_text` (5 languages, `lang` template variable, "none"
skips it) and `prepare_text_prompt`, ported to the chat template: matches the Rust
runtime's fixture (gradium-ai/xn-ptts@preview-0.4.0, 13 normalisation and 5 preparation
rows) and the Python port on 72k random strings, every language.
Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg
* [gradium_phonon_tts] converter, Gradium's timestep norm, integration tests
- Converter: the model package (GGUF, tokenizer, voices) to the TTS checkpoint and a
decoder-only audio tokenizer (sliding window 251, exact GELU); the chat template;
the runtime's generation config. Sanity: 51 frames, first end-of-speech logit
-8.1250, as the Rust runtime.
- GradiumPhononTTSTimestepNorm: Gradium's runtime normalises the time embeddings with
the biased variance rescaled by sqrt((n - 1) / n) (Pocket TTS: the unbiased one), which
only differs in where eps sits but moved the free-running latents by 1e-3.
- No fade-in by default (Gradium's runtime applies none).
- Integration tests (Rust runtime, f32): single, batch, sampled, tokenization.
Claude-Session: https://claude.ai/code/session_01ECprm731kAdBUko4MjrXDg
* [gradium_phonon_tts] follow Pocket TTS: first audio position, voice items, training
- `<|audio_bos|>` closes the prompt (`audio_bos_token_id`, vocab 4003), as Pocket TTS.
- Chat template: the voice placeholder only with an `audio` item (empty for precomputed
latents), the training branch (assistant target audio); the normalisation unchanged.
- Processor: no voice BOS, voices and targets told apart by `audio_bos_token` alone.
- Config docstrings carry Pocket TTS's training fields; converter: the new token, the
training-only weights ignored, generation config returns the dict.
- Converter sanity unchanged (51 frames, first end-of-speech logit -8.1250, as Gradium's
runtime); integration 4/4 on CPU and GPU.
* [gradium_phonon_tts] follow Pocket TTS: flat audio tokenizer config, decoder_seanet
* [gradium_phonon_tts] conversion: latent_proj in the audio tokenizer
* [gradium_phonon_tts] follow Pocket TTS: loss_function, loss weighting on the model
* tmp
* tmp
* use pocket tts proc
* use pocket tts proc
* wip: uncommitted changes
* [gradium_phonon_tts] GradiumPhononTTSTokenizer (modular over PocketTTSTokenizer): text preparation out of the chat template
* [gradium_phonon_tts] tokenizer: normalizers for the character pass, data-driven number rewrite
* [gradium_phonon_tts] drop GradiumPhononTTSTokenizer: the runtime's text preparation as the normalizer of a versioned tokenizer file
* [gradium_phonon_tts] PocketTTSTokenizer does the capitalization: drop the per-letter normalizer steps
* [gradium_phonon_tts] back to TokenizersBackend: the capitalization as per-letter normalizer steps
* [gradium_phonon_tts] conversion: the runtime's tokenizer.json as it is, the placeholder names in tokenizer.5.0.0.json
* [gradium_phonon_tts] follow Pocket TTS: latent statistics in fp32
* [gradium_phonon_tts] follow Pocket TTS: return_tensors default in tokenizer_config.json
* [gradium_phonon_tts] num_speakers as a long tensor
* [gradium_phonon_tts] config: plain defaults for num_speakers and duration_delta_max_period
The modular converter cannot parse an `interval(...)(default=...)` override of an inherited config class.
* [gradium_phonon_tts] tests: no fixtures dir
Integration metadata from the bucket (as Pocket TTS), the chat template from the converted checkpoint. The Rust
runtime goldens (text normalisation, prepare_text_prompt) are dropped for now, to re-add from the bucket.
* [gradium_phonon_tts] no processor tests: it reuses PocketTTSProcessor, tested with Pocket TTS
* [gradium_phonon_tts] fix CI: ruff format, allow config attributes read by generate and the LSD loss
* [pocket_tts] cast the inputs of audio_prompt_proj / embed_audio_latents to the projection's dtype
Voices and generated latents are fp32 whatever the model's dtype, so a bf16 load
(e.g. dtype="auto" on a bf16 config) crashed in the projections.
* [gradium_phonon_tts] config dtype from the stored flow_lm weights (bf16)
- conversion: dtype="auto" now loads the TTS weights in the dtype they are stored in
- embed_audio_latents override: cast to the projection's dtype, as Pocket TTS
- integration tests: pin dtype=torch.float32 (Rust runtime f32 references)
* [processor] forward only the Hub access and placement kwargs to the audio tokenizer
The audio tokenizer lives in another repository, but `from_pretrained` forwarded every kwarg to it:
`revision="refs/pr/1"` looked for that ref in the audio tokenizer's repo, and `tokenizer=...` or another
sub-processor override reached the audio tokenizer model's `__init__`. It now gets `cache_dir`, `force_download`,
`proxies`, `token`, `local_files_only`, `device_map` and `dtype` only.
- Pocket TTS doc: no `encoder_config=None` on the processor any more (no longer forwarded)
- Pocket TTS processor test: the common `test_save_load_pretrained_additional_features` passes as is, a test that
`tokenizer=` and `revision` are not forwarded
* [gradium_phonon_tts] tokenizer normalizer: only the runtime's language-independent text preparation
The normalizer followed the runtime's English normalization as of 1 October, whatever the checkpoint's language. It now
does only what does not depend on the language, as ptts v0.4.0 does it: fold the characters the vocabulary would read as
raw bytes (typographic quotes and dashes, bullets, arrows, emoji, control characters, letters it lacks, subscripts), the
pause punctuation, then `prepare_text_prompt`. Numbers, amounts, symbols read aloud, addresses and the like are left to
the runtime's `ptts.normalize`, whose output these steps leave unchanged (checked on ~46k texts in five languages); on
text without such content they match the runtime exactly.
* [gradium_phonon_tts] doc: the released checkpoints, voices, batches and long text, text normalization
- Overview: the seven gradium/phonon-* checkpoints, voices as precomputed latents (each checkpoint's default voice and
the gradium/phonon-voices catalog)
- Usage tips, as Pocket TTS's: chat template and voice latents, decoding, the runtime's sampling defaults, bf16 (float32
on CPU to match the runtime), split_text
- Usage: a catalog voice; a batch of a long text split into chunks and another voice
- Text normalization: what the tokenizer does and does not, ptts for the rest
* [gradium_phonon_tts] conversion: no training fields in the saved config (training is not supported)
* [gradium_phonon_tts] integration tests: gradium/phonon-31m-en, goldens from Gradium's runtime (ptts 0.4.0)
The integration tests ran the pre-release checkpoint (eustlb/gradium-phonon-tts) against goldens from a pre-release
runtime. They now run gradium/phonon-31m-en (the commit that adds the transformers files) with voices of
gradium/phonon-voices, against goldens computed by ptts 0.4.0 from PyPI on the checkpoint and voice revisions it pins
(f32, temperature 0, one thread): https://gist.github.com/eustlb/b54c2ef3b6d0e92b20b23a974fbefad6
- single and batch: frame counts, then waveforms (AUDIO_ATOL 2e-3: up to 5.3e-4 on the CI A10G, 7.9e-4 on CPU)
- no sampled case and no per-step latents: the released runtime neither replays given noise nor returns latents