Feat -- Stable Audio 3 (#14119)
* Stable Audio 3 First Version Commit, require model dict loading on GPUs
* Fixing bugs for sa3 VAE
* Make quality fix
* Fix the unit test failure
* Fix SA3 pipeline noise: learned text-padding embedding + no cross-attn mask
The diffusers Stable Audio 3 pipeline produced noise instead of music because
the cross-attention conditioning was built differently from the reference:
- The reference text conditioner uses padding_mode="learned": padded text
positions (~245 of 256 for a short prompt) are filled with a trained
`padding_embedding`, and the DiT attends to *all* positions (its
cross-attention mask is intentionally disabled). Our pipeline instead zeroed
padded positions and masked them out, wiping ~95% of the conditioning signal.
Changes:
- transformer_stable_audio3: add learned `prompt_padding_embedding`; in forward,
replace masked text positions with it (in cond_token_dim space, before
to_cond_embed) then attend to the full context, matching the reference.
- pipeline_stable_audio_3: stop zeroing padded positions (the DiT now handles
them); default `silence_padding_duration` to 6.0 (reference headroom default).
- convert_..._to_diffusers: convert
`conditioner.conditioners.prompt.padding_embedding` into the DiT.
- scheduling_ping_pong: pin schedule endpoints (sigmas[0]=1.0, sigmas[-1]=0.0)
to match the reference LogSNRShift endpoint preservation.
- run_..._inference: coerce float16->float32 on CPU.
- tests: update transformer state-dict expectations (523 tensors, add
prompt_padding_embedding) and ping-pong sigma endpoint assertions.
Verified against the reference with identical noise + identical conditioning:
full 8-step ping-pong trajectories agree to 4e-6 (final latent). Removed the
dev-only parity scripts (verify_*_dit_parity, verify_*_vae_parity) from the tree.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Style fix
* Autopipeline available
* Adding document description for testing examples.
* Solving fix for serge review bot suggestion
* Design review comments fix
* CI/CD Fix for ping pong scheduler md file
* Fix doc build: remove autodoc for nonexistent StableAudio3EulerSchedulerOutput
* Fix doc build: qualify non-exported autodoc references with module paths
* Fix details in scheduler and some nits
* suggestion fix
* fix-copies: propagate prepare_latents dtype fix to audio2audio/inpaint pipelines
---------
Co-authored-by: Buffett Liu <jengliu@linkedin.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
