Sam: Perception Encoder Audiovisual (#42905)
* Adding PE-Audio model
* Adding judge and FL-CLAP models
* lint
* Update to flatten ModernBERT
* Remove EMPTY_DICT default parameters
* Use AutoModel for modernbert
* Reference config directly instead of adding new properties to model
* Remove patch_size from `Patcher`
* Remove `Config` base class
* Rename DACVAE to DacEncoderVAE
* Move data_proj out of core `Transormer` module
* import transformer layers from qwen3
* Remove CLSToken
* Condense Patcher
* Remove `PerceptionEncoderAVTextEncoder`
* Remove normalization layer from contrastive heads
* Remove `TransformerWithInputProjection`
* Separate modalities into individual models
* Update processing code and support variable length inputs
* Remove auto_docstring
Causing an issue with `make quality`. Will revisit
* Remove perception_encoder_av, fl-clap, judge
Will add these back later in the new directory structure
* Revert unintended changes to Makefile
* Add frame level contrastive model
* Address comments
* Restructure and rename PerceptionEncoder -> PE
* Add docstrings with examples
* Rename PEAudioWithText -> PEAudioWithTextModel
* Add docstrings
* Add SamAudioJudgeModel
* Use permutted qkv projections
* Remove stochastic sampling in VAE bottleneck
This makes the audio path deterministic
* Compute correct embeddings based on inputs
* Add integration tests
* clean
* pe_audio refacto
* updates
* updates
* add test model pe_audio
* pe_audio passing API test
* pe_video passing API test
* milestone: default match pe-av-large num params
* fix
* fixes
* fixes
* updates
* fix import
* frame level
* updates
* load from main checkpoint
* sam audio judge updates
* udpates
* updates
* updates
* rename PE -> Pe
* use AutoConfig
* apply review comment
* use correct attention in sam judge
* refacto
* modular
* fix num hidden layers
* fix
* fix
* fixes
* updates
* use sub models from audio video model
* fix logit scale naming
* a single forward for pe_audio_video
* remove arch
* fix
* use tied wieghts keys
* allow loading from base model
* update text model and bottleneck
* fix
* fix
* fix
* fixes
* more fixes
* use correct output names
* fix tied_weight_keys
* no need to hardcode eps
* fix main_input_name
* fix test video encoder
* revert conversion_mapping changes
* remove sam audio judge model
* basic doc
* revert modeling common changes
* update output naming
* auto extractors
* make style
* make style
* style
* doc update
* test udpates
* make fix-copies
* updates skip some not important tests + fix some + make sure what we skip is fine in practice
* last?
* style
* fix copies
* update
* update check repo PeAudio skip init
* fixes
* update
* revert unrelated
* fix docstring shinaningans
* update typing of the config for dac
* fix post merge with main
* update
* eust fix later my boy
---------
Co-authored-by: Matt Le <lematt1991@gmail.com>
Co-authored-by: Arthur <arthur.zucker@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
