Sam: Perception Encoder Audiovisual #42905
ArthurZucker
merged
112 commits into
huggingface:main
from
eustlb:sam-changes
Adding PE-Audio model
eca2792f
Adding judge and FL-CLAP models
8cb157a2
lint
904040eb
Update to flatten ModernBERT
0d72555a
Remove EMPTY_DICT default parameters
2891cfd7
Use AutoModel for modernbert
8ec55ac3
Reference config directly instead of adding new properties to model
fdd9a119
Remove patch_size from `Patcher`
60d8b0cc
Remove `Config` base class
386b2aae
Rename DACVAE to DacEncoderVAE
22aa356b
Move data_proj out of core `Transormer` module
75e262b3
import transformer layers from qwen3
a517bd56
Merge pull request #2 from huggingface/sam-qwen3-layers
3b2e5fff
Remove CLSToken
1cfed026
Condense Patcher
cfb35f96
Remove `PerceptionEncoderAVTextEncoder`
9ae4d201
Remove normalization layer from contrastive heads
5327bfef
Remove `TransformerWithInputProjection`
8e791fb4
Separate modalities into individual models
7a740cc0
Update processing code and support variable length inputs
b058cf26
Remove auto_docstring
8e8652f2
Remove perception_encoder_av, fl-clap, judge
4053ab42
Revert unintended changes to Makefile
501b520e
Add frame level contrastive model
f90a5564
Address comments
591a3bc0
Restructure and rename PerceptionEncoder -> PE
2e772528
Add docstrings with examples
83f66916
Rename PEAudioWithText -> PEAudioWithTextModel
3f556a47
Add docstrings
d21e34ae
Add SamAudioJudgeModel
16b8f0ff
Use permutted qkv projections
fa261f2b
Remove stochastic sampling in VAE bottleneck
e7091097
Compute correct embeddings based on inputs
03dc6dc5
Add integration tests
05fbd166
clean
5f753091
pe_audio refacto
6041d936
updates
1f3f457b
updates
b19c326b
add test model pe_audio
80509d1e
pe_audio passing API test
bbf2fcd0
pe_video passing API test
f52db5ab
milestone: default match pe-av-large num params
2adc597f
fix
e7dddc15
fixes
4defbbbe
fixes
ab5c7521
updates
e2e47478
fix import
b605d44e
frame level
f7137bff
Merge branch 'main' into sam-changes
c5197fc8
updates
de5af7f0
load from main checkpoint
032b4522
sam audio judge updates
40e36667
udpates
b2c664df
updates
48ab1f61
updates
f54ce5e8
rename PE -> Pe
e71e50fa
use AutoConfig
f43ddc30
apply review comment
76791d2b
use correct attention in sam judge
4c658904
refacto
2e97eeb4
modular
1bf8b022
fix num hidden layers
abc6387b
fix
df2b1f5a
fix
d9df5902
fixes
10235f14
updates
95a82352
use sub models from audio video model
083dab0d
fix logit scale naming
420f255b
a single forward for pe_audio_video
16b42b5b
remove arch
1ec6c565
fix
b2b2a4e8
use tied wieghts keys
eaa64b73
allow loading from base model
28584751
update text model and bottleneck
334aa44b
fix
5305fed9
fix
84a6a06f
fix
17ce3af7
fixes
12ffe84e
more fixes
4dd3ca4f
use correct output names
e188072a
fix tied_weight_keys
76af9638
no need to hardcode eps
c2607600
fix main_input_name
8be1ce2d
fix test video encoder
95036a84
revert conversion_mapping changes
5cb4ec4b
remove sam audio judge model
5a8edfec
basic doc
4236fc79
revert modeling common changes
1fe8c316
update output naming
c5981504
auto extractors
163dad4f
make style
902a5837
make style
294b3737
style
119d6068
doc update
05ddacf6
test udpates
3140a024
make fix-copies
0ea5234d
updates skip some not important tests + fix some + make sure what we …
02512fce
last?
4413f281
Merge branch 'sam-changes' of https://github.com/eustlb/transformers …
019cff48
style
ccb6ee1e
fix copies
34505a29
update
d34fb728
update check repo PeAudio skip init
d0b41f94
fixes
ef4a221e
update
41601e56
revert unrelated
0512261d
fix docstring shinaningans
ecf3b43d
ArthurZucker
changed the title
Merge branch 'main' into sam-changes
ae7f05bf
ArthurZucker
approved these changes
on 2025-12-16
update typing of the config for dac
9a6e5ddc
fix post merge with main
fe7876ee
update
cbc0db7c
eust fix later my boy
da964132
Login to write a write a comment.
Login via GitHub