PerceptionLM #37878
Cyrilvallez
merged
77 commits into
huggingface:main
from
shuminghu:perception_lm
zucchini-nlp
commented
on 2025-04-30
plm template
0a053627
A working plm with fixed image features
259a8ec2
hacked processor
c18171e3
First version that reproduced PLM output using PE from timm.
2a1169b0
Simplify and fix tie_word_embeddings
e093e206
Use PIL resize. Simplify converstion.
31aa91b5
First version that works with video input.
91de705a
simplifed image preprocessing (not batched)
71582cc6
Minor fixes after rebasing on main.
e1d5f8f8
Video processor based on new API.
74db8848
Revert to use _preprocess for image processor.
fb5ae4bd
refactor with modular
ee716c3f
fix tie_word_embedding
65e52313
Testing with timm PE
cdbeeeb4
check in missed converstion from modular to model.py
303ddff3
First working version of PLM with Eva PE. PLM-1B and 3B outputs are e…
742c8e12
address review comments
63be4c63
Fixed batching if video and image examples mixed.
139f8298
Simplify PE configuration.
0f8663a3
Enable AutoModel for PerceptionEncoder.
82ff8bdb
Update PE config style.
66a021f1
update all headers
7d97732a
Minor fixes.
70480d46
Move lm_head to PerceptionLMForConditionalGeneration.
3d65bc9d
Fix for testing_modeling_perception_lm.py
1137a679
Image processing refactoring to use more common parts.
f1338b14
Fix processor test.
7a28970e
update tests to use model from hub
642d2e73
More test fixes.
2fadbae6
integration test GT update after rebasing; probably due to video prep…
6f2d5a36
update test media path to hub
76c9c4d1
Stop tracking local scripts
14c17551
address some review comments
dbf35c1e
refactor image processing.
15b176aa
small fixes
0337ce1b
update documentation and minor fixes
3bc30960
remove scripts
53d2f011
Minor fix for CI
350aa79a
Fix image processing
d63b4f88
CI and doc fix
836f5467
CI formatting fix
b9e5fa01
ruff fix
6b69945c
ruff formatting
6c820123
ran utils/sort_auto_mappings.py
ed1dd4b8
update docstring
b77f53e6
more docstring udpates
eebcc7a6
add vision_input_type default fallback for image processing
2c73fc4d
more verbose variable naming
6ceb83a7
test update
87c6ca4e
Remove PE and PEConfig use AutoModel(TimmWrapper) instead
d7b47d68
Minor cleanup.
cbc10576
Minor Fix: remove any ref to PE. Ruff format and check.
d62b35d8
fix docstring
f6d095ae
Fix modular/model consistency.Improvex docstringfor .
508117d7
Fix PerceptionLMForConditionalGenerationModelTest
bcedcc05
ruff fix
f25a2ca1
fix for check_repo
e8d08e80
minor formatting
4c05fb3c
dummy size arg to fix for processor test.
c74d6529
Update docstring for PerceptionLMConfig
3c46d3a2
Minor fixes from review feedback.
160b039a
Revert some minor changes per reviewer feedback.
97fbfcab
update base_model_prefix
99a1bf26
Merge branch 'main' into perception_lm
74b7995c
address reviewer feedback
73c919af
fix comment in modeling file
423af521
Merge remote-tracking branch 'hf/main' into perception_lm
c67292c6
address reviewer feedback
9d33b59e
ruff format
e928110d
Merge branch 'main' into perception_lm
ea08fd5c
Cyrilvallez
approved these changes
on 2025-07-10
Pre-merge test update.
984643f2
Merge branch 'main' into perception_lm
a3bc4505
reapply modular and fix checkpoint name
5fd9887f
processor test path
52a514af
use modular a bit more
79927dcb
remove dead code
dc6c9210
add token decorator
b277794a
Cyrilvallez
approved these changes
on 2025-07-11
Reviewers
Assignees
No one assigned
Labels
New model
Milestone
No milestone
Login to write a write a comment.
Login via GitHub