Add Idefics2 #30253
asartran
merged
131 commits into
main
from
add-idefics2
Merge pull request #9 from huggingface/update
e536f6a3
Merge branch 'main' of github.com:huggingface/new-model-addition
ef8c0fb4
Initial add model additions
79277fe3
Test
6c89a99f
All weights loading
2e1155b0
Can perform full forward pass
661b794b
Local and remote the same
dd0a3d29
Matching local and remote
d1248631
Fixup
7aff0b7f
Idefics2Model importable; fixup docstrings
799dc717
Don't skip by default
dbebbb1d
Remove deprecated use_resampler arg
465e3ed1
Remove self.config
ae5b94d3
DecoupledLinear takes config
7983e936
Tidy up
0a000649
Enable eager attention and tidy up
6e4ff1b7
Most tests passing
1aa8f7aa
Update for batch of processed images
ea4bf34c
Add image processor
b6a92dac
Update doc pages
0d09b95a
Update conversion script
3c11158e
Remove erroneous breakpoint
c6d45598
Remove accidendtal spelling change
c6275e98
Update to reflect changes on hub - make generate work
5dd00716
Fix up
015356b4
Image processor tests
8c501699
Update tests
da389b8b
Add a processor
e8b131d4
Add a processor
2fc3ff3c
Update convert script
e06740c8
Update modeling file - remove fixmes
083e82b3
Bug fix
256fa303
Add processing test
0fd54001
Use processor
f537f272
Fix up
d14485aa
Update src/transformers/models/idefics2/modeling_idefics2.py
02371e9a
Update src/transformers/models/idefics2/modeling_idefics2.py
7fba70a5
Fix test
0987d158
Update config - PR comments and defaults align with checkpoint
78ba5778
Reviewer comments
971dd720
Add copied froms for flahs attention
d7dfec96
Update src/transformers/models/idefics2/modeling_idefics2.py
097f4027
Apply suggestions from code review
13708368
Remove qk_layer_norm and freeze_layers functionality
9dff742a
Fix
0e1be292
Remove freeze_layer options from config
c3343070
Sync with upstream main
e5b5bc47
Fix attention shapes siglip
ec867d87
Remove Llava-next refs - TO REBASE
0019bf17
Use AutoModel for text model
b0e40819
Add comment to explain vision embeddings
863b2ee1
Fix issue with tie_word_embeddings
68990f83
Address review comments
e1456a09
Fix and fix up
f4b45d33
Chat templates for idefics
ffb2de34
Fix copies
700119d7
Fix
cefdd1d3
Add layer norms to FA2
4823ecdd
Fix tests
2de1098e
Apply suggestions from code review
5205bbaa
Fix
7edaff58
Review comments
a7a0a2cc
Update src/transformers/models/idefics2/modeling_idefics2.py
16f76663
Update inputs merger
e3a22e45
Merge weights in correct order
1c397b11
Update convert script
182ea5f4
Update src/transformers/models/idefics2/processing_idefics2.py
0ba4cc4f
Update template
65bf2235
Model code examples (fix idefics too)
84ea6e89
More review comments
ee548afb
Tidy up
649563b0
Update processing
4c4f315f
Fix attention mask preparation
f95e76bc
Update inputs_merger inputs
eae3f081
Vectorize inputs_merger
3043e40d
Update src/transformers/models/idefics2/__init__.py
914fa74d
Update src/transformers/models/idefics2/modeling_idefics2.py
877109ae
Review comments
9cde5c21
saying bye to the `qk_layer_norms`
3307e6be
Simplify
ecaac39d
Update latents
366d21d9
Remove erroneuous readme changes
5312a802
Return images when applying chat template
9d1078ba
Fix bug - prompt images are for a single sample
b1e2f428
Update src/transformers/models/idefics2/modeling_idefics2.py
09796a34
image splitting
eaff6e69
fix test
0034f848
some more comment
e2845b1b
some comment
3ae2a1b7
Apply suggestions from code review
833a802b
Update src/transformers/models/idefics2/image_processing_idefics2.py
502c3dc1
Update processor
e8ca7b38
Update model tests
4bde4066
Update src/transformers/models/idefics2/processing_idefics2.py
fea200eb
Update src/transformers/models/idefics2/processing_idefics2.py
33e51a61
Don't add BOS in template
fcad4e40
Update src/transformers/models/idefics2/processing_idefics2.py
1dc90f0c
Remove index in examples
0e16d4a5
Update tests to reflect #13
107693ad
Update src/transformers/models/idefics2/processing_idefics2.py
cd4f76a7
image splitting
eaff6e69
some more comment
e2845b1b
Update processor
e8ca7b38
Update src/transformers/models/idefics2/processing_idefics2.py
fea200eb
Don't add BOS in template
fcad4e40
Update docs
d8c50455
Update checkpoint references
e8b97511
Update examples
b5a76228
Fix and update tests
7ee86812
Small addition
31c6634a
Update tests - remove copied from as no ignore placement copy could b…
75f59efb
Update example
b5ad1354
small fixes
419fba25
Update docs/source/en/model_doc/idefics2.md
ea3838ec
Update docs/source/en/model_doc/idefics2.md
301e1c53
Update README.md
7b1c4dc0
Connector model as bridge
5be2febc
Fix up
34eb76b0
Fix up
7c73ede6
Don't pass model inputs for generation kwargs update
455dccf4
IDEFICS-2 -> Idefics2
3bbd272b
Merge pull request #18 from huggingface/vs/name-change
b122cb2d
Remove config archive name
5414a027
IDEFICS-2 -> Idefics2
3d846540
Add back llava-next
87390922
Update readmes
779a8f8c
Add requirements for processor tester
f8c53014
Use custom convert_to_rgb to avoid possible BC
661b93b8
Fix doc example
0efb5e8e
Fix doc example
fed24d14
Skip model doc tests - as model to large
541ce143
More doc example - account for image splitting
2a563b2d
asartran
commented
on 2024-04-15
Update src/transformers/image_transforms.py
26c8a553
Fix config doctest
16c8317a
VictorSanh
approved these changes
on 2024-04-15
VictorSanh
approved these changes
on 2024-04-15
Login to write a write a comment.
Login via GitHub