Add ColPali to 🤗 transformers #33736
ArthurZucker
merged
137 commits into
huggingface:main
from
tonywu71:add-colpali
tonywu71
commented
on 2024-10-19
feat: run `add-new-model-like`
0f5c6a78
feat: add paligemma code with "copied from"
726f1569
feat: add ColPaliProcessor
9a88bf19
feat: add ColPaliModel
fab4e462
feat: add ColPaliConfig
66656f6e
feat: rename `ColPaliForConditionalGeneration` to `ColPaliModel`
a377d60e
fixup modeling colpali
0addcabf
fix: fix root import shortcuts
e8979b98
fix: fix `modeling_auto` dict
49fb8ba5
feat: comment out ColPali test file
88b02126
fix: fix typos from `add-new-model-like`
cbd781b2
feat: explicit the forward input args
44fcd04c
feat: move everything to `modular_colpali.py`
a6ca45ad
fix: put back ColPaliProcesor
af9ca36e
feat: add auto-generated files
087870bc
fix: run `fix-copies`
cc11ef8c
fix: remove DOCStRING constants to make modular converter work
f69ee9bd
fix: fix typo + modular converter
fbe56650
fix: add missing imports
e58794c2
feat: no more errors when loading ColPaliModel
2dd52183
fix: remove unused args in forward + tweak doc
e05ea433
feat: rename `ColPaliModel` to `ColPaliForRetrieval`
bda69168
fix: apply `fix-copies`
bfff5642
feat: add ColPaliProcessor to `modular_colpali`
da4c5663
fix: run make quality + make style
ae37f186
fix: remove duplicate line in configuration_auto
38f0d8cd
feat: make ColPaliModel inehrit from PaliGemmaForConditionalGeneration
c63a3023
fix: tweak and use ColPaliConfig
d66606e1
feat: rename `score` to `post_process_retrieval`
7f750d3f
build: run modular formatter + make style
41dbbb83
feat: convert colpali weights + fixes
28592c9a
feat: remove old weight converter file
84763a3d
feat: add and validate tests
672bdb24
feat: replace harcoded path to "vidore/colpali-v1.2-hf" in tests
f7ce9b13
fix: add bfloat16 conversion in weight converter
3789a6ec
feat: replace pytest with unittest in modeling colpali test
5e09645d
feat: add sanity check for weight conversion (doesn't work yet)
8ea82734
feat: add shape sanity check in weigth converter
d1007794
feat: make ColPaliProcessor args explicit
e6bdf406
doc: add doc for ColPali
abe32322
fix: trying to fix output mismatch
6ae178cc
feat: tweaks
6d35b27f
fix: ColPaliModelOutput inherits from ModelOutput instead of PaliGemm…
06533400
fix: address comments on PR
97a6468e
fix: adapt tests to the Hf norm
8212717d
wip: try things
a7b297a7
feat: add `__call__` method to `ColPaliProcessor`
592e7162
feat: remove need for dummy image in `process_queries`
f50a9793
build: run new modular converter
25eb21b7
fix: fix incorrect method override
3ed76273
Fix tests, processing, modular, convert
9038eada
fix tokenization auto
cb7e301a
hotfix: manually fix processor -> fixme once convert modular is fixed
3f118cae
fix: convert weights working
3aa11a65
feat: rename and improve convert weight script
8ff89626
feat: tweaks
7a54fec3
fest: remove `device` input for `post_process_retrieval`
2c94eaac
refactor: remove unused `get_torch_device`
2d7e96f7
Fix all tests
11893408
docs: update ColPali model doc
246b67e5
wip: fix convert weights to hf
4a5bc0ca
fix logging modular
afbbc98d
docs: add acknowledgements in model doc
9db013d2
fix logging modular
afbbc98d
docs: add missing docstring to ColPaliProcessor
c4e156cb
docs: add doc for `ColPaliForRetrievalOutput.forward`
d6a0bde3
test: add integration test for ColPali
ffe894a3
chore: rerun convert modular
b0e33be8
build: fix root imports
f052927a
Update docs/source/en/index.md
ad09d672
wip: reduce the prediction gap in weight conversion
b6477881
tests: change dataset path to the new one in hf-internal
a711fa7d
fix: colpali weight conversion works
e9035d9a
test: add fine-grained check for ColPali integration test
9f7299b2
docs: move input docstring in a variable
f6e3155b
fix: remove hardcoded torch device in test
da03264c
docs: fix python example for ColPali
db373449
feat: add option to choose `score_retrieval`'s output dtype and device
e72c379a
feat: add `patch_size` property in ColPali model
c53ffcb5
docs: update description for ColPali cookbooks
b1027389
fix: fix root imports in transformers
da05b700
Fix CI issues
f1008886
feat: remove ColPali config and model from `modular_colpali.py`
aee8d7c6
feat: add `ColPaliPreTrainedModel` and update modeling and configurat…
f53ae20d
fix: fix auto-removed imports in root `__init__.py`
b93c76b0
fix: fix `_init_weight`
fba3b775
fix: fix `resize_token_embeddings`
be6a0bdc
fix: make `input_ids` optional in forward
ecc7982b
feat: rename `projection_layer` to `embedding_proj_layer`
b1a25ce5
fix unprotected torch import
f187bc00
change vlm_backbone_config to vlm_config
c8efb8a5
fix unprotected import in modular this time
a30a74de
fix: load config from Hub + tweaks in convert weight script
c42c61bf
docs: move example usage from model docstring to model markdown
e981b716
docs: fix input docstring for ColPali's forward method
2ce28f5f
fix: use `sub_configs` for ColPaliConfig
a582f484
fix: remove non-needed sanity checks in weight conversion script + tw…
9f34d806
fix: fix issue with `replace_return_docstrings` in ColPali's `forward`
05c29da3
docs: update docstring for `ColPaliConfig`
f67e217e
test: change model path in ColPali test
2ed868c3
fix: fix ColPaliConfig
2aa5e9d4
fix: fix weight conversion script
e6944ad5
test: fix expected weights for ColPali model
337a0a0b
docs: update ColPali markdown
c10e7608
docs: fix minor typo in ColPaliProcessor
69d01fc2
Fix tests and add _no_split_modules
8061469d
yonigozlan
commented
on 2024-11-29
ArthurZucker
commented
on 2024-12-02
add text_config to colpali config
7dce43fa
[run slow] colpali
855f1399
Merge branch 'main' into add-colpali
603e9e48
move inputs to torch_device in integration test
c41bad4e
skip test_model_parallelism
21c13096
docs: clarify quickstart snippet in ColPali's model card
505ad9ed
docs: update ColPali's model card
655bac71
Merge remote-tracking branch 'upstream/main' into add-colpali
e9af3a52
ArthurZucker
approved these changes
on 2024-12-17
Reviewers
Assignees
No one assigned
Labels
New model
Vision
Multimodal
run-slow
Milestone
No milestone
Login to write a write a comment.
Login via GitHub