PR #32238 Attn implementation for composite models

first try

zucchini-nlp committed 2 years ago

codestyle

zucchini-nlp committed 2 years ago

idefics2 is happy

zucchini-nlp committed 2 years ago

[run-slow] llava, llava_next, video_llava, vipllava, llava_next_video, idefics, idefics2, kosmos2, fuyu, blip, blip_2, instructblip, instructblipvideo, paligemma

zucchini-nlp committed 2 years ago

fix-copies

zucchini-nlp committed 2 years ago

[run-slow] llava, llava_next, video_llava, vipllava, llava_next_video, idefics, idefics2, kosmos2, fuyu, blip, blip_2, instructblip, instructblipvideo

zucchini-nlp committed 2 years ago

blip-2 needs to init vision from config

zucchini-nlp committed 2 years ago

when was this removed O_o

zucchini-nlp committed 2 years ago

minor fix

zucchini-nlp committed 2 years ago

tests

zucchini-nlp committed 2 years ago

this way?

zucchini-nlp committed 2 years ago

tests

zucchini-nlp committed 2 years ago

model-agnostic code

zucchini-nlp committed 2 years ago

codestyle

zucchini-nlp committed 2 years ago

add tests for idefics

zucchini-nlp committed 2 years ago

modify general test for VLMs

zucchini-nlp committed 2 years ago

no generation test for vlm yet!

zucchini-nlp committed 2 years ago

no generation test here also

zucchini-nlp committed 2 years ago

wanr in VIT-SDPA if output attn

zucchini-nlp committed 2 years ago

add more tests

zucchini-nlp committed 2 years ago

user can pass dict as attn impl

zucchini-nlp committed 2 years ago

repo consistency

zucchini-nlp committed 2 years ago

update

zucchini-nlp committed 2 years ago

muicgen

zucchini-nlp committed 2 years ago

no prints

zucchini-nlp committed 2 years ago

forgot speech enc-dec and clip

zucchini-nlp committed 2 years ago

how many composite models we have?

zucchini-nlp committed 2 years ago

musicgen meelody is same as mudicgen

zucchini-nlp committed 2 years ago

+siglip

zucchini-nlp committed 2 years ago

fix tests + add some more

zucchini-nlp committed 2 years ago

remove idefics custom overriden code

zucchini-nlp committed 2 years ago

make idefics2 automappable