Granite Vision Support (#35579)
* Add multimodal granite support
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Support multiple image feature layres
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
* Remove failing validation for visual encoders with no cls
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
* Update llava based models / configs to support list of feature layers
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
* Add tests for multiple feature layers
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
* Use conditional instead of except for misaligned feature shapes
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
* crop cls from each hidden state
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
* Fix formatting
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
* Support single vision feature int in vipllava
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
* Fix typo in vision feature selection strategy validation
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
* Add tentative integration test for granite vision models
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
* Add granite vision docs
Replace multimodal granite refs with granite vision
Add granite vision / llava next alias
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
* Use image url in granitevision example
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
---------
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
