Add Granite Vision Support #11794
ngxson
merged
40 commits into
ggml-org:master
from
alex-jw-brooks:granite_vision
alex-jw-brooks
commented
on 2025-02-10
gabe-l-hart
commented
on 2025-02-12
ggerganov
commented
on 2025-02-20
Add super wip scripts for multimodal granite gguf
7db1c510
Add example for converting mmgranite to gguf
f6fbfc25
remove hardcoded path
8b601070
Add vision feature layer to gguf params
72c53dd8
Clean up llava surgery and remove name substitution hacks
4212f373
Add transformers llava next tensor name mapping
77ce6f2d
Make siglip / openclip mutuall exclusive
3fc67a9f
Fix projector linear substitution
e3c791c6
Fix linear 2 substitution index
2d45e0d0
Increase max flattened gridpoints to 64
3e3eebce
Fix hardcoded concat for multiple feature layers
1ceef1a7
Pull vision feature layers out of gguf keys
c788a455
fix num gridpoints and use all layers
6725d6c2
Avoid dropping last image encoder layer in llava models
403575cc
Use 10 for max number of patches
095b8362
Standardize vision feature layers
ff00515c
Cleanup logs
eceee7fe
Update comment for vision feature layer init
be204f0f
Update notes for alternative to legacy llm conversion script
4588b901
Fix notes rendering
264c2ca2
Add v prefix to vision feature layer log
86b43da3
Use current defaults for feature layer
eb545402
Use constant for max gridpoints / feat layers, style fixes
243a8991
clarify non-negative feature layers
ee2cf62b
Remove CLIP_API from func signature
b5735ba3
USE MAX_IMAGE_FEATURE_LAYERS const in layer calc
3670d0ed
Clarify feature layers are non negative ints and not uint
b973f37b
Fix condition for reading feature layers
82e88527
pop last llava layer when feature layers are unset
ee6fb4de
Fix unset vision layer 0
07e77167
Update examples/llava/clip.cpp
5f69fdbd
Reenable assertion for out of bounds get_rows
ab522d79
Use std vector for gridpoints and feature layers
bb3e03a8
Caculate max feature layer at load time
7bab3057
Include base patch for granite vision allocation
65575237
Fix trailing whitespace
cce01b89
Add max num patches = 10 back for minicpmv
188bfb0e
Use unordered set to store feature layers
86763161
Use max feature layer for postnorm
49c08636
Apply suggestions from code review
bec9ef1f
Reviewers
Assignees
No one assigned
Labels
examples
python
ggml
Milestone
No milestone
Login to write a write a comment.
Login via GitHub