llama.cpp
mtmd: Add DeepSeekOCR Support
#17400
Merged
Overview
Commits
143
Changes
View On GitHub

mtmd: Add DeepSeekOCR Support #17400

ngxson merged 143 commits into ggml-org:master from sfallah:sf/deepseek-ocr
sfallah
sfallah mtmd: llama.cpp DeepSeekOCR support
43a130b4
sfallah loading sam tensors
b6b9f02c
bluebread mtmd: fix vision model processing
85c7cda8
sfallah Merge pull request #1 from bluebread/sf/deepseek-ocr
578c8d77
sfallah deepseek-ocr clip-vit model impl
2aab52e2
bluebread mtmd: add DeepSeek-OCR LM support with standard attention
eab28ed3
bluebread mtmd: successfully runs DeepSeek-OCR LM in llama-cli
76305878
bluebread mtmd: Fix RoPE type for DeepSeek-OCR LM.
2de34367
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
e8b26102
sfallah loading LM
97e0907c
sfallah Merge branch 'sf/deepseek-ocr' into sf/deepseek-ocr
13dc6fb3
sfallah Merge pull request #2 from bluebread/sf/deepseek-ocr
b32bb5e7
sfallah sam warmup working
790bbb97
sfallah sam erroneous return corrected
cec9a5c6
sfallah clip-vit: corrected cls_embd concat
8b3d319c
sfallah clip-vit: model convert qkv_proj split
1e081571
sfallah corrected combining of image encoders' results
331cea8f
bluebread fix: update callback for ffn_moe_weighted and add callback for attn_o…
6c0715be
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
a65ddf5b
sfallah concat image_newline and image_seperator tokens
63a042f2
sfallah visual_model warmup (technically) works
89afda8d
sfallah window partitioning using standard ggml ops
88032f46
sfallah sfallah requested a review from ngxson ngxson 348 days ago
sfallah sfallah requested a review from CISC CISC 348 days ago
sfallah sfallah requested a review from ggerganov ggerganov 348 days ago
github-actions github-actions added model
github-actions github-actions added examples
github-actions github-actions added python
sfallah sfallah marked this pull request as draft 348 days ago
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
1268dc3f
sfallah sam implementation without using CPU only ops
68b206b6
bluebread clip: fixed warnings
8bce66d5
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
5e6cf3c6
bluebread mtmd: fix get_rel_pos
7e9fbecc
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
0f5587dc
bluebread mtmd: fixed the wrong scaler for get_rel_pos
7b8d735c
sfallah image encoding technically works but the output can't be checked sing…
86f111f8
sfallah
bluebread mtmd: minor changed
effe6695
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
f8f66a15
sfallah Merge pull request #3 from bluebread/sf/deepseek-ocr
3fcfc3ac
bluebread mtmd: add native resolution support
ee8a1488
sfallah - image encoding debugged
4cfa15fc
ngxson
bluebread mtmd: correct token order
3f711883
sfallah Merge pull request #5 from bluebread/dsocr-debug
a594990f
sfallah Merge branch 'sf/deepseek-ocr' into sf/deepseek-ocr
6dfda99c
sfallah Merge pull request #4 from bluebread/sf/deepseek-ocr
7941f5d8
sfallah - dynamic resizing
206f8abc
bluebread mtmd: quick fix token order
40e7e6e7
bluebread mtmd: fix danling pointer
81533e49
sfallah Merge pull request #6 from bluebread/sf/deepseek-ocr
88109404
bluebread mtmd: SAM numerically works
a488b495
bluebread mtmd: debug CLIP-L (vit_pre_ln)
ccb2f238
bluebread mtmd: debug CLIP-L & first working DeepSeek-OCR model
841a4a88
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
ed3b7f10
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
55430945
bluebread mtmd : add --dsocr-mode CLI argument for DeepSeek-OCR resolution cont…
c5f4c64f
bluebread mtmd: simplify SAM patch embedding
95239f92
sfallah Merge pull request #7 from bluebread/sf/deepseek-ocr
6b0e7cd1
github-actions github-actions added Nvidia GPU
github-actions github-actions added ggml
sfallah Merge branch 'master' into sf/deepseek-ocr
66341666
ngxson
ngxson commented on 2025-12-02
bluebread mtmd: adapt Pillow image resizing function
c914e054
bluebread mtmd: simplify DeepSeek-OCR dynamic resolution preprocessing
e20857ba
bluebread Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
43dfc0c8
bluebread
bluebread commented on 2025-12-03
bluebread mtmd: remove --dsocr-mode argument
b696c547
bluebread mtmd: refactor code & remove unused helper functions
b26b507c
bluebread
sfallah sfallah marked this pull request as ready for review 335 days ago
sfallah
CISC
CISC commented on 2025-12-04
ngxson
ngxson requested changes on 2025-12-04
ngxson
bluebread
sfallah
bluebread mtmd: fix tensor names for image newlines and view separator
7451b841
ngxson
sfallah
sfallah clean up
386ba479
sfallah Merge branch 'sf/deepseek-ocr' into sf/deepseek-ocr-cleanup
c73748ab
ngxson
sfallah reverting automatically removed spaces
a661c529
sfallah reverting automatically removed spaces
0399ddf1
bluebread mtmd: fixed bad ocr check in Deepseek2 (LM)
c89171cf
bluebread Merge branch 'sf/deepseek-ocr-cleanup' of github.com:sfallah/llama.cp…
2dd99240
bluebread mtmd: support combined QKV projection in buid_vit
fc3f625f
sfallah Merge pull request #8 from sfallah/sf/deepseek-ocr-cleanup
4d7d9945
sfallah using common build_attn in sam
5381b9cf
sfallah corrected code-branch when flash-attn disabled
076138a4
bluebread mtmd: minor fix
d0c08e36
sfallah minor formatting and style
f5bd310a
sfallah Merge pull request #9 from sfallah/sf/deepseek-ocr-attn
6687b4e7
CISC
CISC commented on 2025-12-05
sfallah Merge branch 'ggml-org:master' into sf/deepseek-ocr
5f2ee1ae
sfallah fixed flake8 lint issues
1c88647e
sfallah minor editorconfig-check fixes
d981f19e
sfallah minor editorconfig-check fixes
705394c2
bluebread
bluebread commented on 2025-12-05
bluebread
bluebread commented on 2025-12-05
bluebread
bluebread commented on 2025-12-05
bluebread mtmd: simplify get_rel_pos
15f2ada0
bluebread mtmd: make sam hparams configurable
2d918b3e
bluebread mtmd: add detailed comments for resize_bicubic_pillow
5dfcc5ab
bluebread mtmd: fixed wrong input setting
53273f83
bluebread mtmd: convert model in FP16
48c6cf21
bluebread
bluebread
rhjdvsgsgks
rhjdvsgsgks commented on 2025-12-08
bluebread mtmd: minor fix
5174a1e6
gitl33
ngxson
ngxson
ngxson commented on 2025-12-09
ngxson
ngxson
bluebread
ngxson
sfallah
ngxson
bluebread mtmd: remove tweak to llama-mtmd-cli & deepseek-ocr template
01614069
bluebread
sfallah
sfallah fix: test-1.jpg ORC issue with small (640) resolution
ed944cd2
ngxson
sfallah minor: editconfig-check fix
aaf2fd17
sfallah Merge branch 'master' into sf/deepseek-ocr-merge-test
33fabf0b
sfallah merge with changes from https://github.com/ggml-org/llama.cpp/pull/17909
d70f171f
sfallah minor: editconfig-check fix
4cbbe8ab
sfallah testing deepseek-ocr
47f0fee6
ngxson
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr-me…
e0e69fd3
sfallah quick and (potential) dirty merge with https://github.com/ggml-org/ll…
f95a6fe9
sfallah refactoring, one single builder function and static helpers
f7736f23
sfallah added deepseek-ocr test to tests.sh
fb3bb6aa
sfallah Merge pull request #11 from sfallah/sf/deepseek-ocr-merge_#17965
1b38ccf6
sfallah
sfallah minor formatting fixes
6c36c038
sfallah check with fixed expected resutls
dc2066e5
sfallah Merge pull request #10 from sfallah/sf/deepseek-ocr-test-script
3fc61d48
sfallah minor formatting
7f8621c5
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
b3bf8cba
sfallah editorconfig-check fix
8ad98ee6
sfallah sfallah requested a review from ngxson ngxson 322 days ago
sfallah sfallah requested a review from CISC CISC 322 days ago
sfallah Merge branch 'ggml-org:master' into sf/deepseek-ocr
4a4f8296
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
51c3de68
sfallah merge with changes from https://github.com/ggml-org/llama.cpp/pull/18042
512b2c8f
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
00d23570
sfallah minor
87e4a00c
CISC
CISC commented on 2025-12-16
bluebread convert: minor fix
f629d02e
CISC
CISC commented on 2025-12-17
bluebread mtmd: format code
5a741fda
bluebread convert: quick fix
616f009e
bluebread convert: quick fix
e5d426be
sfallah minor python formatting
c739cf20
n00mkrad
bluebread
bluebread
sfallah Merge branch 'master' into sf/deepseek-ocr
9a05e1d1
sfallah fixed merge build issue
4d91711e
ngxson
ngxson
bluebread
sfallah
bluebread
sfallah
ngxson ngxson assigned ngxson ngxson 316 days ago
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
ded92076
sfallah merge resolved
a94c2417
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
6978c37f
aviallon
sfallah
sfallah minor fix
05789f56
gitl33
sfallah Merge branch 'ggml-org:master' into sf/deepseek-ocr
7e47aa88
sfallah Merge branch 'ggml-org:master' into sf/deepseek-ocr
7ffa23c2
aviallon
sfallah
ngxson
sfallah
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
f41d3239
sfallah minor
9b1a1b91
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
52fcb139
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
0031b41e
sfallah sfallah requested a review from CISC CISC 258 days ago
CISC
CISC commented on 2026-02-18
sfallah Update convert_hf_to_gguf.py
5f2283bb
CISC
CISC approved these changes on 2026-02-18
ngxson
ngxson commented on 2026-02-18
ngxson
ngxson commented on 2026-02-18
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
7856e24c
sfallah - removed clip_is_deepseekocr
50c1e15a
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
3e221cf7
sfallah - cleaning commented out code
e037b956
sfallah
sfallah sfallah requested a review from ngxson ngxson 256 days ago
sambartik
sfallah
sambartik
sfallah
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
0b61c6ae
sfallah fixing instabilities issues reintroducing resize_bicubic_pillow
7a53e7e9
sambartik
luckydevil13
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
c2e6701e
sfallah - use f16 model for deepseek-ocr test
49f3ca55
github-actions github-actions added testing
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
21243f3d
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
a493dc15
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
754061e4
sfallah sfallah requested a review 231 days ago
sfallah sfallah requested a review from JohannesGaessler JohannesGaessler 231 days ago
sfallah sfallah requested a review 231 days ago
sfallah Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
77253998
ngxson
sfallah
ngxson
ngxson commented on 2026-03-25
ngxson rename fc_w --> mm_fc_w
3754c324
ngxson Merge branch 'master' into sf/deepseek-ocr
d88b88e4
ngxson add links to OCR discussion
0ea5fa45
ngxson cleaner loading code
edf020df
ngxson add missing .weight to some tensors
80998695
ngxson add default jinja template (to be used by server)
1d900949
ngxson move test model to ggml-org
6faf264d
sfallah rolling back upscale change
8dabfe3a
ngxson
ggerganov
ggerganov approved these changes on 2026-03-25
github-actions github-actions added documentation
ngxson
ngxson approved these changes on 2026-03-25
CISC
CISC commented on 2026-03-25
ngxson Update convert_hf_to_gguf.py
95cc5665
ngxson ngxson requested a review from CISC CISC 223 days ago
ngxson ngxson requested a review from ngxson ngxson 223 days ago
ngxson ngxson requested a review from ggerganov ggerganov 223 days ago
CISC
CISC approved these changes on 2026-03-25
ngxson
CISC
ngxson ngxson removed review request 223 days ago
ngxson ngxson removed review request from JohannesGaessler JohannesGaessler 223 days ago
ggerganov
ggerganov approved these changes on 2026-03-25
ngxson ngxson merged a970515b into master 223 days ago
urumchi
sfallah

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
Labels
documentation model testing Nvidia GPU examples python ggml
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum