llama.cpp
openvino: OV 2026.2, context-shift, Q5_1 support, gemma4 dense/embedding, and -fa off
#24503
Merged
Merged
openvino: OV 2026.2, context-shift, Q5_1 support, gemma4 dense/embedding, and -fa off #24503
ggerganov
merged
129 commits into
ggml-org:master
from
ravi9:dev_backend_openvino
Add interface is_model_splitted() to check the c-graph is splited or not
4bc1eb14
Infer and propagate dynamic-dimension indices for all tensors in the …
c49ec280
Only do this for fallback sub graph
76eb69e9
Move dynamic dims compute in graph missmatch
7e6caef9
ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split…
d306b0bc
ggml-openvino:add comments
01088c21
ggml-openvino: override VIEW op_case to 0 for split model inputs
126d7586
openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO …
32f9cb75
Enable additional mul_mat tests and add tensor data saving function (…
f812f786
ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimensi…
865f121c
OpenVINO: add NORM/TANH support and rework SOFT_MAX translation
ca3a1764
ggml-openvino: extend VIEW handling
a73a6dc0
Enable -fa off (#118)
bfa4c539
Enable --context-shift
9c922b19
Fix llm param compute error for normal softmax not the softmax in att…
59f0e3c9
OpenVINO backend: fix error for attention size compute in llm param
c8e9ce42
use tensor->extra in infer_request i/o
9f355eda
OpenVINO backend: refacter the compute_llm_params() func add get_atte…
dc5ed75d
OpenVINO backend: clean unused code
e2ce59c5
1to1 match op update (#146)
130ef39c
initial gemma4 support
13ddbf30
removed hardcoded names for kv cache slicing
75977737
OpenVINO backend: Add new attention pattern for llm parameters compute
a1baa1aa
flash attn Q shape static conversion
dad8acda
Remove slice in permute translation when n_seq is 1
760e86dd
return optional in extract_layer_from_name
5a399671
OpenVINO backend: refactor VIEW related operation (#148)
d289bbd9
OpenVINO backend: Add ops l2_norm & pad
e0caf43e
OpenVINO backend does not support CPY with non-contiguous data or mis…
8c830921
add op SSM_CONV GATED_DELTA_NET
a08546fa
OpenVINO backend: fix error for bf16 in OV gpu plugin
0d0fb423
reverted static Q input shape for attention layer
c064e870
OpenVINO backend: remove hardcode name inp_tokens, which ignore some …
d44fa9cd
Disable remote tensor due to bug in ov gpu
6afe6521
Disable n_token > 1 GATED_DELTA_NET on gpu
d2279aef
OpenVINO backend: fix the view op dynamic handling issue in gemma4 & …
e6a7a9e9
OpenVINO backend: clean code
418c5e51
OpenVINO backend: enable view + norm/rms_norm
c6efcb69
OpenVINO backend: concat op
eafd08e9
OpenVINO backend: argsort op
89858ec2
OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS
e25ed8f7
Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops re…
996f0c7f
OpenVINO backend: enable sum_rows
12863b8c
OpenVINO backend: enable clamp
404d6b3c
OpenVINO backend: enable DIV
41c35a34
OpenVINO backend: enable GGML_OP_MUL_MAT_ID
03e835cf
OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed
08438be2
OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed
904c608c
OpenVINO backend: fix issue in mul_mat_id
d2ca0f85
OpenVINO backend: Disable DIV with broadcast on GPU
2aa3b2d5
OpenVINO backend: update DIV
59e3d641
use ov internal op GatedDeltaNet
4472ce00
OpenVINO backend: enable llama erch test qwen3next
4bbb85ff
OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope
3032423e
OpenVINO backend: fix error
c4bd20ff
suggested changes, need review
2c2541ce
suggested changes, need review
d11e198d
OpenVINO backend: clean unused code & fix build warning
c4f2ec74
OpenVINO backend: enable minicpm3 for arch test
46bddb18
Disable GDN op (#177)
bb384836
disable gated_delta_net
645df271
update stateful_kv_size correctly in mismatch case
08b4fd62
OpenVINO backend: enable arch test for qwen3vl
d2c75494
OpenVINO backend: enable cohere2 for arch test
e05da27b
OpenVINO backend: enable t5 for arch test
c3c4dba7
OpenVINO backend: enable jamba for arch test
a32aeb53
OpenVINO backend: remove warning for tmp
a0155c41
OpenVINO backend: enable kimi-linear for arch test
b1f6fb44
Remove unused
603c7dcc
Fix gpt-oss accuracy issue
21bab71d
OpenVINO backend: enable arctic for arch test
f49b0267
OpenVINO backend: enable grok for arch test
65ec35a2
Gemma4 initial npu support (#179)
292b1569
ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control deco…
c8321535
Revert "Gemma4 initial npu support (#179)"
af2a8e17
OpenVINO backend: disable debug log print
a16cfb44
Update TBB discovery. Delegated to OpenVINOs own config.
36c5cd58
OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1
6df01a7e
OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend
2ab4121c
Add raw ov infer profiling metric
6b1c5aa3
Add OV raw infer time metric to static compute path
d1943912
Modify precision of static profiling
f1a53403
update to OV 2026.2, add OV windows CI
88f22fdf
fix editorconfig-checks
ccb1b239
Initiall gemma4 npu support
df50c52e
temp. fix for gemma4 accuracy bug on npu
b397e94b
Remove hardcoded names for npu-fold handling
41ce1c7c
revert static n tokens for cont translation as it is not needed
7baa2133
removed unused variable
5fa8e5e2
test-llama-archs fix
b9cba9d4
Fix gemma4 flash_attn fallback
e8324aca
support im2col
10a2cfdd
fix code style
38e9d59f
disable add_rope_sin_cos optimization
9c0ca74a
stateless boradcast and rope optimizations
bbc43193
Enable manual gqa attn by default for stateless gpu
e3bdd6b8
manual gqa: fixed static batch
bc11c32b
gemma4 llama-bench ctx update fix
d551f5bb
Update OV win CI
2e33f252
stateful rope fusion temp. fix
699fd7d9
removed hardcoded names for kv cache slicing
75977737
OpenVINO backend: Add new attention pattern for llm parameters compute
a1baa1aa
flash attn Q shape static conversion
dad8acda
Remove slice in permute translation when n_seq is 1
760e86dd
OpenVINO backend: refactor VIEW related operation (#148)
d289bbd9
OpenVINO backend: Add ops l2_norm & pad
e0caf43e
OpenVINO backend does not support CPY with non-contiguous data or mis…
8c830921
add op SSM_CONV GATED_DELTA_NET
a08546fa
reverted static Q input shape for attention layer
c064e870
OpenVINO backend: remove hardcode name inp_tokens, which ignore some …
d44fa9cd
Disable remote tensor due to bug in ov gpu
6afe6521
Disable n_token > 1 GATED_DELTA_NET on gpu
d2279aef
OpenVINO backend: fix the view op dynamic handling issue in gemma4 & …
e6a7a9e9
OpenVINO backend: enable view + norm/rms_norm
c6efcb69
OpenVINO backend: concat op
eafd08e9
OpenVINO backend: argsort op
89858ec2
OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS
e25ed8f7
Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops re…
996f0c7f
OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed
904c608c
OpenVINO backend: fix issue in mul_mat_id
d2ca0f85
OpenVINO backend: fix error
c4bd20ff
suggested changes, need review
d11e198d
OpenVINO backend: clean unused code & fix build warning
c4f2ec74
Disable GDN op (#177)
bb384836
OpenVINO backend: enable arch test for qwen3vl
d2c75494
OpenVINO backend: enable cohere2 for arch test
e05da27b
OpenVINO backend: enable t5 for arch test
c3c4dba7
OpenVINO backend: enable jamba for arch test
a32aeb53
OpenVINO backend: remove warning for tmp
a0155c41
OpenVINO backend: enable kimi-linear for arch test
b1f6fb44
OpenVINO backend: enable arctic for arch test
f49b0267
ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control deco…
c8321535
Revert "Gemma4 initial npu support (#179)"
af2a8e17
OpenVINO backend: disable debug log print
a16cfb44
OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1
6df01a7e
Add raw ov infer profiling metric
6b1c5aa3
Add OV raw infer time metric to static compute path
d1943912
Modify precision of static profiling
f1a53403
update to OV 2026.2, add OV windows CI
88f22fdf
fix editorconfig-checks
ccb1b239
Initiall gemma4 npu support
df50c52e
temp. fix for gemma4 accuracy bug on npu
b397e94b
revert static n tokens for cont translation as it is not needed
7baa2133
test-llama-archs fix
b9cba9d4
Fix gemma4 flash_attn fallback
e8324aca
disable add_rope_sin_cos optimization
9c0ca74a
stateless boradcast and rope optimizations
bbc43193
Enable manual gqa attn by default for stateless gpu
e3bdd6b8
manual gqa: fixed static batch
bc11c32b
gemma4 llama-bench ctx update fix
d551f5bb
OpenVINO backend: Conslolidate supported ops
d05ce549
Exclude unsupported GGML_OP_SUB cases
b32c04ed
OpenVINO Backend: MUL_MAT enhancements
71ba1135
support f16 mask input for npu
f7bbe7c5
Make GGML_OPENVINO_* env vars usage uniform
efbc5656
OpenVINO backend: Enhance envvar handling
34b2bee0
more cleanup
b3f21ea8
move ggml_openvino_env_flag to appropriate place
e68a1030
OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-i…
65e2eccb
Merge pull request #208 from mostafafaheem/envvar_cleanup
8ea91dd0
ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data
341b6156
Merge pull request #209 from cavusmustafa/op_translations_q51
4c878bdf
Update openvino.Dockerfile
971816c5
ggml-openvino: centralize env var access via *getenv_str/getenv_int h…
835121df
OpenVINO backend: Enable GGML_OP_ADD_ID
3365e31c
Merge pull request #210 from zhaixuejun1993/xuejun/add_op_add_id
906a48da
Uptade openvino backend clamg-format
dd5c58d1
clang-format
a9045e0e
Update OPENVINO.md (#211)
fb924cbe
Merge branch 'master' into dev_backend_openvino
ba6c06d2
Merge branch 'master' into dev_backend_openvino
1d3035b4
OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision
90ae9176
Merge pull request #214 from zhaixuejun1993/xuejun/fix-error-op-concat
383d163a
Remove strict concurrency for gpu-openvino-low-perf
00e80a99
Update openvino CI keynames; add ccache-clear
65d4041e
Apply suggestions from code review
ce52f0a7
CISC
approved these changes
on 2026-06-16
Fix formatting
3481530d
Login to write a write a comment.
Login via GitHub