llama.cpp
mtmd: add Gemma 4 audio conformer encoder support
#21421

Merged

mtmd: add Gemma 4 audio conformer encoder support #21421

ngxson merged 19 commits into ggml-org:master from stephencox-ict:gemma4-audio-pr

github-actions added documentation

github-actions added examples

stephencox-ict commented on 2026-04-04

stephencox-ict commented on 2026-04-04

ngxson commented on 2026-04-04

stephencox-ict commented on 2026-04-04

stephencox-ict force pushed from 6bf9d4a2 to 9729486d 10 days ago

github-actions added testing

stephencox-ict force pushed from 9729486d to fd432f6d 10 days ago

stephencox-ict force pushed from fd432f6d to 83d1f37f 10 days ago

stephencox-ict force pushed from 83d1f37f to 13e9f5ee 10 days ago

ngxson commented on 2026-04-04

stephencox-ict force pushed from 13e9f5ee to 8c10029d 10 days ago

stephencox-ict force pushed from 8c10029d to 29dd32e4 10 days ago

stephencox-ict force pushed from 29dd32e4 to 7435a596 10 days ago

stephencox-ict marked this pull request as ready for review 10 days ago

stephencox-ict requested a review 10 days ago

stephencox-ict requested a review from

JohannesGaessler 10 days ago

stephencox-ict requested a review from

ngxson 10 days ago

JohannesGaessler commented on 2026-04-05

stephencox-ict force pushed from 63aaa5a4 to 9a5b23ae 10 days ago

stephencox-ict force pushed from 9a5b23ae to 1cbecb47 10 days ago

ngxson requested changes on 2026-04-05

stephencox-ict force pushed from 1cbecb47 to 76b7281e 10 days ago

stephencox-ict force pushed from 76b7281e to ef3e5b29 10 days ago

stephencox-ict requested a review from

ngxson 10 days ago

stephencox-ict requested a review from

JohannesGaessler 10 days ago

ngxson commented on 2026-04-05

stephencox-ict force pushed from ef3e5b29 to 81b02024 9 days ago

stephencox-ict force pushed from 81b02024 to f3b827d9 9 days ago

stephencox-ict force pushed from f3b827d9 to 6e971d68 9 days ago

stephencox-ict requested a review 9 days ago

github-actions added Nvidia GPU

github-actions added ggml

stephencox-ict force pushed from 6e971d68 to 883dfbcc 9 days ago

stephencox-ict requested a review from

ngxson 9 days ago

stephencox-ict force pushed from 883dfbcc to f0484a75 9 days ago

stephencox-ict force pushed from f0484a75 to 8a1494c4 9 days ago

stephencox-ict force pushed from 8a1494c4 to 2852c7c7 8 days ago

stephencox-ict requested a review from

CISC 8 days ago

github-actions added model

github-actions added python

stephencox-ict marked this pull request as draft 8 days ago

ORippler commented on 2026-04-07

stephencox-ict force pushed from e81b1f0f to e6bfa64d 7 days ago

stephencox-ict requested a review from

ORippler 7 days ago

stephencox-ict marked this pull request as ready for review 7 days ago

ngxson requested changes on 2026-04-08

stephencox-ict marked this pull request as draft 7 days ago

stephencox-ict force pushed from 6b820a21 to 9e181e95 7 days ago

ngxson commented on 2026-04-08

stephencox-ict force pushed from 9ffd33fb to 896d3e81 6 days ago

stephencox-ict marked this pull request as ready for review 6 days ago

mtmd: add Gemma 4 audio conformer encoder support

05e72a1c

gemma4: fix audio encoder and LM precision issues

2d0961eb

mtmd: use double-precision math for audio preprocessing constants

6bfeb87d

Revert "mtmd: use double-precision math for audio preprocessing const…

56c8304e

gguf-py: restore gemma3n mappings in tensor_mapping.py and fix swappe…

59e4872b

address ngxson review: fix tensor mapping in C++, remove dup comment,…

df3ec225

address review: remove cross-load, keep per-layer scale, derive softcap

15f2272f

address review: auto-detect swapped conv norms, remove dup comment

1cd09244

address review: simplify conv norm swap, move scaling to PR #21625

a67decf1

mtmd: fix CUDA/Vulkan conformer encoder by making sigmoid input conti…

62986e69

revert tensor_mapping.py, gemma4-iswa.cpp and llama-model.cpp changes

e3c68548

gemma4: restore BF16-rounded scales and per-layer multimodal scaling

282a6690

stephencox-ict force pushed from 9b5efb8e to 282a6690 5 days ago

mtmd: fix conv norm swap comment in gemma4 audio conformer

90968593

stephencox-ict requested a review from

ngxson 5 days ago

ngxson commented on 2026-04-11

revert gemma4-iswa.cpp changes per reviewer request

4107adf4

mtmd: swap conv_norm/norm_conv at load site instead of post-load

558399ce

mtmd: replace ggml_roll with view+concat for Metal compatibility

1389eea2

tests: fix unused parameter warning in test-llama-archs save_models

bf3beed8

ngxson commented on 2026-04-11

tests: remove non-gemma4 skip list changes from test-llama-archs

23fd8fcb

Revert "mtmd: replace ggml_roll with view+concat for Metal compatibil…

af69fccb

ngxson approved these changes on 2026-04-11

pwilkin approved these changes on 2026-04-11

ngxson merged 547765a9 into master 3 days ago

Reviewers

pwilkin

ngxson

ORippler

JohannesGaessler

stephencox

CISC

Assignees

No one assigned

Labels

documentation model testing Nvidia GPU examples python ggml

Milestone

No milestone

llama.cpp mtmd: add Gemma 4 audio conformer encoder support #21421 Merged

mtmd: add Gemma 4 audio conformer encoder support #21421

llama.cpp
mtmd: add Gemma 4 audio conformer encoder support
#21421

Merged