mtmd : add Nemotron 3 Nano Omni support (parakeet) #22520
ngxson
merged
42 commits into
ggml-org:master
from
danbev:nemotron-3-omni-mtmd-audio
mtmd : add Nemotron 3 Nano Omni support (parakeet)
0024aa74
mtmd : generate rel pos tensor in graph instead of in conversion [no ci]
e557406b
ngxson
commented
on 2026-04-30
ngxson
commented
on 2026-04-30
mtmd : add clip_get_model to clip API [no ci]
a9929d42
mtmd : read mel_filters and window into hparams
8e279f49
mtmd : use set_input_f32 lambda [no ci]
ffd1b997
mtmd : add better asserts for mel_filters and hann window [no ci]
8af100fe
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
b5a35e03
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
7ed92942
mtmd : add missing size_t cast
49658ba1
mtmd : change type of pad to size_t
9a8398e3
mtmd : zero initialize samples_padded
6ba52fc5
mtmd : remove unsued ctx member from parakeet preprocessor
385b2d40
mtmd : make log_mel_spectrogram_parakeet_worker_thread private static
cef7ff7c
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
681a199a
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
44cb51f6
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
0cd9e16b
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
78e28f45
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
96b1326a
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
656437b5
Merge branch 'upstream/master' into nemotron-3-omni-mtmd-audio
79e1dba5
mtmd : sync/update parakeeet impl with latest whisper.cpp
1fcd9db5
mtmd : add audio_conv_kernel_size to model conversion
5b741d1f
mtmd : cleanup [no ci]
4f8882b6
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
01a1f581
CISC
approved these changes
on 2026-06-18
conversion : call super().filter_tensors [no ci]
c1d465f9
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
e49c091d
do not discard result of super filter_tensors
3378340f
mtmd : use build_mm instead of ggml_mul_mat
816d776c
mtmd : use build_ffn
882c9b79
mtmd : move and reuse get_vector lambda
79baf6ce
ngxson
commented
on 2026-06-18
mtmd : use build_inp_raw for parakeet
8835abe5
mtmd : throw exception in get_scalar instead of assert
602c2180
mtmd : fix std::min call
2ab3bebc
mtmt : use .c_str in throw clause in get_vector
1337d749
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
dbdd96b1
mtmd : check for F32 type and non-empty tensor in get_vector
81937ca5
Merge remote-tracking branch 'upstream/master' into nemotron-3-omni-m…
b9042a28
mtmd : replace hardcoded 1101 for n_tokens_real
6a1606ff
mtmd : assert subsampling_factor is 8
24bfed80
mtmd : remove unused ggml_tensors attn_pos_w and mm_norm_w
c111a5ff
mtmd : remove single thread path
95084827
fix some security issues
6ebaf0fb
ngxson
approved these changes
on 2026-07-28
Login to write a write a comment.
Login via GitHub