llama.cpp
hexagon: MUL_MAT and MUL_MAT_ID rework : 32x32 tiled weight repack, kernel-params, cached graphs
#24954
Merged
Merged
hexagon: MUL_MAT and MUL_MAT_ID rework : 32x32 tiled weight repack, kernel-params, cached graphs #24954
max-krasnyansky
merged
78 commits into
ggml-org:master
from
qualcomm:hexagon-mm-new-weight-layout
hex-mm: new weight layout and fusion updates
e262741e
hvx-mm: unroll the new tiled vec_dots to optimize hvx register util
fb09b04f
hex-mm: optimize dyn.quant format for q8_0 and q8_1 to reduce overhea…
a4afb7a8
hvx-mm: parallel quantizer per block for large rows
a06b224d
hvx-mm: simplify and futher optimize dyn.quant and vec_dots
935a886e
hvx-mm: keep intermediate per tile accumulators in fp16
18599224
hmx-mm: optimize weight dequant by aligning the repacked tiles with t…
56fcca9e
hmx-mm: remove qweight scratch and just use vtcm_weight
6bf204ff
hmx-mm: remove all unused and obsolete code
e39d260d
hmx-mm: the new tiled repack format is here to stay -- rename all x4x…
56353472
hmx-mm: improve activation processing with dma prefetch
ffa665a8
hex-mm: fix hmx/hvx fallback logic and MUL_MAT_ID allocation (unbreak…
c4c60cca
hex-mm: align the weight tiles with dma just like we did in hmx-mm
5b55c747
hex-mm: factor out common mm bits into htp/matmul-ops.h
e640c957
hex-mm: start moving mm kernel selection to the host
4411fe00
hex-mm: move all of the matmul param compute into the host
b2a67891
hmx-mm: restore pipelined mode
f65150d5
hmx-mm: unroll the dequant functions to optimize register usage
2cbc7195
hmx-mm: further improve activation process
887cef15
hex-mm: use vtcm_seq_alloc for all vtcm allocations and define more c…
05960a94
hex-mm: improve mm optimizer to acount for number of activation threads
7e901392
hex-mm: fix matmul-id kernel params selection (unbreaks OLMoE and LFM)
24f44fbc
hexagon: remove support for arch < v73 since HMX is now required for …
39cdaf38
hex-mm: cleanup naming for consistency
b920cfbc
hex-mm: make sure matmul fusion accounts for vtcm allocation
c1fd6401
hex-mm: minor cleanup for kernel_params definition
35d4e61f
hex-mm: replace hardcoded limits with proper checks for vtcm requirem…
97fcf6ab
hex-mm: add support for non-tiled mm as a fallback option and factor …
c2cf841f
hex-mm: remove unused functions
121f40d6
hex-mm: add shorthand for MM_SELECT in run-tool script
1450d3cf
hvx-mm: factor out hvx/hmx microkernels and unify matmul entry and di…
c45e77dd
hex-mm: further cleanup matmul fallback path
8369d763
hex-mm: refactor matmul entry point and dispatch a bit further
5dc3e65c
hexagon: update cmake build to enable hmx for everything
09cdefb0
hex-ops: optimize kernel_param updates and include summary in the logs
7f63f94e
hex-mm: add support for GGML_HEXAGON_MM_SELECT
891603be
hex-mm: add hex-common header
30184849
hex-mm: pass correct number of tasks to workpool
03a11a00
hex-mm: add proper checks for no-work in dyn.quant tasks
2b6df589
hex-mm: convert all quantizers into a macro
95872146
hex-mm: fix hvx-flat fallback to pass all MUL_MAT tests
33bde7fe
hex-mm: vectorize q8_1 quantizer
a09732c5
hex-mm: improve fused ffn mm stride handling
bff2d14a
hex-mm: consistent use of n_threads and pipeline in kernel_params
7362cd53
hexagon: minor formatting
9a71719b
hex-mm: update MUL_MAT_ID kernel_param handling to make sure host/npu…
3cabc20e
hvx-mm: go back to accumulating in fp32 in tiled hvx kernels, more ac…
7fae23bb
hvx-mm: unroll the loops and remove masking that is not needed for ti…
e187f5ad
hmx-mm: optimize activation processing (slit loops, some unrolling, etc)
1fb65001
hmx-mm: minor optimization for output processing
7d0c55aa
hex-mm: consistent use of uint32_t and size_t in mm kernels
3fd34448
hex-mm: remove legacy restrictions for rows to be multiple of 256
7f6be481
hexagon: replace sprintf with snprintf
aeaad354
hex-mm: relax hardcoded nrows checks and rely on VTCM size requirements
5357dbdc
hexagon: minor alignment fix
56487861
hexagon: fix trailing spaces
abebb9cf
hex-mm: relax padding from 256 to 128 (leftovers)
886dad85
hex-mm: remove redundant checks for weight align to 128
ab1b9e1e
hmx-mm: MUL_MAT_ID better work distribution between hvx threads and h…
50601fde
hex-mm: specialize per-token mmid activation handling
c17fdd9d
hex-profile: update python scripts to handle kernel-params section in…
df6414d5
hex-mm: move n_prefetch (aka dma_depth) into kernel params and remove…
27c73ccc
hex-trace: use easier to parse format, simply and fix post-proc scripts
12db1dad
hmx-mm: relax 32 row limit for output processing which helps utilization
2ace3010
hmx-mm: use start-chunk idx for tracing info
01562663
hmx-mm: parameterize activation dma pipeline
a5da341a
hexagon: add support for simple graph caching to avoid recomputing ke…
fcd3f354
hex-mm: remove left-over repack functions
4ae13d5f
hex-mm: tighten n_prefetch asserts
62031247
hex-mm: remove duplicate round/align_up helper
8c434c8d
hexagon: cleanup common header used in host/npu
7f2eed5a
hexagon: update early wakeup threshold
e951e842
hmx-mm: define cost constants and update solver to assume that repack…
1365b241
hmx-mm: make precompute_matmul a bit more readable (split into smalle…
95df7d25
hex-mm: remove n_threads constraint
9cc6587f
hex-mm: minor formatting updates
203f4ef8
hex-mm: remove obsolete profiling logs
e35b2b93
hex-mm: restore hardcode gate to refuse lm-head to avoid repacking th…
6200365b
lhez
approved these changes
on 2026-06-24
CISC
approved these changes
on 2026-06-24
Login to write a write a comment.
Login via GitHub