onnxruntime

Mlas int4 int8 with avx2/512

#20687

Merged

fire

tile 2x4 SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1542487160 ns 1539062500 ns

use one_16_epi16 and accumulate_2blk_dot: SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1434872720 ns

apply to M1, BQuant layout pack block (subblk) larger than blklen: SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1265060620 ns 1265625000 ns

use new AQuant layout (not work if total M is not RangeCountM): SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 1214042220 ns

apply blksum to blklen32 and 64: SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 784668090 ns; SQNBITGEMM<4>/BlkLen:64/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 754939430 ns

blklen16

impl avx512: SQNBITGEMM<4>/BlkLen:32/M:2048/N:4096/K:4096/Threads:1/Symmetric:1/ComputeType:4/real_time_mean 664029830 ns

matmul_nbit & fix alignment for sgemm

merge main

fix mlas benchmark not using multi threads

profiling

Merge branch 'liqun/mlas-q4-tile-avx' of https://github.com/microsoft/onnxruntime into liqun/mlas-q4-tile-avx

sgemm after sq4bit for avx2

avx512

layout to follow compute, M1 separate with M > 1

make avx512 run

Merge branch 'main' into liqun/mlas-q4-tile-avx

avx512 blklen64 pass

pass avx512 blklen32

pass avx512 blklen 16, 128, 256

pass fp32, refactor sqnbitgemm

merge main

avx512vnni

merge main

avxvnni

rm unused ComputeParallelTasksSGemm

avoid _mm256_dpbusds_avx_epi32 in avx512vnni

fix linux build

Merge branch 'main' into liqun/mlas-q4-tile-avx

refactor for Arm64

more refactor for Arm64

hsum_float_16

hsum_float_16

condition for -mavxvnni

CMAKE_CXX_COMPILER_VERSION VERSION_GREATER 10

missed 2 files from (__GNUC__ > 10)

missed _mm256_dpbusds_avx_epi32 and print out cmake msgs

unused zp, etc.

unused zp, etc.

remove test code changes

remove test code changes

lint

lint

code name

update reviewers' comments

Merge branch 'main' into liqun/mlas-q4-tile-avx