PR #2707 Quantization improvements for k_quants

Quantization improvements for k_quants #2707

ikawrakow merged 12 commits into master from ik/better_q234_k

Improve LLaMA-2 2-, 3- and 4-bit quantization

f26f9ef4

Minor 4-bit quantization improvement

77aea721

Some more fine tuning

ec9cb753

Adding make_qkx2_quants

4f8dcb16

Another minor improvement

e9f1340c

Q2_K improvement

1c1f985b

Iterating

404e43cc

Revert Q5_K back to make_qkx1_quants

9f78d4cd

Better Q6_K

e2af308c

make_qkx2_quants is better for Q5_K after all

b7063393

Fix after rebasing on master

35a0b974

Fix for changed tensor names

fdf73db5

ggerganov approved these changes on 2023-08-22

Green-Sky changed the title ~~Quantization imrovements for k_quants~~ Quantization improvements for k_quants 2 years ago

ikawrakow merged bac66994 into master 2 years ago

ikawrakow deleted the ik/better_q234_k branch 2 years ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone