text-generation-inference
Inference support for GPTQ (llama + falcon tested) + Quantization script
#438
Merged
Merged
Inference support for GPTQ (llama + falcon tested) + Quantization script #438
OlivierDehaene
merged
19 commits into
main
from
support_gptq
gsaivinay
commented
on 2023-06-09
[WIP] Inference support for GPTQ (llama at least)
9a12941b
Removing dead code.
92f85c96
Fixing the dockerfile (require triton + gcc for compiling).
0b585921
Typo.
da8ebf16
Adding quantization scripts.
5a727153
Functionning quantization script.
a0a194c3
Some fixes.
ae308f88
Re-enabling dim=dim in TensorParallelColumn because llama.
3fb8979a
Neox.
dadbbc27
Fixing few things
ffe8fc46
Fixing register bias + gptq_bits type.
ee1f94e6
Falcon
e5e552b4
Tiny fixes for falcon.
55cf4d25
No one saw that, therefore it didn't happen.
5de68637
Remove lots of dead code, move triton to hard requirement
732da694
Triton is actually a dependency of torch on linux.
054a3d09
Narsil
changed the title
Typo.
983c813f
Santacoder GPTQ support (quantized model seems awful, not sure if it's
16d0fb04
psinger
commented
on 2023-06-20
Apply suggestions from code review
a8aa688a
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub