text-generation-inference
[Backend] Add Llamacpp backend
#2975
Merged
Overview
Commits
63
Changes
View On GitHub

[Backend] Add Llamacpp backend #2975

angt merged 63 commits into huggingface:main from angt:llamacpp
angt
angt angt force pushed from 78857ecb to fb744095 1 year ago
mfuntowicz mfuntowicz changed the title Add Llamacpp backend [Backend] Add Llamacpp backend 1 year ago
angt angt force pushed from 76516da7 to 58775423 1 year ago
angt Add llamacpp backend
95e221ee
angt Get rid of llama_batch_get_one()
bd0cc990
angt Use max_batch_total_tokens
3eb4823f
angt Handle max_batch_size
e7facf69
angt Add some input validation checks
a7b4b04c
angt Handle ctx args & fix sampling
8d2dfdf6
angt Add GPU args
f3887479
angt Add --defrag-threshold
e07835c5
angt Add a stupid batch mechanism
d6ded897
angt Cleanup
390f0ec0
angt Add --numa
7a3ed417
angt Fix args
3f199134
angt Enable flash attention by default
ae5bb789
angt Add --offload-kqv
e88a527f
angt Fix batch_pos
f38c34ae
mfuntowicz backend(llama): add CUDA Dockerfile_llamacpp for now
960c12bd
angt Only export the latest logits
161280f3
angt Output real logprobs
2a51e415
angt Fix batching
96434a1e
angt Fix seq iterations
27534d8e
angt Auto-detect n_threads when not provided
c8505fb3
angt Clear request cache after completion
8ed362d0
angt Remove warmup
104a968d
angt Cleanup
ea28332b
mfuntowicz backend(llama): add CUDA architectures build argument for Dockerfile
e6a8d339
angt Add specific args for batch
bfb8e03e
angt Add --type-v & --type-k
38b33e96
angt Bump llamacpp to b4623
207041a9
angt Disable graceful shutdown in debug mode
d883109d
angt Update Dockerfile_llamacpp
df2a4fbb
angt angt force pushed from 58775423 to df2a4fbb 1 year ago
angt Cleanup Dockerfile
906c265a
angt Update Cargo.lock
e0075295
angt Update args
d3a772a8
angt Simplify batching logic
dbee8041
angt Set TGI_LLAMA_PKG_CUDA from CUDA_VERSION
c52f0835
angt Rename bindings
051ff2d5
angt Remove n_ctx
09a745f1
angt Make max_batch_total_tokens optional
5b777877
angt Ensure all samplers are freed on error
695b1292
angt Initialize penalty_last_n with llamacpp default value
0f62401b
angt Cleanup
f22e2fb5
angt Improve default settings
b3e40c4b
angt Add doc
1641c22a
angt Update docs
e4d5fa7e
angt Thanks clippy
fb81c0d1
angt Thanks cargo fmt
2b0d99c1
angt Update docs
8bc10d37
HuggingFaceDocBuilderDev
angt angt requested a review from mfuntowicz mfuntowicz 1 year ago
angt angt requested a review from Hugoch Hugoch 1 year ago
angt angt marked this pull request as ready for review 1 year ago
angt Do not use HOSTNAME env
7bff88bb
angt Bump llama.cpp & cuda
df723e64
angt Fix requirements.txt
5367d94f
angt Fix fmt
809e288b
mfuntowicz
mfuntowicz dismissed these changes on 2025-02-06
angt Enable KQV offload by default
3b1b049b
angt Remove Ngrok tunneling
acca9c3e
angt angt dismissed their stale review via acca9c3e 1 year ago
angt angt force pushed from bc935dc3 to acca9c3e 1 year ago
angt Remove .cargo/config.toml
0d27ee74
angt Fix Dockerfile
4841f71a
angt Add missing cuda prefix
b6cfa0fb
angt Handle custom llama.cpp dir
6bdb644f
angt Cleanup
0702e0bf
angt Add README.md
508d47f8
angt Add HF transfer
14014182
angt Fix bool args
b77d05d3
angt Update doc
d96a7770
angt Update doc
5fb4afbf
angt angt requested a review from mfuntowicz mfuntowicz 1 year ago
mfuntowicz
mfuntowicz approved these changes on 2025-02-14
Hugoch
Hugoch approved these changes on 2025-02-14
angt angt merged cfd4fbb4 into main 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum