text-generation-inference
[TENSORRT-LLM] - Implement new looper thread based backend
#2357
Merged
Merged
[TENSORRT-LLM] - Implement new looper thread based backend #2357
Narsil
merged
63 commits into
main
from
trtllm-executor-thread
(backend) use parking_lot crate for RwLock fairness
25b20cba
(launcher) default new server::run parameters to false for now
a3f7d76f
(chore) fmt ... why?
cea64e23
(ffi) use const for GetSamplingConfig
0cd7538a
(server) expose new SchedulingError
169e1f45
(trt)
2a339f99
(build) setup ccache if available
f6f689f5
(ffi) add max_new_tokens parameters
38b5263c
(backend) cleanup a bit
b8a40a0a
(backend) expose PullNewTokens
f4a74be3
(ffi) cleanup again
2883c042
(ffi) add missing headers imports
33c962ef
(ffi) add template specialization to catch and convert to Rust Result…
5f7c0b67
(looper) new looper initial implementation
fb759bdd
(ffi) remove narrowing type warning
0b0c30fe
(ffi) encode the provided user prompt within each request thread
933ab67a
(misc) change scope identifiers
0dca168b
(backend) implement the post_processor background thread
c2e21d87
(misc) missing Result types for Rust
7bebc629
use blocking_recv in looper to consume awaiting_requests at max befor…
291eaa99
(server) forward auth_token to server::run
089c5fe6
(build) fetchcontent use archives instead of git
dddc9a44
(ffi) fix usage of wrong vector constructor making a capacity fill call
8e648ce4
(ffi) missing namespace for tle::Response
3d0e90b6
(ffi) do not use reference capture in lambda as we are not capturing …
483f1729
(backend) refactor & cleanup
b1846fb4
(Dockerfile.trtllm) delete for now
0f50539b
(misc) simplify [make_]move_iterator by using c++20 type inference
b41875c1
(misc) no need to move for uint32_t items
42ccf4e7
(scheduler) rework submit/pull logic
fa63db0d
(post) impl postprocessing
984ae979
(misc) delete backend.rs
b242f45c
(misc) rerun-if-changed all the cmake modules
507ff666
(misc) move to latest trtllm
213acc6e
(fix): HOPPER_SM_MAJOR is 9 not 8
544c9d9d
(misc: build for sm_{75,80,86,89,90} by default
188e4dc6
(misc): build with trtllm 0.13.0
ce0cd1fc
(misc): increase verbosity of spdlog
eb13d8d1
(fix): do not recreate the stateful hashmap at every it
c8a99af6
(misc): update dependency in trtllm dockerfile
cb69c9a9
(misc): update dependency in trtllm dockerfile
437c2aa1
(misc): disable logging in release mode
0c3ba932
(misc): improve trtllm download script robustness
f9f10a66
(fix): ore fixes for Dockerfile
dd94ccc9
misc(cuda): require 12.6
819c9537
chore(cmake): use correct policy for download_timestamp
f20ec288
feat(looper): check engine and executorWorker paths exist before crea…
629153b4
chore(cmake): download timestamp should be before URL
027756c5
feat(looper): minor optimizations to avoid growing too much the conta…
6687c06a
chore(trtllm): move dockerfile to right place
62f33d7e
chore(trtllm): disable tokenizer parallelism by default
e3bce407
chore(trtllm): fmt
85c03e33
chore(trtllm): post-rebase commit
fb00f985
chore(trtllm): remove unused method
31747163
feat(trtllm): cache maxNumTokens to avoid calling JSON everytime
e6da2124
misc(router): remove SchedulingError
1a3da05f
feat(trtllm): do not tokenize twice
8d1c3c8a
Revert "chore(trtllm): remove unused method"
f5b9ee36
chore(rebase): fix invalid references
d73401ac
chore(router): add python dependency
18b473b0
Narsil
dismissed these changes
on 2024-10-25
Merge branch 'main' into trtllm-executor-thread
01b82b58
Lint.
b4b6322e
Fix bad rebase
4463856c
Narsil
approved these changes
on 2024-10-25
Login to write a write a comment.
Login via GitHub