PR #3231 Trtllm backend improvements

feat(trtllm): add new finish reasons

leejuyuu committed 71 days ago

fix: fix prometheus_port CLI short arg conflict

leejuyuu committed 71 days ago

fix(trtllm): fix segfault when canceling request

leejuyuu committed 71 days ago

feat(trtllm): add stop sequence support

leejuyuu committed 71 days ago

feat(trtllm): catch broader exception

leejuyuu committed 71 days ago

feat(trtllm): check existence of config files

leejuyuu committed 71 days ago

fix(trtllm): fix do_sample being ignored

leejuyuu committed 71 days ago

feat(trtllm): get more accurate start time

leejuyuu committed 71 days ago

perf(trtllm): reduce futile loop iterations

leejuyuu committed 71 days ago

refactor: add interior mutability to tensorrt_llm_backend_t

leejuyuu committed 71 days ago

feat(trtllm): separate request and response loop

leejuyuu committed 71 days ago

fix(trtllm): handle single eos_token_id in generation_config

leejuyuu committed 71 days ago

feat(trtllm): support guided decoding

leejuyuu committed 71 days ago

text-generation-inference Trtllm backend improvements #3231 Open