vllm
[Frontend] Set server's maximum number of generated tokens using generation_config.json
#12242
Merged
Overview
Commits
34
Changes
View On GitHub

[Frontend] Set server's maximum number of generated tokens using generation_config.json #12242

youkaichao merged 34 commits into vllm-project:main from main
mhendrey
github-actions
mergify mergify added frontend
DarkLight1337 DarkLight1337 changed the title Enable setting server's maximum number of generated tokens using generation_config.json [Frontend] Set server's maximum number of generated tokens using generation_config.json 333 days ago
DarkLight1337
mhendrey mhendrey requested a review from DarkLight1337 DarkLight1337 331 days ago
mhendrey mhendrey requested a review from robertgshaw2-redhat robertgshaw2-redhat 331 days ago
mhendrey mhendrey requested a review from simon-mo simon-mo 331 days ago
mhendrey Adding max_new_tokens support to generation_config.json
5c85448d
mhendrey Changed default_max_tokens to server_max_tokens
4ad6b45c
mhendrey Renamed default_max_tokens to server_max_tokens
95f9c973
mhendrey Removed the float("inf") bug
4786e563
mhendrey Renamed default_max_tokens to server_max_tokens
4980a73f
mhendrey Rearranged lines to make the changes with existing as small as possible
39d7d767
mhendrey Limit generated tokens by server's max_tokens setting when available
b6a24c47
mhendrey Changed syntax to pass format.sh tests
aa7cff13
ShangmingCai [Bugfix] Fix num_heads value for simple connector when tp enabled (#1…
2f6e43be
youkaichao [torch.compile] fix sym_tensor_indices (#12191)
6baa0ea5
hmellor Move linting to `pre-commit` (#11975)
35b59487
terrytangyuan [DOC] Fix typo in docstring and assert message (#12194)
0c2f332e
terrytangyuan [DOC] Add missing docstring in LLMEngine.add_request() (#12195)
46249e5f
terrytangyuan [Bugfix] Fix incorrect types in LayerwiseProfileResults (#12196)
0b2e3de3
Isotr0py [Model] Add Qwen2 PRM model support (#12202)
090eca3c
DarkLight1337 [Core] Interface for accessing model from `VllmRunner` (#10353)
5d36c1fd
youkaichao [misc] add placeholder format.sh (#12206)
df331a75
DarkLight1337 [CI/Build] Remove dummy CI steps (#12208)
881964d0
DarkLight1337 [CI/Build] Make pre-commit faster (#12212)
5cc6a09f
DarkLight1337 [Model] Upgrade Aria to transformers 4.48 (#12203)
9f3d5a68
youkaichao [misc] print a message to suggest how to bypass commit hooks (#12217)
957ca23c
youkaichao [core][bugfix] configure env var during import vllm (#12209)
399d224c
heheda12345 [V1] Remove `_get_cache_block_size` (#12214)
df065037
wangxiyuan [Misc] Pass `attention` to impl backend (#12218)
b89529bf
DarkLight1337 [Bugfix] Fix `HfExampleModels.find_hf_info` (#12223)
a5d57f1e
heheda12345 [CI] Pass local python version explicitly to pre-commit mypy.sh (#12224)
b1af379f
mhendrey Added tests to check max_tokens is properly set
0e3a719f
mhendrey mhendrey requested a review from mgoin mgoin 331 days ago
mhendrey mhendrey requested a review from ywang96 ywang96 331 days ago
mhendrey mhendrey requested a review from WoosukKwon WoosukKwon 331 days ago
mhendrey mhendrey requested a review from njhill njhill 331 days ago
mhendrey mhendrey requested a review from comaniac comaniac 331 days ago
mhendrey mhendrey requested a review from alexm-redhat alexm-redhat 331 days ago
mhendrey mhendrey requested a review from zhuohan123 zhuohan123 331 days ago
mhendrey mhendrey requested a review from youkaichao youkaichao 331 days ago
mergify mergify added documentation
mergify mergify added ci/build
mergify
mergify mergify added needs-rebase
mhendrey
DarkLight1337
DarkLight1337
mhendrey
mhendrey mhendrey closed this 331 days ago
mhendrey Merge branch 'server_max_tokens'
6867b374
mhendrey Mucked up the rebasing. Fixing that now.
99243cf6
mhendrey
mhendrey mhendrey reopened this 331 days ago
mergify mergify removed needs-rebase
DarkLight1337 DarkLight1337 removed review request from mgoin mgoin 331 days ago
DarkLight1337 DarkLight1337 removed review request from comaniac comaniac 331 days ago
DarkLight1337 DarkLight1337 removed review request from njhill njhill 331 days ago
DarkLight1337 DarkLight1337 removed review request from zhuohan123 zhuohan123 331 days ago
DarkLight1337 DarkLight1337 removed review request from youkaichao youkaichao 331 days ago
DarkLight1337 DarkLight1337 removed review request from WoosukKwon WoosukKwon 331 days ago
DarkLight1337 DarkLight1337 removed review request from alexm-redhat alexm-redhat 331 days ago
DarkLight1337 DarkLight1337 removed review request from ywang96 ywang96 331 days ago
DarkLight1337
DarkLight1337 commented on 2025-01-23
mhendrey Reverting the serving_chat & serving_completion back and putting all …
1a15431a
mhendrey Didn't quite revert back. Deleting empty line from both
c10eb1f3
DarkLight1337
DarkLight1337 commented on 2025-01-23
mhendrey Changed to using one-liner and edited engine arg for generation-config
a3fc62b4
mhendrey Merge branch 'vllm-project:main' into main
98949f68
mhendrey Converted to a one-liner for taking minimum value & added to generati…
c71f429d
DarkLight1337
DarkLight1337 approved these changes on 2025-01-25
DarkLight1337 DarkLight1337 enabled auto-merge (squash) 329 days ago
github-actions github-actions added ready
mhendrey
DarkLight1337
disabled auto-merge 328 days ago
Manually disabled by user
youkaichao youkaichao merged 9ddc3522 into main 328 days ago
JGSweets
DarkLight1337
JGSweets
DarkLight1337

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
documentation frontend ready ci/build
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum