PR #22851 Add preemptive priority scheduling

Add preemptive priority scheduling #22851

lowlyocean wants to merge 10 commits into ggml-org:master from lowlyocean:unload_priority

Remove sleeping before adding priority scheduling

0d99e064

Add preemptive priority scheduling to router mode

61da93e0

Fix new methods declared inside unload_lru() instead of as standalone…

f481f684

Fix bad_function_call bug

087eb36a

OK- priority is respected

653a0b31

Add --models-idle-timeout to replace old --sleep-idle-seconds flag

9349beb1

lowlyocean changed the title ~~Feature Request: [Router mode] Preemptive Priority scheduling~~ Add preemptive priority scheduling 40 days ago

github-actions added server/webui

github-actions added examples

github-actions added python

github-actions added server

Do not reset idle timer for /metrics, /models, /props, /health

68ad4509

Cleanup logs and documentation

a359e8eb

Merge branch 'master' into unload_priority

f401f4de

lowlyocean marked this pull request as ready for review 37 days ago

lowlyocean requested a review 37 days ago

Merge branch 'master' into unload_priority

46f63a22

ngxson closed this 36 days ago

Reviewers

No reviews

Assignees

No one assigned

Labels

server/webui examples python server

Milestone

No milestone