PR #50717 POC: Experts offload

POC: Experts offload #50717

tarekziade wants to merge 5 commits into main from experts-offload

Add experts_offload: stream MoE experts from disk through a bounded d…

412192f6

tweaks

cc6b2e28

tarekziade marked this pull request as draft 2 days ago

experts_offload: faster reads, bounded prefill memory, MetalConfig fo…

ac255713

experts_offload: commit queued MPS work early; install only finished …

4cacb570

experts_offload: optional host tier of quantized experts (host_cache_…

2dd2dd68

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone