onnxruntime
Update MoE offload implementation plan and implements a global state for MoE/QMoE
#32738

Merged

Update MoE offload implementation plan and implements a global state for MoE/QMoE #32738

xadupre merged 38 commits into main from docs/moe-offload-step-5

Update MoE offload implementation plan

80219371

xadupre requested a review from

copilot-pull-request-reviewer 17 days ago

Focus MoE plan on selected strategy

0f3ee97e

copilot-pull-request-reviewer commented on 2026-09-22

Define PR deliverables for MoE offload plan

aed8af9e

Specify MoE offload strategy in integration PR scope

a91e27d7

List four MoE offload policy session options

8a92ce6b

Separate session-global MoE counting from offload integration

c5db52ab

xadupre changed the title ~~Update MoE offload implementation plan~~ [DRAFT] Update MoE offload implementation plan and implements a global state for MoE/QMoE 17 days ago

Add session-global MoE expert state and invocation counting

b891acfc

Fix output-name span in MoE counter integration tests

4d7d6ebc

Apply global MoE expert counter policy

0d231ec0

Index MoE counters by kernel and expert at initialization

417ef7f1

Encapsulate MoE expert usage and gate optional diagnostics

c62002d1

Overlap MoE expert usage counting with CUDA computation

ca9cacaa

xadupre requested a review from

copilot-pull-request-reviewer 16 days ago

copilot-pull-request-reviewer commented on 2026-09-23

Preserve existing OpKernelContext virtual method slots

7bea5b39

Fix MoE counting minimal-build size and CUDA plugin tests

ab4ea614

Simplify MoE counters with bounded coefficients

bccb9c9a

xadupre requested a review from

copilot-pull-request-reviewer 16 days ago

copilot-pull-request-reviewer commented on 2026-09-23

Record expert usage directly in MoeExpertState

834ba457

Share expert collection through framework KernelUsage

6a17e1cb

Expose session-owned KernelUsage through kernel contexts

39409e6c

Rename CUDA usage adapter to CudaRoutingSnapshot

199d32ec

Add MoeExpertState::GetExpertStats() flat accessor

e9702672

Replace KernelUsage with generic KernelPilot + MoeExpertSelection

fe5c003f

Remove NodeCounters/GetSnapshot; SessionState exposes GetKernelPilot …

fbe2ac56

Simplify MoeExpertSelection: drop membership mask, keep only expert list

7d9f0f40

Remove NodeInfo; nodes_ now maps graph identity to the kernel pointer

9bfd1058

Free nodes_ once FinalizeInitialization() runs; it is Load()-only state

7ffcc525

Stop storing nodes_; build the graph-identity index locally inside Lo…

1383d336

Remove redundant MoE node type storage

5a080660

Refactor MoE kernel pilot state

7731b4a9

github-actions commented on 2026-09-24

Move MoE logging into kernel pilot state

1225e9ca

github-actions commented on 2026-09-24

Log MoE counter updates from kernel pilot state

9394e1c5

Clarify MoE redistribution synchronization

e0e53030

xadupre requested a review from

copilot-pull-request-reviewer 15 days ago

copilot-pull-request-reviewer commented on 2026-09-24

Address MoE state review findings

97a5eee5

xadupre requested a review from

copilot-pull-request-reviewer 15 days ago

copilot-pull-request-reviewer commented on 2026-09-24

Keep MoE counter commits lock-free

e4aad2f7

xadupre requested a review from

copilot-pull-request-reviewer 15 days ago

copilot-pull-request-reviewer commented on 2026-09-24

xadupre requested a review from

copilot-pull-request-reviewer 15 days ago

Migrate QMoE tools to counter logs

8f36fc6f

copilot-pull-request-reviewer commented on 2026-09-24

xadupre requested a review from

copilot-pull-request-reviewer 15 days ago

copilot-pull-request-reviewer commented on 2026-09-24

Fix minimal KernelPilot source exclusion

84c7712d

xadupre requested a review from

copilot-pull-request-reviewer 15 days ago

xadupre changed the title ~~[DRAFT] Update MoE offload implementation plan and implements a global state for MoE/QMoE~~ Update MoE offload implementation plan and implements a global state for MoE/QMoE 15 days ago

copilot-pull-request-reviewer commented on 2026-09-24

xadupre requested a review from

titaiwangms 15 days ago

Move usage commit into KernelPilot

77ce9cba

xadupre requested a review from

copilot-pull-request-reviewer 14 days ago

copilot-pull-request-reviewer commented on 2026-09-25

Bound MoE counter logs and clarify collector lifecycle

c4b3fc5e

xadupre requested a review from

copilot-pull-request-reviewer 14 days ago

copilot-pull-request-reviewer commented on 2026-09-25

Exclude CUDA MoE collector sources from minimal builds

81684882

xadupre requested a review from

copilot-pull-request-reviewer 14 days ago

copilot-pull-request-reviewer commented on 2026-09-25

xadupre requested a review from

copilot-pull-request-reviewer 14 days ago

copilot-pull-request-reviewer commented on 2026-09-25

titaiwangms approved these changes on 2026-09-28

xadupre merged c2650ccb into main 10 days ago

xadupre deleted the docs/moe-offload-step-5 branch 10 days ago

Reviewers

titaiwangms

github-actions

copilot-pull-request-reviewer

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

onnxruntime Update MoE offload implementation plan and implements a global state for MoE/QMoE #32738 Merged

Update MoE offload implementation plan and implements a global state for MoE/QMoE #32738

onnxruntime
Update MoE offload implementation plan and implements a global state for MoE/QMoE
#32738

Merged