transformers
fd204af8 - Standardise Aria's MoE onto the experts interface (#48907)

Commit
65 days ago
Standardise Aria's MoE onto the experts interface (#48907) * Standardise Aria's MoE onto the experts interface Aria's routed experts predate the standard `use_experts_implementation` interface. They were a Megatron-style grouped GEMM (`AriaGroupedExpertsGemm` over `fc1`/`fc2`) that did its own routing inside `AriaExperts.forward` and synced to host on every call via `tokens_per_expert.cpu()`. Replace them with `AriaTextTopKRouter` + `AriaExperts`, the same shape GraniteMoe and Qwen3-MoE use: the router returns `(top_k_index, top_k_weights, router_logits)` and the experts take them as arguments. Aria's checkpoints already store each expert's projections as (in_features, out_features), which is `grouped_mm`'s native layout, so the experts are declared `is_transposed=True` and the legacy weights map across by rename alone. Outputs are bit-identical on existing checkpoints. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Skip Aria's two pre-existing export failures Standardising the experts removed the blanket "eager MoE implementation which is not torch exportable" skip that `_skip_if_not_exportable` applied to Aria, so its export tests run for the first time. The text model now exports cleanly, but two unrelated pre-existing failures become visible: - `AriaProjector` picks its query count by looking the patch dim up in a Python dict, which raises `TypeError: unhashable type: non-nested SymInt` under dynamic shapes. - Exported decode returns only `logits` while eager also surfaces the populated KV cache, the same shape as the existing Voxtral/Gemma3n entries. Record both in `EXPORT_SKIPS` with a TODO rather than widening this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix Aria's export failures instead of skipping them Replaces the `EXPORT_SKIPS` entries from the previous commit with fixes for the two problems they papered over. Both are pre-existing; they only became visible because standardising the experts removed the blanket "eager MoE implementation" skip that had been hiding every Aria export test. `AriaProjector` looked its query count up in a Python dict keyed on the patch dim, which `torch.export` cannot hash when that dim is symbolic. Comparing the keys instead lets export specialise on the equality guard. The lookup is over two entries and the `KeyError` path is unchanged. `AriaModel.forward` returned `past_key_values` only when `use_cache` was truthy, so exported decode surfaced just `logits` while eager also returned the populated cache. Every other Llava-derived model returns it unconditionally, and the language model already returns `None` when caching is off, so this aligns Aria with `LlavaModel` rather than changing what callers see. Aria's export tests now pass in full: 24 subtests, against 19 before this PR (with the language-model exports failing) and 7 with the skips. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
Parents
Loading