Commit
7 days ago
Add h4 (#48473)
* feat: add HYV4 open-source model (transformers 5.15)
Standalone open-source HYV4 (MLA + DeepSeek-style sparse attention, gated MLA,
learnable sink, independent Hyper-Connections, MoE) on the os v5.14.2 branch:
- modular_hy_v4.py as canonical source; config/modeling regenerated with this
tree's converter against its glm4_moe_lite / glm_moe_dsa parents
- Auto registration (config/model/causal-lm), models/__init__ lazy import,
conversion_mapping entry
- adopt the 5.15 native DSA cache: layer_types=deepseek_sparse_attention so the
base DynamicCache provisions DynamicIndexedLayer per layer; the indexer uses
past_key_values.update_indexer(); the bespoke HYV4Cache is removed
- point config.head_dim at qk_rope_head_dim so the rotary embedding sizes cos/sin
to the RoPE slice (same convention as glm_moe_dsa); qk_head_dim keeps the full
MLA Q/K dimension for the projections
- config/modeling carry no custom validation or exception handling beyond the
canonical input_ids/inputs_embeds check
Validated on transformers 5.15.0.dev0: 17 unit tests pass, modular output
stable, real checkpoint loads (0 missing/unexpected) with bit-identical logits
to the 5.10 reference (argmax match, max_abs_diff=0), and greedy chat generation
produces coherent text with correct EOS.
* feat(hy_v4): apply review_0818 fixes and reuse GLM4-MoE-Lite router (#11)
* feat(hy_v4): apply review_0818 fixes and reuse GLM4-MoE-Lite router
Address the community review on the HYV4 open-source PR:
- Reuse standard components: HYV4Indexer inherits GlmMoeDsaIndexer;
HYV4TopKRouter now inherits Glm4MoeLiteTopkRouter (n_group=topk_group=1
collapses grouped selection to the released global sigmoid top-k;
verified numerically identical on the released checkpoint).
- Drop fixed-value runtime branches: gated MLA, learnable sink, iHC and
normalized top-k are release invariants, enforced in __post_init__ and
no longer double-pathed.
- Rework the attention sink to GPT-OSS sink-in-softmax semantics; eager
only (_supports_sdpa=False, as in gpt_oss).
- FP32 lm_head via _keep_in_fp32_modules_strict instead of a hand-rolled
F.linear + all_gather.
- MTP is not executed by Transformers: remove the MTP config fields and
only ignore the model.mtp_layers.* tensors on load (kept in the shared
checkpoint for other runtimes).
- Tests: remove redundant auto/configuration test files and fold key
assertions into the modeling test.
* refactor(hy_v4): drop config invariant ValueErrors and standardize tests
Follow-up to the review_0818 changes:
- Remove the release-invariant AND-chain ValueError and the bespoke
layer_metadata / first-indexer validation from HYV4Config.__post_init__.
GLM-MoE-DSA raises none and DeepSeek-V4 only validates layer-type lists,
so these defensive checks are not upstream-idiomatic; __post_init__ now
only computes derived fields and builds the default layer/indexer lists.
- Rewrite tests/models/hy_v4/test_modeling_hy_v4.py to the standard
CausalLMModelTester + CausalLMModelTest(unittest.TestCase) pattern used
by glm_moe_dsa and deepseek_v4, replacing the hand-rolled forward/loss/
rank/iHC/sink assertions. Keep only HYV4-specific unit tests (default
layer types, fp32 lm_head, MTP-ignore, iHC/sink layout, DSA sentinel and
shared-indexer guard) plus the DSA hard-top-k skip decorators.
* fix(hy_v4): correct TP plan, scope fp32 lm_head, and pass the full model test suite
The standardized CausalLMModelTest surfaced three real issues that the
previous bespoke tests hid:
- Remove `layers.*.self_attn.q_proj` from base_model_tp_plan: HYV4 uses MLA
(q_a_proj / q_b_proj), there is no `q_proj` parameter, so the entry was
invalid (caught by test_tp_plan_matches_params).
- Move `lm_head` out of the shared PreTrainedModel `_keep_in_fp32_modules_strict`
onto HYV4ForCausalLM only; the base HYV4Model has no lm_head
(caught by test_keep_in_fp32_modules_exist).
- Add iHC-aware test overrides mirroring DeepSeek-V4: the per-layer hidden
states carry the 4D `[batch, seq, hc_mult, hidden]` iHC stream (collapsed by
hc_head only at the top), and MLA changes the KV cache head_dim. Override
test_hidden_states_output / _check_hidden_states_for_generate /
_check_past_key_values_for_generate accordingly.
Full suite now: 117 passed, 1207 subtests passed, 0 failed. End-to-end
generation on the released checkpoint is unchanged.
* test(hy_v4): drop redundant skips, keep only the two that are truly needed
Verified each skip by removing it and running the base implementation:
- test_eager_matches_sdpa_inference, test_sdpa_padding_matches_padding_free,
test_generate_{from_inputs_embeds_with_static_cache,compile_model_forward_fullgraph,
compilation_all_outputs,with_static_cache}, test_eager_matches_batched_and_grouped:
the base tester already auto-skips these (HYV4 has _supports_sdpa=False and no
static-cache / compile support), so the explicit @unittest.skip was redundant.
- test_left_padding_compatibility, test_eager_padding_matches_padding_free_with_position_ids:
actually PASS on HYV4 — the DSA "selection flips" rationale copied from GLM-MoE-DSA
does not apply, so they should run.
Only test_assisted_decoding_matches_greedy_search / test_assisted_decoding_sample
genuinely fail (HYV4 has no default dynamic cache, which assisted decoding requires),
so those keep an accurate skip. Suite: 119 passed, 0 failed.
---------
Co-authored-by: manayang <manayang@tencent.com>
* Feat/hyv4 review 0818 followup (#13)
* feat(hy_v4): apply review_0818 fixes and reuse GLM4-MoE-Lite router
Address the community review on the HYV4 open-source PR:
- Reuse standard components: HYV4Indexer inherits GlmMoeDsaIndexer;
HYV4TopKRouter now inherits Glm4MoeLiteTopkRouter (n_group=topk_group=1
collapses grouped selection to the released global sigmoid top-k;
verified numerically identical on the released checkpoint).
- Drop fixed-value runtime branches: gated MLA, learnable sink, iHC and
normalized top-k are release invariants, enforced in __post_init__ and
no longer double-pathed.
- Rework the attention sink to GPT-OSS sink-in-softmax semantics; eager
only (_supports_sdpa=False, as in gpt_oss).
- FP32 lm_head via _keep_in_fp32_modules_strict instead of a hand-rolled
F.linear + all_gather.
- MTP is not executed by Transformers: remove the MTP config fields and
only ignore the model.mtp_layers.* tensors on load (kept in the shared
checkpoint for other runtimes).
- Tests: remove redundant auto/configuration test files and fold key
assertions into the modeling test.
* refactor(hy_v4): drop config invariant ValueErrors and standardize tests
Follow-up to the review_0818 changes:
- Remove the release-invariant AND-chain ValueError and the bespoke
layer_metadata / first-indexer validation from HYV4Config.__post_init__.
GLM-MoE-DSA raises none and DeepSeek-V4 only validates layer-type lists,
so these defensive checks are not upstream-idiomatic; __post_init__ now
only computes derived fields and builds the default layer/indexer lists.
- Rewrite tests/models/hy_v4/test_modeling_hy_v4.py to the standard
CausalLMModelTester + CausalLMModelTest(unittest.TestCase) pattern used
by glm_moe_dsa and deepseek_v4, replacing the hand-rolled forward/loss/
rank/iHC/sink assertions. Keep only HYV4-specific unit tests (default
layer types, fp32 lm_head, MTP-ignore, iHC/sink layout, DSA sentinel and
shared-indexer guard) plus the DSA hard-top-k skip decorators.
* fix(hy_v4): correct TP plan, scope fp32 lm_head, and pass the full model test suite
The standardized CausalLMModelTest surfaced three real issues that the
previous bespoke tests hid:
- Remove `layers.*.self_attn.q_proj` from base_model_tp_plan: HYV4 uses MLA
(q_a_proj / q_b_proj), there is no `q_proj` parameter, so the entry was
invalid (caught by test_tp_plan_matches_params).
- Move `lm_head` out of the shared PreTrainedModel `_keep_in_fp32_modules_strict`
onto HYV4ForCausalLM only; the base HYV4Model has no lm_head
(caught by test_keep_in_fp32_modules_exist).
- Add iHC-aware test overrides mirroring DeepSeek-V4: the per-layer hidden
states carry the 4D `[batch, seq, hc_mult, hidden]` iHC stream (collapsed by
hc_head only at the top), and MLA changes the KV cache head_dim. Override
test_hidden_states_output / _check_hidden_states_for_generate /
_check_past_key_values_for_generate accordingly.
Full suite now: 117 passed, 1207 subtests passed, 0 failed. End-to-end
generation on the released checkpoint is unchanged.
* test(hy_v4): drop redundant skips, keep only the two that are truly needed
Verified each skip by removing it and running the base implementation:
- test_eager_matches_sdpa_inference, test_sdpa_padding_matches_padding_free,
test_generate_{from_inputs_embeds_with_static_cache,compile_model_forward_fullgraph,
compilation_all_outputs,with_static_cache}, test_eager_matches_batched_and_grouped:
the base tester already auto-skips these (HYV4 has _supports_sdpa=False and no
static-cache / compile support), so the explicit @unittest.skip was redundant.
- test_left_padding_compatibility, test_eager_padding_matches_padding_free_with_position_ids:
actually PASS on HYV4 — the DSA "selection flips" rationale copied from GLM-MoE-DSA
does not apply, so they should run.
Only test_assisted_decoding_matches_greedy_search / test_assisted_decoding_sample
genuinely fail (HYV4 has no default dynamic cache, which assisted decoding requires),
so those keep an accurate skip. Suite: 119 passed, 0 failed.
* refactor(hy_v4): inherit HYV4MoE.forward from Glm4MoeLiteMoE
Address the review comment "HYV4MoE.forward can be inherited". HYV4MoE now
inherits Glm4MoeLiteMoE and only overrides __init__ to install the HYV4 gate /
experts / shared_experts; the flatten -> router -> experts -> shared-combine ->
reshape flow is reused from the parent.
The only prior difference was that HYV4 combined the routed and shared outputs
in float32 before casting back. Dropping that in favour of the parent's native-
dtype add is numerically identical on the released checkpoint (full-vocab
logits bit-exact: fp64 sum -2940884.9999939157, argmax 12518; 119 tests /
1207 subtests pass; end-to-end generation unchanged).
---------
Co-authored-by: manayang <manayang@tencent.com>
* Feat/hyv4 review 0819 (#14)
* feat(hy_v4): apply review_0818 fixes and reuse GLM4-MoE-Lite router
Address the community review on the HYV4 open-source PR:
- Reuse standard components: HYV4Indexer inherits GlmMoeDsaIndexer;
HYV4TopKRouter now inherits Glm4MoeLiteTopkRouter (n_group=topk_group=1
collapses grouped selection to the released global sigmoid top-k;
verified numerically identical on the released checkpoint).
- Drop fixed-value runtime branches: gated MLA, learnable sink, iHC and
normalized top-k are release invariants, enforced in __post_init__ and
no longer double-pathed.
- Rework the attention sink to GPT-OSS sink-in-softmax semantics; eager
only (_supports_sdpa=False, as in gpt_oss).
- FP32 lm_head via _keep_in_fp32_modules_strict instead of a hand-rolled
F.linear + all_gather.
- MTP is not executed by Transformers: remove the MTP config fields and
only ignore the model.mtp_layers.* tensors on load (kept in the shared
checkpoint for other runtimes).
- Tests: remove redundant auto/configuration test files and fold key
assertions into the modeling test.
* refactor(hy_v4): drop config invariant ValueErrors and standardize tests
Follow-up to the review_0818 changes:
- Remove the release-invariant AND-chain ValueError and the bespoke
layer_metadata / first-indexer validation from HYV4Config.__post_init__.
GLM-MoE-DSA raises none and DeepSeek-V4 only validates layer-type lists,
so these defensive checks are not upstream-idiomatic; __post_init__ now
only computes derived fields and builds the default layer/indexer lists.
- Rewrite tests/models/hy_v4/test_modeling_hy_v4.py to the standard
CausalLMModelTester + CausalLMModelTest(unittest.TestCase) pattern used
by glm_moe_dsa and deepseek_v4, replacing the hand-rolled forward/loss/
rank/iHC/sink assertions. Keep only HYV4-specific unit tests (default
layer types, fp32 lm_head, MTP-ignore, iHC/sink layout, DSA sentinel and
shared-indexer guard) plus the DSA hard-top-k skip decorators.
* fix(hy_v4): correct TP plan, scope fp32 lm_head, and pass the full model test suite
The standardized CausalLMModelTest surfaced three real issues that the
previous bespoke tests hid:
- Remove `layers.*.self_attn.q_proj` from base_model_tp_plan: HYV4 uses MLA
(q_a_proj / q_b_proj), there is no `q_proj` parameter, so the entry was
invalid (caught by test_tp_plan_matches_params).
- Move `lm_head` out of the shared PreTrainedModel `_keep_in_fp32_modules_strict`
onto HYV4ForCausalLM only; the base HYV4Model has no lm_head
(caught by test_keep_in_fp32_modules_exist).
- Add iHC-aware test overrides mirroring DeepSeek-V4: the per-layer hidden
states carry the 4D `[batch, seq, hc_mult, hidden]` iHC stream (collapsed by
hc_head only at the top), and MLA changes the KV cache head_dim. Override
test_hidden_states_output / _check_hidden_states_for_generate /
_check_past_key_values_for_generate accordingly.
Full suite now: 117 passed, 1207 subtests passed, 0 failed. End-to-end
generation on the released checkpoint is unchanged.
* test(hy_v4): drop redundant skips, keep only the two that are truly needed
Verified each skip by removing it and running the base implementation:
- test_eager_matches_sdpa_inference, test_sdpa_padding_matches_padding_free,
test_generate_{from_inputs_embeds_with_static_cache,compile_model_forward_fullgraph,
compilation_all_outputs,with_static_cache}, test_eager_matches_batched_and_grouped:
the base tester already auto-skips these (HYV4 has _supports_sdpa=False and no
static-cache / compile support), so the explicit @unittest.skip was redundant.
- test_left_padding_compatibility, test_eager_padding_matches_padding_free_with_position_ids:
actually PASS on HYV4 — the DSA "selection flips" rationale copied from GLM-MoE-DSA
does not apply, so they should run.
Only test_assisted_decoding_matches_greedy_search / test_assisted_decoding_sample
genuinely fail (HYV4 has no default dynamic cache, which assisted decoding requires),
so those keep an accurate skip. Suite: 119 passed, 0 failed.
* refactor(hy_v4): apply review_0819 (mask, experts, eager, HC form)
Address ArthurZucker's 2026-08-19 review:
- Indexer mask: `_build_sparse_mask` now uses the GLM-MoE-DSA pattern
(all-masked bool mask + scatter selected keys to False), with a padded
column to keep HYV4's `-1` empty-slot sentinel out of the key range.
- Experts: drop the local/non-local `valid_routes` filtering; use the
standard stacked-expert dispatch (one_hot with the sentinel column) and
let `RouterParallel` correct locality, matching Mixtral/GPT-OSS.
- Attention: reuse `eager_attention_forward` imported from gpt_oss; add a
`sinks` property on HYV4Attention aliasing `learnable_sink_param` so the
released checkpoint tensor name is preserved. (Drops the FP32 QK matmul:
top-1 token and end-to-end generation unchanged, raw logits differ ~0.1%.)
- Hyper-Connections: converge onto the DeepSeek-V4 form — HYV4HyperConnection
has a single forward returning (post_gates, collapsed, streams) plus a
static add_residual, and HYV4HyperHead is the single-forward head (renamed
from HYV4HCHeadLayer). Removed the prepare_input/pre/post split, the
one-liner _hc_rms_gated_logits helper, and the param-less HYV4HCPostLayer.
Parameter keys are unchanged (hc_pre retained as the param holder).
FP32 LM head + its forward override are intentionally kept (see review reply);
the bf16-head alternative changed released generation, so that choice is left
to the reviewer. Full suite: 119 passed / 1207 subtests; HC refactor bit-exact;
end-to-end generation unchanged.
* refactor(hy_v4): flatten HyperConnection params to DSv4 form via load-time renaming
Follow-up to the review_0819 HC change: fully adopt the DeepSeek-V4
`HyperConnection` form by dropping the intermediate `hc_pre` sub-module and
placing the iHC parameters (`hc_fn` / `hc_scale` / `hc_base`) directly on
`HYV4HyperConnection`, as the reviewer asked ("remove one-liner functions like
the pre func").
The released checkpoint stores these under `...hc_attn_layer.hc_pre.hc_*`, so a
`WeightRenaming(r"\.hc_pre\.hc_", ".hc_")` entry is registered for `hy_v4` in
conversion_mapping.py to map them at load time — no checkpoint edit, keys on disk
unchanged. `hy_v4` no longer aliases the qwen2_moe expert-merge conversion, which
was a no-op anyway since the released experts are already stored stacked.
Verified on the released checkpoint: strict load reports 0 missing / 0 unexpected
(only `model.mtp_layers.*` ignored), full-vocab logits bit-exact vs. before
(fp64 sum -2944464.4418902863, argmax 12518), 119 tests / 1207 subtests pass,
end-to-end generation unchanged.
* style(hy_v4): trim excess comments and docstrings
---------
Co-authored-by: manayang <manayang@tencent.com>
* fix(hy_v4): normalize num_key_value_heads for MLA attention (#15)
HYV4 uses MLA: `kv_b_proj` expands the compressed latent into one key/value
per query head, so keys are never grouped. The shared eager attention path
calls `repeat_kv(key, num_key_value_groups)`, which is only an identity when
`num_key_value_heads == num_attention_heads`.
Released checkpoints can still carry a grouped-attention value (HY4-SFT0805
stores `num_attention_heads=64` with `num_key_value_heads=8`), which repeated
already per-head keys 8 times and made attention fail:
RuntimeError: The size of tensor a (4) must match the size of tensor b (32)
at non-singleton dimension 1
Normalize the field in `__post_init__`, matching the MLA convention in
DeepSeek-V3/V3.2 and GLM-MoE-DSA, whose configs keep both head counts equal.
The default HYV4 config is also equal, so the existing tests never exercised a
grouped value; a regression test now pins the invariant.
Verified on HY4-SFT0805 with 16-way tensor parallel: strict load reports
0 missing / 0 unexpected keys, all 16 ranks produce bit-identical
full-vocabulary logits (fp64 sum spread 0.0), and greedy generation matches
across ranks.
Co-authored-by: manayang <manayang@tencent.com>
* docs(hy_v4): add the model doc page (#16)
`utils/check_repo.py` failed because `HYV4Config`, `HYV4Model` and
`HYV4ForCausalLM` are exported from the public init but had no doc page:
Exception: The following objects are in the public init, but not in the docs:
- HYV4Config
- HYV4ForCausalLM
- HYV4Model
Add `docs/source/en/model_doc/hy_v4.md` describing MLA, the shared DSA indexer,
gated MLA with learnable sinks, and independent Hyper-Connections, plus tensor-
and expert-parallel loading, and register it in the toctree next to HYV3.
Co-authored-by: manayang <manayang@tencent.com>
* Fix/hyv4 review 0825 (#17)
* docs(hy_v4): add the model doc page
`utils/check_repo.py` failed because `HYV4Config`, `HYV4Model` and
`HYV4ForCausalLM` are exported from the public init but had no doc page:
Exception: The following objects are in the public init, but not in the docs:
- HYV4Config
- HYV4ForCausalLM
- HYV4Model
Add `docs/source/en/model_doc/hy_v4.md` describing MLA, the shared DSA indexer,
gated MLA with learnable sinks, and independent Hyper-Connections, plus tensor-
and expert-parallel loading, and register it in the toctree next to HYV3.
* refactor(hy_v4): address review feedback
* fix(hy_v4): reject unclamped SonicMoE backend
* fix(hy_v4): preserve HC numerics and expert dispatch
* test(hy_v4): keep distributed smoke local
---------
Co-authored-by: manayang <manayang@tencent.com>
* post merge quick fixes
* some simplifications
* attention refactor
* hc refactor
* some cleanup
* fix repo
* fixes per check repo
* fix last tests
* style
* last fixes on export and offloading
* use maybe auto cast
* fix date
---------
Co-authored-by: manayang <manayang@tencent.com>
Co-authored-by: manayang <jackmanayang@gmail.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
References
Author
Parents
Loading