Add inkling2 (#50682)
* draft
* simplify processor a bit
* reduce modular, 4d pixel values and fix mappings
* small iteration
* ternary
* refactor decoder layer forward
* refactor block_attention_residual
* add Inkling2BlockAttentionResidual
* keep vision encoder in block order
* remove copy?
* docs + checkerboard testing
* fix router bias precision, parallel plans and log-scaling positions
* remove shared experts and unused config options
* remove host syncs from the vision path
* address review comments on config and modules
* share test overrides and move the vision parity check out
* add a processor test
* better naming
* torch.Tensor
* [inkling2] load the NVFP4 / MXFP8 experts through the quantization plan
Replaces the `DequantizeInkling2Experts` converter with renamings and the quantizers' own dequantize ops:
- conversion mapping: `w13_weight*` / `w2_weight*` become `gate_up_proj*` / `down_proj*`, companions included
- NVFP4: fused experts as `serialized_names`; `scale_2 = amax / outer_scale_denominator`; dequantized by
chunks of experts with a single lookup per byte
- FP8: `{proj}_scale` -> `{proj}_scale_inv` via `renamed_names` (scoped to FP8 layers); dequantized by chunks
of experts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* add Inkling2ScaledL2Clip
* add linear
* add comment
* update router/moe
* fix is_concatenated
* add Inkling2Softcap
* remove n_shared_experts
* inherit vision config
* cleanup image processor
* processor image token handling
* split attention classes
* use norm layer
* update docs
* update docstrings
* enable sdpa for vision tower
* small improvements
* update config
* hardcode missing config values
* keep block states as tensors
* update tests
* [inkling2] keep the convolution weight renaming off `weight_proj`
* [inkling2] quantization config as glob patterns
* fix export
* fix bos token
* add tokenizer
* [inkling2] move block_states to each layer's device (device_map)
* [inkling2] default pad_token to <|endoftext|> (checkpoints don't set one)
* [inkling2] small integration test: pin logits measured on CUDA with tight tolerances
* Revert "[inkling2] default pad_token to <|endoftext|> (checkpoints don't set one)"
This reverts commit 4d0429be1b3b54becca206d73a28ef739420e769.
* remove tokenizer
* small improvement
* update docs
* Fix onnx export
* add CI expectations
* small fixes
* update docs
* make it work
* update docs
* cleanup
* rename
---------
Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com>
Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
