Add Inkling model (#47347)
* processor very rough draft
* whatever claude says for now, clean next
* mm towers
* .
* import base model
* make it init model from scratch
* conversion maybe
* vision loads
* tiny change
* add normalization
* more nits
* more
* oops
* typo
* copy linear stuff from qwen, seems quite similar
* a bit more
* relative position bias draft
* no act
* dtype force fp32 to match as closely as we can
* make it run
* torchaudio dep doesn't work for me, just skip for now
* moe loads kinda
* oke now the keys match
* moe router fix
* .
* hmm, I think mlp has to be deinterlaved not chunked
* i am defi breaking the model again XD
* same for routed experts mlp
* dont shard shared gammas ig
* well we still need cache for full vs slliding
* this is ugly!
* ig?
* delete
* fix norm
* fix normer
* tied + inits
* sync modular
* Add TML audio support (feature extractor, processor, audio tower wiring)
Ports the dMel audio pipeline onto the synced skeleton-modular base:
- feature extractor returns the log-mel spectrogram + mask
- processor quantizes it into audio_input_ids (dMel tokens) + mask
- TmlAudioModelEmbeddings/TmlAudioModel embed the dMel tokens
- get_audio_features + generalized get_placeholder_mask wire audio into the forward
- conversion_mapping renames audio.encoder -> embed_audio_tokens, final_norm -> norm
* parallelism wars
* fix position_bias -> use cache primitives directly
* do not slice the mask
* position_ids
* re add some inheritance
* init global scales at 1: super important if we want to train correctly
* inherit more
* simplify conv
* no need to pad before update, the update does it
* add accelerate hook decorator
* big cache refactor to have native implem
* small fix
* add static cache support
* make it compileable
* remove cache param completely
* do not transpose every time
* oupsi already had conversions
* remove the very suspicious dtype casting
* support sdpa
* fix bad reshape
* add flex support
* fix
* oupsi, pass layer_idx
* upstream TP fix
* big model fixes + conv1d update diff from qwen
* fix_compile
* TP transpose conversion is free
* Reuse HiggsAudioV2Embeddings for TmlAudioModelEmbeddings
The controlled port had inlined the dMel embeddings because the standalone-era
skeleton-modular had no cross-model inheritance. Now that skeleton-modular uses
inheritance again, restore TmlAudioModelEmbeddings(HiggsAudioV2Embeddings) + the
TmlAudioConfig num_codebooks/codebook_size/hidden_size attribute map. Behavior and
weight names unchanged (verified: logits + load identical).
* add kernels
* log scaling + image processing changes
* Ink it up
* add load_audio_backend
* allow to rollback linear cache layers
* make repo
* record past
* test audio utils
* audio/ image bos
* TmlProcessingIntegrationTest
* add TmlIntegrationTest
* nit
* nit
* add requires_backends
* can rollback prefill as well
* no rescale_image_frac
* avoid downcasting for linear attn
* add kernels + fix env vars for kernel disablement
* format
* fix save pretrained for Interleave
* better mappings
* more automappings
* change name
* implement mtp cache rollback and correction
* attempts at fixing MTP (noisy outputs)
* solve all indexing issues between main_model index and ntp layers
* small fix
* oupsi forgot to apply modular
* replace sliding layers for now
* allow sliding in mtp layers
* fix the masking for mtp layers
* fix cropping and a lot of issues by replacing get_seq_length to get offsets
* oupsi
* replace layers
* bucket to hf-internal-testing
* add test_apply_chat_template_audio_without_attention_mask
* fix mtp
* update test path
* improve cache invalidation a lot
* fix tests 1 / N
* update integration test
* update bucket path
* fix tests 2/N
* fix tests 3 / N
* fix tests 4/ N: Chunk dim 1
* fix fe edge case
* fix tests 5 / N
* update all mamba models
* upddate token ids
* slice them
* few missing
* [core] force contiguity on chunking
* fix tests 6 / N
* chat template testing 7 / N
* format + processor test fix 8 / N
* small fix
* skip tiny model offload fix 9 / N
* small fix
* conflict dropped the crochet
* style
---------
Co-authored-by: raushan <raushan@huggingface.co>
Co-authored-by: Eustache Le Bihan <eulebihan@gmail.com>
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>
Co-authored-by: eustlb <94853470+eustlb@users.noreply.github.com>
