transformers

fc501343 - Add Fun-ASR-Nano model (#46180)

Add Fun-ASR-Nano model (#46180) * Add Fun-ASR-Nano model (800M end-to-end ASR) Fun-ASR-Nano is a speech recognition model from Alibaba DAMO Academy consisting of a SANM audio encoder, Transformer adaptor, and Qwen3-0.6B LLM. This commit adds: - configuration_fun_asr_nano.py: Config classes for all components - modeling_fun_asr_nano.py: SANM encoder, adaptor, CTC decoder, main model - feature_extraction_fun_asr_nano.py: Mel + LFR feature extractor - convert_fun_asr_nano_to_hf.py: Weight conversion from FunASR format Resolves #46179 * Add processor, tests, and documentation for Fun-ASR-Nano * Register Fun-ASR-Nano in Auto classes * Fix code quality: ruff format, remove auto-generated file edits * Fix code quality: ruff format, correct Auto class registration * Address review: add integration tests, replace build_chat_input with apply_chat_template * Fix docstring check: use @auto_docstring on processor __call__ * Fix: document sampling_rate param for auto_docstring check * Fix processor docstring to match Qwen2Audio pattern * Add test file to TEST_FILES_WITH_NO_COMMON_TESTS * Rewrite tests with ALMModelTester/ALMModelTest common test pattern * Fix: simplify conditional_generation_class assignment for check_repo regex * Add AdaptorConfig and CtcConfig to docs * Retrigger CI + remove build_chat_input from docs * Add **kwargs to FunAsrNanoEncoder.forward for check_repo compatibility * Add @auto_docstring with checkpoint to config classes * Fix all config classes: use @strict + class-level annotations + @auto_docstring * Document all config parameters for auto_docstring check * Remove unused positional_dropout_rate from encoder config * Add dates to model doc * tests: add integration tests for FunAsrNano (generate Chinese/English) * Fix FunAsrNano tests and audio processor * Refactor fun_asr_nano to modular pattern; address review - Add modular_fun_asr_nano.py as the canonical source; modeling/config/processing now generate from it (modular conversion in sync, idempotent). - Rename encode_audio -> get_audio_features (audioflamingo3 convention), returning BaseModelOutputWithPooling; add a proper FunAsrNanoPreTrainedModel base class. - Fold the adaptor params into the main config (Voxtral pattern); keep the encoder config standalone (Parakeet pattern); drop the unused CTC config/head (no CTC weights in the checkpoint, not on the generation path). - Processor: drop custom batch_decode/decode forwarders and the default chat template; the chat template now lives in the checkpoint (chat_template.jinja), written by the convert script (vibevoice_asr pattern). - Docs: add single/batch/training/torch.compile usage sections. - Tests: real single/batch integration tests against the checkpoint (output byte-identical to the pre-refactor baseline). * Fix fun_asr_nano model card contribution date (repo-consistency check) * Refactor fun_asr_nano to modular pattern and fix checkpoint dtype - Migrate the model to the modular pattern: split lm_head out of the base FunAsrNanoModel (mirroring AudioFlamingo3 / Voxtral), and regenerate modeling from modular. - Move the audio feature extraction to a torch-based FunAsrNanoFeatureExtractor and tidy the processor. - Fix the conversion script to emit a dtype-consistent bf16 checkpoint (cast every weight to bfloat16, set config.dtype=bfloat16). This shrinks the checkpoint from 2.75GB to 1.97GB, matching the LLM dtype. lm_head.weight is kept explicitly (deduplicated by safetensors against the tied embeddings), giving 0 missing / 0 unexpected keys on load. - Make the integration test model id overridable via env for local checkpoint verification. * fix(fun_asr_nano): declare tie_word_embeddings in config and refresh model card date - Declare tie_word_embeddings on FunAsrNanoConfig (the model defines _tied_weights_keys, so the config must expose tie_word_embeddings per the modeling-structure rule TRF015). - Refresh the model card contribution date (add_dates check). * Address Fun-ASR-Nano processor review comments * Fix Fun-ASR-Nano modular mask handling * Fix Fun-ASR-Nano model doc date * Fix Fun-ASR-Nano copyright years * Generate Fun-ASR-Nano config from modular * Generate Fun-ASR-Nano processor from modular * Format Fun-ASR-Nano modular processor * Use standard Fun-ASR-Nano config names * Reuse sinusoidal position embedding helper * Reuse audio chat-template normalization * Document Fun-ASR-Nano conversion flow * Guard Fun-ASR-Nano feature extractor imports * Clarify Fun-ASR-Nano checkpoint key mapping * Use real audio in Fun-ASR-Nano training docs * Trim Fun-ASR-Nano config docstrings * Reuse Llama eager attention in Fun-ASR-Nano * Trigger Fun-ASR-Nano CI rerun * Fix FunASR Nano processor isinstance style * Fix FunASR Nano GPU test coverage * Refactor Fun-ASR-Nano with shared audio components * Fix FunASR Nano Bandit check * Fix FunASR Nano quality checks * Fix Fun-ASR-Nano model contribution date * Address remaining Fun-ASR-Nano review feedback * Address Fun-ASR-Nano integration review * Remove redundant Fun-ASR-Nano config docs * Fix legacy Fun-ASR-Nano checkpoint config loading * Fix Fun-ASR-Nano config attribute usage * Update Fun-ASR-Nano model date * Adjust Fun-ASR-Nano offload test split * Refresh Fun-ASR-Nano model doc date * Address Fun-ASR-Nano modular review feedback * Reuse AudioFlamingo3 processor call * Align Fun-ASR-Nano processor review fixes * Use Fun-ASR-Nano attention directly in adaptor * Address Fun-ASR-Nano encoder review comments * Add Fun-ASR-Nano adaptor subconfig * Update Fun-ASR-Nano adaptor auto mapping * Align Fun-ASR-Nano generated processor * Sync Fun-ASR-Nano generated attention * Export Fun-ASR-Nano adaptor config * Document Fun-ASR-Nano adaptor config * Allow derived Fun-ASR-Nano input attributes * Update Fun-ASR-Nano model card date * Separate config since doesn't need modular, and cleanup legacy args. Separate multimodal projector. * Nit naming. * Nit naming. * Switch to ternary. * Fix Fun-ASR-Nano consistency checks Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * refactor ternary * Add Fun-ASR-Nano model (800M end-to-end ASR) Fun-ASR-Nano is a speech recognition model from Alibaba DAMO Academy consisting of a SANM audio encoder, Transformer adaptor, and Qwen3-0.6B LLM. This commit adds: - configuration_fun_asr_nano.py: Config classes for all components - modeling_fun_asr_nano.py: SANM encoder, adaptor, CTC decoder, main model - feature_extraction_fun_asr_nano.py: Mel + LFR feature extractor - convert_fun_asr_nano_to_hf.py: Weight conversion from FunASR format Resolves #46179 * Add processor, tests, and documentation for Fun-ASR-Nano * Register Fun-ASR-Nano in Auto classes * Fix code quality: ruff format, remove auto-generated file edits * Fix code quality: ruff format, correct Auto class registration * Address review: add integration tests, replace build_chat_input with apply_chat_template * Fix docstring check: use @auto_docstring on processor __call__ * Fix: document sampling_rate param for auto_docstring check * Fix processor docstring to match Qwen2Audio pattern * Add test file to TEST_FILES_WITH_NO_COMMON_TESTS * Rewrite tests with ALMModelTester/ALMModelTest common test pattern * Fix: simplify conditional_generation_class assignment for check_repo regex * Add AdaptorConfig and CtcConfig to docs * Retrigger CI + remove build_chat_input from docs * Add **kwargs to FunAsrNanoEncoder.forward for check_repo compatibility * Add @auto_docstring with checkpoint to config classes * Fix all config classes: use @strict + class-level annotations + @auto_docstring * Document all config parameters for auto_docstring check * Remove unused positional_dropout_rate from encoder config * Add dates to model doc * tests: add integration tests for FunAsrNano (generate Chinese/English) * Fix FunAsrNano tests and audio processor * Refactor fun_asr_nano to modular pattern; address review - Add modular_fun_asr_nano.py as the canonical source; modeling/config/processing now generate from it (modular conversion in sync, idempotent). - Rename encode_audio -> get_audio_features (audioflamingo3 convention), returning BaseModelOutputWithPooling; add a proper FunAsrNanoPreTrainedModel base class. - Fold the adaptor params into the main config (Voxtral pattern); keep the encoder config standalone (Parakeet pattern); drop the unused CTC config/head (no CTC weights in the checkpoint, not on the generation path). - Processor: drop custom batch_decode/decode forwarders and the default chat template; the chat template now lives in the checkpoint (chat_template.jinja), written by the convert script (vibevoice_asr pattern). - Docs: add single/batch/training/torch.compile usage sections. - Tests: real single/batch integration tests against the checkpoint (output byte-identical to the pre-refactor baseline). * Fix fun_asr_nano model card contribution date (repo-consistency check) * Refactor fun_asr_nano to modular pattern and fix checkpoint dtype - Migrate the model to the modular pattern: split lm_head out of the base FunAsrNanoModel (mirroring AudioFlamingo3 / Voxtral), and regenerate modeling from modular. - Move the audio feature extraction to a torch-based FunAsrNanoFeatureExtractor and tidy the processor. - Fix the conversion script to emit a dtype-consistent bf16 checkpoint (cast every weight to bfloat16, set config.dtype=bfloat16). This shrinks the checkpoint from 2.75GB to 1.97GB, matching the LLM dtype. lm_head.weight is kept explicitly (deduplicated by safetensors against the tied embeddings), giving 0 missing / 0 unexpected keys on load. - Make the integration test model id overridable via env for local checkpoint verification. * fix(fun_asr_nano): declare tie_word_embeddings in config and refresh model card date - Declare tie_word_embeddings on FunAsrNanoConfig (the model defines _tied_weights_keys, so the config must expose tie_word_embeddings per the modeling-structure rule TRF015). - Refresh the model card contribution date (add_dates check). * Address Fun-ASR-Nano processor review comments * Fix Fun-ASR-Nano modular mask handling * Fix Fun-ASR-Nano model doc date * Fix Fun-ASR-Nano copyright years * Generate Fun-ASR-Nano config from modular * Generate Fun-ASR-Nano processor from modular * Format Fun-ASR-Nano modular processor * Use standard Fun-ASR-Nano config names * Reuse sinusoidal position embedding helper * Reuse audio chat-template normalization * Document Fun-ASR-Nano conversion flow * Guard Fun-ASR-Nano feature extractor imports * Clarify Fun-ASR-Nano checkpoint key mapping * Use real audio in Fun-ASR-Nano training docs * Trim Fun-ASR-Nano config docstrings * Reuse Llama eager attention in Fun-ASR-Nano * Trigger Fun-ASR-Nano CI rerun * Fix FunASR Nano processor isinstance style * Fix FunASR Nano GPU test coverage * Refactor Fun-ASR-Nano with shared audio components * Fix FunASR Nano Bandit check * Fix FunASR Nano quality checks * Fix Fun-ASR-Nano model contribution date * Address remaining Fun-ASR-Nano review feedback * Address Fun-ASR-Nano integration review * Remove redundant Fun-ASR-Nano config docs * Fix legacy Fun-ASR-Nano checkpoint config loading * Fix Fun-ASR-Nano config attribute usage * Update Fun-ASR-Nano model date * Adjust Fun-ASR-Nano offload test split * Refresh Fun-ASR-Nano model doc date * Address Fun-ASR-Nano modular review feedback * Reuse AudioFlamingo3 processor call * Align Fun-ASR-Nano processor review fixes * Use Fun-ASR-Nano attention directly in adaptor * Address Fun-ASR-Nano encoder review comments * Add Fun-ASR-Nano adaptor subconfig * Update Fun-ASR-Nano adaptor auto mapping * Align Fun-ASR-Nano generated processor * Sync Fun-ASR-Nano generated attention * Export Fun-ASR-Nano adaptor config * Document Fun-ASR-Nano adaptor config * Allow derived Fun-ASR-Nano input attributes * Update Fun-ASR-Nano model card date * Separate config since doesn't need modular, and cleanup legacy args. Separate multimodal projector. * Nit naming. * Nit naming. * Switch to ternary. * refactor ternary * Fix Fun-ASR-Nano consistency checks Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * Fix Fun-ASR-Nano repository consistency Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * Fix Fun-ASR-Nano legacy adaptor loading Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * Address Fun-ASR-Nano review feedback Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * Fix Fun-ASR-Nano batched transcription inputs Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * Update Fun-ASR-Nano model contribution date Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * Update Fun-ASR-Nano model date Refresh the generated contribution date after synchronizing the model integration with current main. Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> * style: apply current Ruff formatting Signed-off-by: LauraGPT <lauragpt@users.noreply.github.com> * Update Fun-ASR-Nano model contribution date Signed-off-by: LauraGPT <lauragpt@users.noreply.github.com> * Update Fun-ASR-Nano documentation examples Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * Align Fun-ASR-Nano with modeling conventions Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * Regenerate Fun-ASR-Nano on current main Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * Clean up tests, standarizing with Transformers conventions (modular, feature extractor, processor). * Revert AF3 date. * Fix Fun-ASR-Nano checkpoint fixtures Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * Switch to new audio utils. * feature extractor nits: naming and batched * regenerate processor from modular * regenerate processor from modeling * nit refactor * Fix Fun-ASR-Nano review nits * Standardize Fun-ASR-Nano audio configuration * Update Fun-ASR-Nano model card date * Refactor Fun-ASR-Nano encoder components * Refine Fun-ASR-Nano review feedback Signed-off-by: LauraGPT <lauragpt@users.noreply.github.com> * Fix Fun-ASR-Nano CI consistency checks Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * Reference Fun-ASR-Nano audio input dimensions directly Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * test: address Fun-ASR-Nano review nits Cache the remote audio fixture and update the new model test copyright. Signed-off-by: LauraGPT <lauragpt@users.noreply.github.com> * Address comments, shared layer for encoder and adaptor, use more recent attribute names. * better FunAsrNanoEncoderLayer, and nits * fix: preserve optional Fun-ASR-Nano masks Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * align with llama attentino * modular after sync * fuse layers in encoder. * nits * Modular nits * fix: honor Fun-ASR-Nano attention and normalization configuration Signed-off-by: zhifu gao <zhifu.gzf@alibaba-inc.com> * Remove second application of mask * Address modular comments, and other nits. * tests: use accelerator gate for FunASR Nano integration Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> * Address comments * Fix Fun-ASR-Nano internal audio return type and cover projector kwargs Signed-off-by: zhifu gao <zhifu.gzf@alibaba-inc.com> * Test Fun-ASR-Nano with the reconverted official checkpoint Signed-off-by: zhifu gao <zhifu.gzf@alibaba-inc.com> * remove tests * Set headdim for flash attention tests. --------- Signed-off-by: LauraGPT <LauraGPT@users.noreply.github.com> Signed-off-by: LauraGPT <lauragpt@users.noreply.github.com> Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> Signed-off-by: zhifu gao <zhifu.gzf@alibaba-inc.com> Co-authored-by: xiaoyunchong.xyc <xiaoyunchong.xyc@alibaba-inc.com> Co-authored-by: LauraGPT <LauraGPT@users.noreply.github.com> Co-authored-by: Eric B <ebezzam@gmail.com> Co-authored-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com> Co-authored-by: Eric Bezzam <4757445+ebezzam@users.noreply.github.com>