transformers
fe95f542 - Add MuseGlimmer (#47867)

Commit
99 days ago
Add MuseGlimmer (#47867) * add-new-model-like gemma2 * Copy trust_remote_code implementation * Start modular for LLM * draft image model and config * prob packed attn will be better in terms of perf, than padding everything * come back on monday, keys match but logits dont * Fix guard order for loading * Fix rms norm eps * some config keys got lost/renamed? * fix permute for rope * images weren't being passed at all, fix it * oops, wrong dim * some defaults are not correct * temporal patch comes first? * still not identical but should be right to match align-corner=False * so positions are flipped, and we need to pad in zeros * and here comes video processing * correctness updates * original expectations + tester (with local paths for now) * small changes * more cleanup + some renaming * enable_circleci * enable_circleci * check_install * fix circleci: use large resource class and clean up parallelism expression Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * halve pytest num workers to fit large resource class Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * check_install * Conversion script * Remove on-load renames * Fix auto mapping * skip some tests for now * Update IMAGE_PROCESSOR_MAPPING_NAMES * more modular for vision * config docs * Make lists explicit * Remove legacy key support * Remove no longer used keys * Use one of the supported layer_types We only check for "full_attention" * Remove legacy flat keys * Apply modular * Adapt converter * Fix typo after rename * make style * more renames to be like qwen * i will better fix on main, not needed anymore here * tests - rename config keys * can delete no? we got a multimodal class working now * why does this not crash more often? * make things green up a bit * check not running * check will run with [run-circleci] commit message * check not running * modular video processing * biiig address of comments and review * Fix Onyx chat generation prompt * style * [run-circleci] * config fixes and docstrings * [run-circleci] * clean up + unbloat with autodoc * format * [run-circleci] * new naming (no need to reconvert) * Remaining downsample_factor reference * onyx(modular): port Meta tool-calling template + response_schema + eos into modular structure Ports the model-team-owned pieces onto HF's modular onyx structure (onyx-modular): - onyx_chat_template.py: the authoritative ATEM tool-calling chat template (byte-identical to internal genai/msl/guac/hf). Wired into processing_onyx.py via build_chat_template(); replaces the modular stub template. - Generation prompt: OVERRIDES the modular hardcode '<|start|>assistant to=user<|message|>' with the bare '<|start|>assistant'. The to=user hardcode structurally prevents tool calls (calls emit on to=<tool> channels). Documented inline; decision to be shared with HF. - convert_onyx_weights_to_hf.py: bake ONYX_RESPONSE_SCHEMA onto the tokenizer (parse_response output parser, byte-identical to internal) + add write_generation_config() emitting generation_config.json with eos=[200001,200008] (drops <|eom|>=200007, the proven eos bug). Verified: template renders bare gen-prompt + ATEM tool-defs + reasoning-strength + valid-recipients + multimodal image sentinel; response_schema md5 == internal; all files compile. NOT yet done (follow-ups): safety-probe gate port (model.model.layers -> modular language_model path), make fix-copies/style, end-to-end convert+generate test. * onyx(modular): port safety-probe gate for the TARGET (modular) architecture Adapts Sid's --safety_probe gate (D113317217) to HF's modular Onyx structure, where the deliverable arch is OnyxForConditionalGeneration (image-text-to-text), not OnyxForCausalLM. The probe concatenates per-layer decoder hidden states, so it must reach the text decoder layers. In modular they nest as: OnyxForConditionalGeneration.model (OnyxModel) -> .language_model (OnyxTextModel) -> .layers whereas the old text-only export was model.model.layers. Hardcoding model.model.layers (as landed) breaks on the modular arch. Changes (deliverable_hf/inference_hf.py, staged copy of genai/msl/guac/hf/inference_hf.py): - add _get_decoder_layers(model): resolves the decoder ModuleList across arches (model.model.language_model.layers -> model.model.layers -> model.language_model.layers), preserving layer order 0..N-1 which the probe requires; raises clearly if none match. - probe uses _get_decoder_layers(model) instead of model.model.layers. - loader: try AutoModelForImageTextToText (modular target, MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING) then fall back to AutoModelForCausalLM, so the script works against either deliverable. Verified: _get_decoder_layers resolves 52 layers for both modular and causal mocks and raises on a bad model; file compiles. NOT yet run end-to-end against a real converted modular checkpoint + the rc1__v07e probe .pt (follow-up). * fixing max token lenght * reverting formatting changes * Increase model max length to 131072 * onyx(modular): fix max_position_embeddings 16384 -> 131072 (RoPE cache crash on convert) build_config() and the OnyxTextConfig default set max_position_embeddings=16384, but the tokenizer model_max_length and generation max_length are 131072. max_position_embeddings sizes the RoPE cos/sin cache (max_seq_len_cached), so any sequence >16384 crashes on RoPE position indexing. A per-checkpoint config.json edit (16384->131072) was applied manually, but every fresh conversion re-emitted 16384 and reintroduced the crash. This is the real fix: emit 131072 from build_config() + make it the OnyxTextConfig default, consistent with model_max_length/max_length=131072. Vision config (32*32) unchanged. Verified: build_config().text_config.max_position_embeddings == 131072; OnyxTextConfig() default == 131072. * vLLM transformers modeling backend changes * rename * parallel tool call line - remove * onyx: move chat template to standalone chat_template.jinja file Extract the Jinja chat template out of the inline ONYX_CHAT_TEMPLATE Python string constant into a sibling chat_template.jinja file, loaded at import time. Matches the HF convention (cf. the Gemma release, which reads chat_template.jinja via cached_file rather than embedding it in the converter). Addresses PR review feedback to move the template off the .py constant before release. - onyx_chat_template.py now reads chat_template.jinja (byte-identical output; build_chat_template() and ONYX_MM_CHAT_TEMPLATE unchanged for consumers) - setup.py: add **/*.jinja to package_data so the template ships in wheels - template content is Ruan's 662a1072d5 (parallel-tool-call line removed) Verified: load + greedy generation of the converted rl_v1 checkpoint on GPU renders correctly via the file-loaded template (plain chat stops on eos; single tool call emits a well-formed <atem:invoke> and stops on eos). * onyx(modular): sync max_position_embeddings in modular source + regen Commit cb5a4d03a7 bumped max_position_embeddings 16384->131072 in the generated configuration_onyx.py but not in the modular source (modular_onyx.py), leaving check_modular_conversion failing. Fix the value in OnyxTextConfig in the modular source and regenerate, which also picks up a stale docstring ordering in modeling_onyx.py (image_grid_thw/video_grid_thw moved after labels, matching current converter output). check_modular_conversion now passes (RC=0); OnyxTextConfig().max_position_embeddings == 131072. * onyx: load chat template from the model Hub repo at convert time Follow the HF convention (cf. the Gemma release, which reads chat_template.jinja from its model repo via cached_file) so the model repo is the single source of truth for the template. The converter now fetches chat_template.jinja from ONYX_HUB_REPO and bakes it into the converted checkpoint, falling back to the copy packaged with transformers when the Hub is unreachable (offline / CI). ONYX_HUB_REPO points at the temporary early-release repo (someorgtoo/onyx_early) and must be updated to the final public repo before release. Runtime processing_onyx.py is unchanged: it still reads the packaged chat_template.jinja locally (no network at import). Only the convert script touches the Hub. * onyx: migrate response parser to declarative response_template Replace the legacy x-regex ONYX_RESPONSE_SCHEMA with the new declarative response_template format (transformers 5.x), addressing PR review feedback that the x-regex response_schema is being deprecated in favor of the streaming-capable response-parsing API. Fields map 1:1 to the Onyx channels: - reasoning_content: open_pattern to=self<|message|>, close <|eom|>, text - tool_calls: open_pattern captures the <atem:invoke> name, xml-inline parses each <atem:parameter> (value as lax JSON), repeats -> list (parallel calls), transform wraps into {type: function, function: {name, arguments}} - content: open_pattern to=user<|message|>, close on <|eot|>/<|eom|>, text Channel-scoping is now structural instead of regex-substitution based: while an explicit region is open the parser only watches that region's close, so an <atem:invoke> echoed inside reasoning or the final answer is body text, not a spurious tool call. Assigned as tokenizer.response_template (was response_schema). The old schema joined multiple to=self blocks; the model emits at most one to=self block per turn (verified across 1768 captured turns), so a single-string reasoning_content region is behaviorally equivalent and matches the standard reasoning_content contract. Verified end-to-end: converter template validates; tokenizer_config.json gets response_template (no legacy response_schema); reload + tokenizer.parse_response() on a real model generation yields correct reasoning_content + tool_calls; parallel calls, nested-object args, mixed scalars, and echoed-in-reasoning all parse correctly. check_modular_conversion RC=0. * Image placeholder -> <|patch|> * downsample_factor -> merge_size * onyx: load chat template from hf/ subfolder of onyx_early_v2 * onyx: remove in-tree chat template, load only from model repo * draft, not checked yet, needs weight conversion * maybe tmrw * make replace image token match reference format * commit before i mess it up again. Shapes match but cropping should go differently * greedy decoding matches assistant, cand suggestion are garbage * needs smth to actually compare logits, i ma just blind shooting based on code inspection... * [run-circleci] * permute rope and fix some cropping * oops * interleave * this gets hugher acceptance, and i need to have lunch * add proper conversion script * nit * unmerged emebds * revert back interpolation and image processor * revert unrelated * do same as in main * just copy test from glm for now, quick sanity check * revert unrelated * add the Text prefix everywhere * small style * natural module order in files * remove unused rope function * [run-circleci] style * [run-circleci] fix record output * fix audio features * [run-circleci] empty commit for CI * simplify and fix cache past recording * un-merge the norm for dflash * rename * put it after mtp in the file * crop correctly in dflash * oupsi forgot to remove * Update src/transformers/models/onyx/processing_onyx.py Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com> * Update src/transformers/models/onyx/processing_onyx.py Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com> * revert and fix cache and position ids, use target states for cache * simplify attention * frogot to revert , comment doesn't match code * fix inference and add docstring * oupsi double equal * correct mask similar to diffusion gemma * clean up * revert all that is left * tests, yet not integration tests * fix integration test * permute rope * update prefix image * doc and remove unused init args * update arg name * a bit more doc and variable name matching * more doc in the model for sensitive aspects * fix bug on q_offset, and use native mask API * simplify a bit * style * no checks at all if speculation_type is dflash * rename * update test * rework and finish tests * namings/configs * minor fixes * Rename onyx_assistant * [revert] get back image delimiters * style * update test * Final location of chat template * align the config defaut value * no need to just call the parent it already exists * finalize the renaming * list cannot be default * allow sampling in dflash * add doc and public inits * update repo name * assistant checkpoint name as well * skip --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: raushan <raushan@huggingface.co> Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Raushan Turganbay <raushan.turganbay@alumni.nu.edu.kz> Co-authored-by: Benjamin Burtenshaw <benjamin_burtenshaw@users.noreply.github.com> Co-authored-by: Beto de Paola <betodepaola@meta.com> Co-authored-by: albertodepaola <albertodepaola@gmail.com> Co-authored-by: ruanrms <ruanslv@gmail.com> Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> Co-authored-by: Aritra Roy Gosthipaty <aritra.born2fly@gmail.com>
Author
Parents
Loading