Add NeoMME and NeoMME-Retriever (#47992)
* feat: add neomme
* tests: temporarily remove integration tests (need to make the weights public first)
* chore: add aurelien as co-author
Co-authored-by: uminaty <lacaurelien@yahoo.fr>
* docs: fix neomme copyright year
* docs: simplify neomme model loading
* docs: use shared patch size description
* docs: use shared image token description
* test: use configured neomme initializer
* refactor: serialize neomme layer types
* refactor: use strict neomme config validation
* refactor: modularize neomme image processor
* refactor: inline neomme resize scale
* refactor: standardize neomme image normalization
* fix: match neomme image resize rounding
* docs: clarify neomme resize defaults
* test: reject invalid neomme template tasks
* refactor: use processor input helpers
* fix: keep neomme image grid internal
* refactor: source neomme markers from tokenizer
* test: use tokenizer markers in template
* refactor: split neomme patch projection
* refactor: reuse parameter-free rms norm
* fix: remove obsolete attention mask workaround
* refactor: use standard value embeddings
* refactor: inline neomme rotary dimensions
* refactor: store neomme residual scale
* refactor: register neomme input embeddings
* refactor: remove wrapper embedding accessors
* style: move neomme shape comments
* refactor: remove redundant config assignments
* refactor: reuse base retrieval output fields
* refactor: reuse nemotron mlp
* fix: require exactly one model input
* refactor: name retrieval outputs before return
* refactor: narrow neomme weight initialization
* refactor: remove redundant retrieval helpers
* refactor: centralize neomme prompt formatting
* refactor: name neomme retrieval heads
* style: clarify neomme shape comments
* style: inline neomme shape comments
* refactor: use standard neomme rope layout
* fix: harden neomme input handling
- Preserve chat task state while forwarding processing options
- Validate image layouts, model dimensions, and rotary settings
- Clarify public documentation and expand regression coverage
* style: tighten neomme comments
* fix: validate neomme position ID shapes
* refactor: simplify neomme processor helpers
* fix: satisfy neomme repository checks
* feat: support pixel area image size bounds
- Add min_pixels and max_pixels to the shared SizeDict schema
- Move NeoMME pixel budgets into the standard size configuration
- Preserve max-side rounding while enforcing pixel floors and caps
- Validate size keys and positive bound values
- Cover serialization, resizing, rounding, and patch-count behavior
* refactor: make neomme value embeddings mandatory
* style: group neomme code by phase
* style: clarify neomme tensor comments
* style: tighten neomme test comments
* refactor: validate neomme dimensions from field definitions
* refactor: validate neomme rotary dimensions after standardization
* docs: remove duplicate neomme image size docs
* refactor: reuse standard neomme image normalization constants
* style: compact neomme image processor defaults
* refactor: validate neomme image kwargs before preprocessing
* fix: compute neomme rotary embeddings outside autocast
* docs: clarify neomme pixel area size bounds
* refactor: rename neomme residual multiplier
* refactor: remove legacy neomme rope scaling support
* refactor: validate neomme image dimensions once
* refactor: reuse laguna rotary initialization in neomme
* refactor: configure neomme windows per layer
* docs: simplify neomme config arguments
* refactor: split neomme key and value projections
* feat: add neomme image feature helpers
* perf: batch same-shape neomme images
* style: format neomme follow-up changes
* perf: batch neomme patch conversion
* refactor: keep neomme checkpoint conversion in exporter
* refactor: reuse batched image patch conversion
* fix: validate pixel area image size keys
* refactor: remove unused neomme config defaults
* refactor: reuse laguna rope frequencies
* refactor: centralize neomme input validation
* refactor: source neomme image token from tokenizer
* refactor: use standard neomme image validators
* refactor: simplify neomme patch count helper
* refactor: iterate neomme layer configs directly
* style: sync generated neomme files
* refactor: load neomme windows from config
* refactor: reuse neomme image token placeholder
* refactor: move neomme validation into jinja
* refactor: use standard neomme image hook
* refactor: use standard neomme vlm flow
* docs: use neomme chat template processing
* docs: document neomme retrieval task
* refactor: remove neomme retrieval scoring helper
* test: remove private neomme checkpoint coverage
* docs: clarify neomme retrieval scoring
* refactor: remove neomme embedding tie flag
* feat: support direct neomme retrieval inputs
* fix: support generic neomme processor inputs
* docs: refine neomme usage examples
* refactor: reuse base neomme text processing
* refactor: simplify neomme assistant mask policy
* test: cover neomme processor without chat template
* refactor: organize neomme processor methods
* refactor: simplify neomme input finalization
* docs: clarify neomme processor helpers
* refactor: simplify NeoMME processor flow
- Route retrieval formatting through apply_chat_template and delegate tokenization, padding, and keyword handling to ProcessorMixin
- Keep only NeoMME-specific image placeholder expansion, two-axis positions, and validation for task markers and truncation
- Remove direct task routing, manual sequence padding, and the duplicate query expansion processor setting
- Update documentation and tests for the template-first API, malformed layouts, truncation, and ragged image batches
- Add the PR review notes, chat template draft, and handoff context used to resolve feedback on #47992
* docs: simplify neomme retrieval example
* style: align neomme processor variable names
* fix: enforce NeoMME input and image limits
* fix: allow task-free NeoMME chat templates
* docs: simplify NeoMME encode helper
* fix: remove hardcoded NeoMME processor defaults
* style: clarify NeoMME implementation comments
* docs: pad NeoMME retrieval batches
* feat: support untied NeoMME word embeddings
* docs: update neomme model doc
* clean up a bit
* docs: improve and fix snippets model doc
* docs: clarify neomme processor docstring
* fix: define neomme per-layer defaults
* refactor: modularize neomme retrieval heads
* refactor: modularize neomme attention output
* refactor: require input ids for neomme
* refactor: make tied neomme decoder explicit
* chore: remove stale neomme fixme
* fix: sync generated neomme model
* fix: expose neomme word embeddings
* fix: validate neomme layer types
* fix: validate neomme image options
* refactor: remove neomme image option validation
* refactor: expose neomme unembedding projection
* chore: sync neomme repository files
* refactor: simplify neomme unembedding projection
* refactor: build neomme positions with torch
* refactor: require neomme input ids
* refactor: rely on neomme forward signature
* docs: update neomme contribution date
---------
Co-authored-by: uminaty <lacaurelien@yahoo.fr>
Co-authored-by: raushan <raushan@huggingface.co>
