transformers
2d359f33 - fix: apply idefics3 \n\n tokenization fix to colmodernvbert processing

Commit
29 days ago
fix: apply idefics3 \n\n tokenization fix to colmodernvbert processing ColModernVBert's processing_colmodernvbert.py is generated from modular_colmodernvbert.py which inherits from idefics3. Sync the _get_num_multimodal_tokens fix (extra_split_newline) so check_modular_conversion.py passes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
Committer
Parents
Loading