transformers
4e441e52 - fix LayoutLMv3TokenizerFast subword label after 'Ġ' token (#21695)

Commit

2 years ago

fix LayoutLMv3TokenizerFast subword label after 'Ġ' token (#21695) LayoutLMv3TokenizerFast produces empty 'Ġ' token with `offset_mapping = (0, 0)`. Next token is wrongly assumed to also be beginning of word and isn't correctly assigned `pad_token_label`. Modify test with text that produce 'Ġ' token. Remove copy check from LayoutLMv2TokenizerFast for `_batch_encode_plus`. solves issue: #19978

References

#21695 - fix LayoutLMv3TokenizerFast subword label after 'Ġ' token

#27720 - Add common processor tests

#29969 - [SigLIP] Add fast tokenizer

#32831 - [Docs] Update resources

#33111 - [Backbone] Remove out_features everywhere

#33174 - [Zero-shot image classification pipeline] Remove tokenizer_kwargs

#39821 - Support MetaCLIP 2

#58 - Add EoMT DINOv3 model

#59 - Fix attention mask handling in EoMT-DINOv3 converter

#41212 - Add EoMT with DINOv3 backbone

#62 - Add initial DEIMv2 model implementation

Author

thibaultdouzon

Parents

a6001056

transformers 4e441e52 - fix LayoutLMv3TokenizerFast subword label after 'Ġ' token (#21695)

transformers
4e441e52 - fix LayoutLMv3TokenizerFast subword label after 'Ġ' token (#21695)