Add FastTokenizer to REALM (#15211)

Commit

4 years ago

Add FastTokenizer to REALM (#15211) * Remove BertTokenizer abstraction * Add FastTokenizer to REALM * Fix config archive map * Fix copies * Update realm.mdx * Apply suggestions from code review

References

#15211 - Add FastTokenizer to REALM

#15748 - Fix segformer reshape last stage

#19449 - [WIP] Fix weights initialization of several vision models

#27720 - Add common processor tests

#29969 - [SigLIP] Add fast tokenizer

#32831 - [Docs] Update resources

#33111 - [Backbone] Remove out_features everywhere

#33174 - [Zero-shot image classification pipeline] Remove tokenizer_kwargs

#39821 - Support MetaCLIP 2

#59 - Fix attention mask handling in EoMT-DINOv3 converter

#62 - Add initial DEIMv2 model implementation

#65 - Fix RTDetrV2 sine position embedding ordering

Author

qqaatw

Parents

021b52e7

transformers 841d9791 - Add FastTokenizer to REALM (#15211)

transformers
841d9791 - Add FastTokenizer to REALM (#15211)