transformers
126eafe3 - Adding support for BPE merge creation from scores instead of ids. (#22582)

Commit

2 years ago

Adding support for BPE merge creation from scores instead of ids. (#22582) * Adding support for BPE merge creation from scores instead of ids. * Revert warn -> raise. * Update src/transformers/convert_slow_tokenizer.py * Quality.

References

#22582 - Adding support for BPE merge creation from scores instead of ids.

#27720 - Add common processor tests

#29969 - [SigLIP] Add fast tokenizer

#32831 - [Docs] Update resources

#33111 - [Backbone] Remove out_features everywhere

#33174 - [Zero-shot image classification pipeline] Remove tokenizer_kwargs

#39821 - Support MetaCLIP 2

#58 - Add EoMT DINOv3 model

#59 - Fix attention mask handling in EoMT-DINOv3 converter

#41212 - Add EoMT with DINOv3 backbone

#62 - Add initial DEIMv2 model implementation

Author

Narsil

Parents

12f1a3bb

transformers 126eafe3 - Adding support for BPE merge creation from scores instead of ids. (#22582)

transformers
126eafe3 - Adding support for BPE merge creation from scores instead of ids. (#22582)