transformers.js
9a42cf32 - Add support for new `tokenizers>=0.2.0` BPE serialization format

Commit

233 days ago

Add support for new `tokenizers>=0.2.0` BPE serialization format

References

#545 - 🚀🚀🚀 Transformers.js V3 🚀🚀🚀

Author

xenova

xenova

Parents

Files2

src
- tokenizers.js
tests
- tokenizers.test.js