transformers.js
6d5901e6 - Add `ignore_merges` option to BPE tokenizers (#716)

Commit

1 year ago

Add `ignore_merges` option to BPE tokenizers (#716)

References

#716 - Add `ignore_merges` option to BPE tokenizers

Author

xenova

xenova

Parents

Loading