llama.cpp
233fc1c6 - Minor improvements in GPT2 tokenizer (#3567)

Commit

1 year ago

Minor improvements in GPT2 tokenizer (#3567) * Fixing minor bugs in bpe_gpt2_preprocess * Don't add bos token in test

References

Author

goerch

Parents

llama.cpp
tests
- test-tokenizer-0-falcon.cpp
- test-tokenizer-0-falcon.py
- test-tokenizer-0-llama.cpp
- test-tokenizer-0-llama.py