llama.cpp
233fc1c6 - Minor improvements in GPT2 tokenizer (#3567)

Commit

2 years ago

Minor improvements in GPT2 tokenizer (#3567) * Fixing minor bugs in bpe_gpt2_preprocess * Don't add bos token in test

References

Author

goerch

Parents