llama.cpp
b06fbc96 - convert : import bytes_to_unicode from convert_slow_tokenizer (#26217)

Commit
8 days ago
convert : import bytes_to_unicode from convert_slow_tokenizer (#26217) bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2 in huggingface/transformers#40936, but it had already been copied into transformers.convert_slow_tokenizer in huggingface/transformers#30334 (transformers 4.54.1), so import it directly from there. Applies the same fix to chatglm.py.
Author
Parents
Loading