llama.cpp
732b5fbf - convert : avoid calls to tokenizer.added_tokens_decoder (#12473)

Commit
230 days ago
convert : avoid calls to tokenizer.added_tokens_decoder (#12473) tokenizer.added_tokens_decoder returns a fresh dict every time relatively slowly (~0.04s on average) which results in massive slowdowns when we have a huge number of added tokens
Author
Parents
Loading