llama.cpp
73122473 - fix(gguf-py): special tokens are no longer skipped when add_<token>_token is set to false (#5487)

Comment changes are shownComment changes are hidden
Commit
1 year ago
fix(gguf-py): special tokens are no longer skipped when add_<token>_token is set to false (#5487) * fix(gguf-py): special tokens are no longer skipped when add_<token>_token is set to false * fix(gguf-py): added missing cls and mask token ids to the gguf metadata
Parents
  • gguf-py/gguf
    • File
      constants.py
    • File
      gguf_writer.py
    • File
      vocab.py