llama.cpp
Validate special token ids are in range when loading GGUF model
#3635

Merged

Validate special token ids are in range when loading GGUF model #3635

ggerganov merged 8 commits into ggml-org:master from KerfuffleV2:fix-handle-bad-special-token-ids

KerfuffleV2 marked this pull request as draft 2 years ago

KerfuffleV2 changed the title ~~Valid special token ids are in range when loading GGUF model~~ Validate special token ids are in range when loading GGUF model 2 years ago

KerfuffleV2 marked this pull request as ready for review 2 years ago

cebtenzzre commented on 2023-10-16

ggerganov approved these changes on 2023-10-17

ggerganov commented on 2023-10-17

Add validation for special token ids to llama.cpp

d1075f6e

Fix BPE newline check, only I could break something so simple

14be9d91

Killll meeeeee

32383bbd

Account for GGUF_KEY_KEY only setting when the key exists

4079668c

Minor code cleanups.

22b914e0

Fix convert.py error msg when added tokens are out of range

3a007e2c

Make gguf SpecialVocab vocab size-aware

8796025b

Avoid a string copy

76b05fc4

KerfuffleV2 force pushed to 76b05fc4 2 years ago

ggerganov merged a5e7dbd6 into master 2 years ago

KerfuffleV2 deleted the fix-handle-bad-special-token-ids branch 2 years ago

Reviewers

ggerganov

cebtenzzre

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

llama.cpp Validate special token ids are in range when loading GGUF model #3635 Merged

Validate special token ids are in range when loading GGUF model #3635

llama.cpp
Validate special token ids are in range when loading GGUF model
#3635

Merged