llama.cpp
common : allow raw byte in SPM vocabs; don't crash if newline token is not found
#5478

Merged

common : allow raw byte in SPM vocabs; don't crash if newline token is not found #5478

ggerganov merged 2 commits into ggml-org:master from akx:spm-no-linefeed-hack

akx marked this pull request as draft 1 year ago

akx changed the title ~~common : don't crash if newline token is not found~~ common : allow raw byte in SPM vocabs; don't crash if newline token is not found 1 year ago

ggerganov commented on 2024-02-13

akx marked this pull request as ready for review 1 year ago

ggerganov commented on 2024-02-13

common : don't crash if newline token is not found

93aed759

common : llama_byte_to_token: allow falling back to finding just the …

72b353f5

akx force pushed from dce9bc97 to 72b353f5 1 year ago

akx requested a review from

ggerganov 1 year ago

ggerganov approved these changes on 2024-02-13

ggerganov merged c4e6dd59 into master 1 year ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

llama.cpp common : allow raw byte in SPM vocabs; don't crash if newline token is not found #5478 Merged

common : allow raw byte in SPM vocabs; don't crash if newline token is not found #5478

llama.cpp
common : allow raw byte in SPM vocabs; don't crash if newline token is not found
#5478

Merged