llama.cpp
Respect tokenizer.ggml.add_bos_token value when tokenizing
#4040

Merged

Respect tokenizer.ggml.add_bos_token value when tokenizing #4040

KerfuffleV2 merged 4 commits into ggml-org:master from KerfuffleV2:fix-respect-use-bos-token

gguf-py: gguf-dump: Respect --no-tensor flag in JSON mode.

0c201b91

Respect add_bos_token GGUF metadata value

c7ff2d55

KerfuffleV2 added enhancement

KerfuffleV2 added generation quality

KerfuffleV2 changed the title ~~Fix respect use bos token~~ Respect tokenizer.ggml.add_bos_token value when tokenizing 2 years ago

KerfuffleV2 requested a review from

Green-Sky 2 years ago

Merge branch 'master' into fix-respect-use-bos-token

eef5ae38

gguf-py: Try to fix SpecialVocab giving up too easily for the Nth time

a169862c

KerfuffleV2 requested a review from

ggerganov 2 years ago

Green-Sky approved these changes on 2023-11-17

KerfuffleV2 merged 91f64993 into master 2 years ago

Reviewers

Green-Sky

ggerganov

Assignees

No one assigned

Labels

enhancement generation quality

Milestone

No milestone