llama.cpp
fix : lookup word in vocab before doing BPE merges
#7193

Merged

fix : lookup word in vocab before doing BPE merges #7193

ggerganov merged 8 commits into ggml-org:master from tonyfettes:llama3-tokenizer-ignore-merge

tonyfettes marked this pull request as draft 1 year ago

tonyfettes changed the title ~~Llama3 tokenizer ignore merge~~ fix : lookup word in vocab before doing BPE merges 1 year ago

tonyfettes marked this pull request as ready for review 1 year ago

mofosyne added Review Complexity : Medium

mofosyne added bugfix

mofosyne requested a review from

goerch 1 year ago

ggerganov commented on 2024-05-10

tonyfettes force pushed from 4ba2e5c9 to 63207d17 1 year ago

ggerganov approved these changes on 2024-05-10

fix: llama-3 ignore_merges

c21d5e13

test: add test for llama-3 bpe ignore_merges

c7614930

fix: set ignore_merges only for llama-3

8a51d3b1

fix: test-tokenizer-1-bpe --ingore-merges detection

5d30a6dd

fix: copy to fix fallthrough

1fb5b558

fix: change ignore_merges to bool

c3d0f41d

fix: add ignore merges tests to cmake

0c9a0aef

tonyfettes force pushed from 0f48f9e0 to 0c9a0aef 1 year ago

ggerganov commented on 2024-05-11

llama : alternative merge ignore logic

b8d3cd53

ggerganov merged f99e1e45 into master 1 year ago

Reviewers

ggerganov

goerch

Assignees

No one assigned

Labels

bugfix Review Complexity : Medium

Milestone

No milestone