PR #3538 tokenizer : special token handling

tokenizer : special token handling #3538

ggerganov merged 11 commits into ggml-org:master from staviq:specialtokens

Rewrite special token handling from #1931

b592c70d

ggerganov commented on 2023-10-08

ggerganov commented on 2023-10-10

shorten param name, add st verification by type

fc634d87

use offsets instead of copy by substr

29e6b46e

Merge branch 'master' into specialtokens

eac5f544

cebtenzzre commented on 2023-10-11

formatting, remove copying iterator on delete

f7b1205a

Merge branch 'master' into HEAD

04ac0558

llama : normalize code-style

5c6b2be1

ggerganov marked this pull request as ready for review 1 year ago

ggerganov approved these changes on 2023-10-12

ggerganov changed the title ~~(wip) tokenizer: special token handling~~ tokenizer : special token handling 1 year ago

ggerganov added need feedback

swift fix

0f1c5695

print pfx/sfx if verb, main: split pfx input sfx

5974d617

dont add space when using special tokens

1c28116d

ggerganov requested a review from

ggerganov 1 year ago

minor : comment + spacing

fc82541b

ggerganov merged 1a159553 into master 1 year ago

slaren commented on 2023-10-17

ArthurZucker commented on 2023-10-19

Reviewers

ggerganov

slaren

ArthurZucker

cebtenzzre

goerch

Assignees

No one assigned

Labels

need feedback

Milestone

No milestone

llama.cpp tokenizer : special token handling #3538 Merged

tokenizer : special token handling #3538

llama.cpp
tokenizer : special token handling
#3538

Merged