PR #4081 llama : restore prefix space in llama tokenizer

llama : restore prefix space in llama tokenizer #4081

cebtenzzre merged 2 commits into master from ceb/restore-prefix-space

Revert "dont add space when using special tokens"

735ffe3d

cebtenzzre requested a review from

staviq 1 year ago

do not add space prefix if the first token is special

5e899428

cebtenzzre requested a review from

shibe2 1 year ago

shibe2 approved these changes on 2023-11-15

cebtenzzre merged a6fc554e into master 1 year ago

cebtenzzre deleted the ceb/restore-prefix-space branch 1 year ago

Reviewers

shibe2

staviq

Assignees

No one assigned

Labels

None yet

Milestone

No milestone