PR #5847 Support special tokens as reverse/anti prompt.

Support special tokens as reverse/anti prompt. #5847

ggerganov merged 3 commits into ggml-org:master from dranger003:fix_antiprompt_special

Support special tokens as reverse/anti prompt.

ef651247

ggerganov commented on 2024-03-03

Tokenize antiprompts only once.

60325ec7

main : minor

98dae326

ggerganov approved these changes on 2024-03-04

ggerganov merged 5a51cc1b into master 1 year ago

dranger003 deleted the fix_antiprompt_special branch 1 year ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone