PR #7332 tokenization: add warning for double BOS

tokenization: add warning for double BOS #7332

JohannesGaessler merged 1 commit into ggml-org:master from JohannesGaessler:tokenize-bos-warning

mofosyne added enhancement

mofosyne added Review Complexity : Low

ggerganov approved these changes on 2024-05-17

tokenization: add warning for double BOS

6520c7a1

JohannesGaessler force pushed from c7f00fcc to 6520c7a1 1 year ago

JohannesGaessler merged 29c60d8c into master 1 year ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

enhancement Review Complexity : Low

Milestone

No milestone