PR #5341 py : handle byte tokens in `get_token_type`

py : handle byte tokens in `get_token_type` #5341

ggerganov merged 2 commits into master from gg/convert-fix-byte-tokens

py : handle byte tokens in `get_token_type`

ded2ad5b

cebtenzzre commented on 2024-02-05

py : fix empty bytes arg

adcf16fd

felladrin approved these changes on 2024-02-05

cebtenzzre approved these changes on 2024-02-05

ggerganov merged 906cff55 into master 1 year ago

Reviewers

cebtenzzre

felladrin

Assignees

No one assigned

Labels

None yet

Milestone

No milestone