llm-foundry
Fix error when decoding a token in the id gap (or out of range) in a tiktoken tokenizer
#841

Merged

Fix error when decoding a token in the id gap (or out of range) in a tiktoken tokenizer #841

dakinggg merged 5 commits into mosaicml:main from dakinggg:tiktoken-gap

fix gap token errors

23760568

change approach

c6e716b0

update comment

46fba400

dakinggg requested a review from

irenedea 2 years ago

dakinggg requested a review from

rajammanabrolu 2 years ago

dakinggg requested a review from

sashaDoubov 2 years ago

dakinggg marked this pull request as ready for review 2 years ago

add encoding

8100f63b

another one

991a80eb

sashaDoubov approved these changes on 2024-01-08

irenedea approved these changes on 2024-01-08

dakinggg merged 5b994884 into main 2 years ago

dakinggg deleted the tiktoken-gap branch 1 year ago

Reviewers

irenedea

sashaDoubov

rajammanabrolu

Assignees

No one assigned

Labels

None yet

Milestone

No milestone