PR #681 Tiktoken wrapper add_eos_token option

Tiktoken wrapper add_eos_token option #681

rajammanabrolu merged 6 commits into main from raj/tiktoken-changes

add add_eos_token arg to tiktoken wrapper

5609d3a2

add add_eos_token arg to tiktoken wrapper

11e8ecc9

rajammanabrolu requested a review from

dakinggg 2 years ago

yapf

7798ff37

dakinggg commented on 2023-10-19

encode_plus tests

457f38d5

rajammanabrolu requested a review from

dakinggg 2 years ago

Merge branch 'main' into raj/tiktoken-changes

1a4454e0

dakinggg approved these changes on 2023-10-20

rajammanabrolu enabled auto-merge (squash) 2 years ago

Merge branch 'main' into raj/tiktoken-changes

526e7ffd

rajammanabrolu merged f65b07e8 into main 2 years ago

dakinggg deleted the raj/tiktoken-changes branch 2 years ago

Reviewers

dakinggg

Assignees

No one assigned

Labels

None yet

Milestone

No milestone