PR #33538 Fix Llama 3 TikToken conversion

Fix Llama 3 TikToken conversion #33538

pcuenca merged 2 commits into main from fix-llama-tokenizer-conversion

Fix Llama 3 TikToken conversion

01cc9cd9

pcuenca commented on 2024-09-17

ArthurZucker commented on 2024-09-18

ArthurZucker approved these changes on 2024-09-18

No need to add tokens again

1a0506b0

itazap approved these changes on 2024-09-19

pcuenca merged 0c718f16 into main 1 year ago

pcuenca deleted the fix-llama-tokenizer-conversion branch 1 year ago

Reviewers

itazap

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone