PR #22264 Adding Llama FastTokenizer support.

Adding Llama FastTokenizer support. #22264

Narsil merged 4 commits into huggingface:main from Narsil:fast_llama_tokenizer

Narsil requested a review from

SaulLu 3 years ago

Narsil requested a review from

sgugger 3 years ago

Narsil requested a review from

ArthurZucker 3 years ago

Narsil changed the title ~~Adding Llama FastTokenizer support.~~ [WIP] Adding Llama FastTokenizer support. 3 years ago

Narsil marked this pull request as draft 3 years ago

ArthurZucker approved these changes on 2023-03-24

Narsil force pushed 3 years ago

Narsil changed the title ~~[WIP] Adding Llama FastTokenizer support.~~ Adding Llama FastTokenizer support. 3 years ago

Narsil marked this pull request as ready for review 3 years ago

ArthurZucker commented on 2023-04-04

sgugger commented on 2023-04-04

ArthurZucker commented on 2023-04-05

sgugger commented on 2023-04-05

Adding Llama FastTokenizer support.

2c052e25

Fixing comments.

ea90a994

Narsil force pushed to ea90a994 3 years ago

Adding more to docstring.

25889122

Doc rewriting.

fbef95c2

sgugger approved these changes on 2023-04-05

Narsil merged 1670be4b into main 3 years ago

Narsil deleted the fast_llama_tokenizer branch 3 years ago

Reviewers

ArthurZucker

sgugger

SaulLu

Assignees

No one assigned

Labels

None yet

Milestone

No milestone