transformers
[`LlamaTokenizer`] make unk_token_length a property
#25689

Merged

[`LlamaTokenizer`] make unk_token_length a property #25689

ArthurZucker merged 1 commit into huggingface:main from ArthurZucker:llama-unk-tok-length

make unk_token_length a property

06d5e05c

ArthurZucker requested a review from

sgugger 2 years ago

ArthurZucker changed the title ~~[`Llama`] make unk_token_length a property~~ [`LlamaTokenizer`] make unk_token_length a property 2 years ago

sgugger approved these changes on 2023-08-23

ArthurZucker merged 6e6da5e4 into main 2 years ago

ArthurZucker deleted the llama-unk-tok-length branch 2 years ago

Reviewers

sgugger

Assignees

No one assigned

Labels

None yet

Milestone

No milestone