PR #30492 Add missing Flaubert tokenizer tests

Add missing Flaubert tokenizer tests #30492

ArthurZucker merged 6 commits into huggingface:main from bastrob:feature/flaubert_tokenizer_test

amyeroberts added Core: Tokenization

ArthurZucker requested a review from

itazap 2 years ago

itazap commented on 2024-05-21

itazap approved these changes on 2024-05-21

bastrob force pushed 2 years ago

ArthurZucker requested a review from

ArthurZucker 2 years ago

ArthurZucker approved these changes on 2024-05-24

add flaubert tokenization test, enrich inheritance in FlaubertTokenizer.

54305c99

fix quality code ci

52f889fb

ensure parameter consistency

5ee165f9

fix ci

8377fc90

fix copyright year and flatten vocab list.

127a5eba

bastrob force pushed to 127a5eba 2 years ago

fix style

8e8f5005

ArthurZucker merged 464d986b into main 2 years ago

Reviewers

ArthurZucker

itazap

Assignees

No one assigned

Labels

Core: Tokenization

Milestone

No milestone