PR #11075 Big Bird Fast Tokenizer implementation

Big Bird Fast Tokenizer implementation #11075

LysandreJik merged 19 commits into huggingface:master from tanmaylaud:feature/big-bird-fast-tokenizer

Added Big Bird Fast Tokenizer initial file

bc425720

tanmaylaud changed the title ~~Added Big Bird Fast Tokenizer initial file~~ Added Big Bird Fast Tokenizer 4 years ago

tanmaylaud changed the title ~~Added Big Bird Fast Tokenizer~~ Add Big Bird Fast Tokenizer 4 years ago

tanmaylaud changed the title ~~Add Big Bird Fast Tokenizer~~ Big Bird Fast Tokenizer implementation 4 years ago

style fixes

635219a4

flake fixes

ca0a2997

LysandreJik commented on 2021-04-06

Added big bird fast tokenizer to init files

c0d57bcd

Added big bird fast to Auto tokenization

afda7e34

fix styles

a34af2a1

minor quality fixes

a0aa18cb

Added initial test code

969a27f7

Fix SpmConverter when precompiled_charsmap doesn't exist

6e7b7a73

fixed post processor

54bb7f7c

minor style fix

7e888b65

minor fix input names

408d47fa

Actually fix identity normalization

ab94cd96

style

90356757

Merge branch 'master' of https://github.com/tanmaylaud/transformers i…

5cf8bc65

Added token type ids to fast tokenizer

e7cbfd7d

style

a2cd7ed0

flake fix

95f60e14

fix copies

3dfd113a

tanmaylaud requested a review from

LysandreJik 4 years ago

LysandreJik approved these changes on 2021-05-10

LysandreJik merged f7f87295 into master 4 years ago

Reviewers

LysandreJik

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Big Bird Fast Tokenizer implementation #11075 Merged

Big Bird Fast Tokenizer implementation #11075

transformers
Big Bird Fast Tokenizer implementation
#11075

Merged