huggingface/transformers

Pull Requests Commits

add test tokenizer without sentencepiece for huggyllama

itazap committed 2 years ago

1de64574

itazap committed 2 years ago

78bccc63

fix test and ruff

itazap committed 2 years ago

423bce29

updating feedback 2

itazap committed 2 years ago

80ba03ed

remove commont test and ruff

itazap committed 2 years ago

632a21cb

applying feedback to simplify pretokenizer and normalizer updates, + update decoder

itazap committed 2 years ago

f48f1b46

update tests to only test legacy=False

itazap committed 2 years ago

d50bbc8a

reverting sentencepiece test

itazap committed 2 years ago

d2990afe

Trigger CI attempt

itazap committed 2 years ago

83af87b5

Trigger CI attempt

itazap committed 2 years ago

20dc2bb5

removing unused imports

itazap committed 2 years ago

61fd1023

modify tokenization test to test without sentencepiece

itazap committed 2 years ago

ebf781ec

added llama and t5 tests for legacy prefix space in fast tokenizer

itazap committed 2 years ago

2e054a7a

fix case for pretokenizer

itazap committed 2 years ago

3e645791

set pretokenizer if unset

itazap committed 2 years ago

610c319b

remove t5 copy from semaless m4

itazap committed 2 years ago

fafe5b79

itazap committed 2 years ago

c36e6168

itazap committed 2 years ago

f8f8f388

itazap committed 2 years ago

406ccde2

add sentencepiece req

itazap committed 2 years ago

584f899d

fix sentencepiece case

itazap committed 2 years ago

b0d4456a

itazap committed 2 years ago

b9fd8825

itazap committed 2 years ago

0f1c5433

adding no sentencepiece test

itazap committed 2 years ago

9b8257ed

update cases for pretokenizer updating

itazap committed 2 years ago

328c12b6

consider bytelevel

itazap committed 2 years ago

19ca2af5

remove force_froms_slow

itazap committed 2 years ago

2e2db921

itazap committed 2 years ago

aeca3f1a

itazap committed 2 years ago

014d3961

itazap committed 2 years ago

50ec1225

Older