PR #30081 Token healing

Token healing #30081

ArthurZucker merged 17 commits into huggingface:main from token_healing

ahmed-moubtahij marked this pull request as ready for review 1 year ago

gante approved these changes on 2024-04-18

gante requested a review from

ArthurZucker 1 year ago

gante requested a review from

amyeroberts 1 year ago

gante removed review request from

ArthurZucker 1 year ago

gante removed review request from

amyeroberts 1 year ago

gante requested a review from

ArthurZucker 1 year ago

ArthurZucker commented on 2024-05-09

ArthurZucker requested a review from

ArthurZucker 1 year ago

ArthurZucker approved these changes on 2024-05-24

token healing impl + trie with extensions

e139b0ee

make fixup

d1f920aa

prefix-robust space tokenization

afd5b002

examples readme and requirements

9d7d6405

make fixup

d5aae0f3

allow input prompt and model

5a7a0553

redundant defaults

c0a56574

Specialized Trie

f2eb4359

make fixup

fba94506

updated tests with new inherited Tree

3cfebcfd

input ids to auto device_map

7a08acd2

rm unused import

b853a103

Update src/transformers/generation/utils.py

59c0dbca

naming convention

a85041ef

Revert "naming convention"

226bee54

naming convention

333b519f

last -hopefully- changes

1beeac6c

ArthurZucker approved these changes on 2024-06-03

ArthurZucker merged 39b2ff69 into main 1 year ago

ahmed-moubtahij deleted the token_healing branch 1 year ago

Reviewers

ArthurZucker

gante

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Token healing #30081 Merged

Token healing #30081

transformers
Token healing
#30081

Merged