transformers
Optimize the speed of the truncate_sequences function.
#28263

Merged

Optimize the speed of the truncate_sequences function. #28263

ArthurZucker merged 6 commits into huggingface:main from ikkvix:fix/Optimize-speed-while-truncate_sequences

change truncate_sequences

8337462d

Update tokenization_utils_base.py

1bb67b39

change format

c9945009

fix when ids_to_move=0

fc032bbf

fix

a0c5cb58

ArthurZucker requested a review from

ArthurZucker 2 years ago

ArthurZucker approved these changes on 2024-01-09

Update src/transformers/tokenization_utils_base.py

d4e12661

ikkvix requested a review from

ArthurZucker 2 years ago

ArthurZucker merged 2a85345a into main 2 years ago

ArthurZucker removed review request from

ArthurZucker 2 years ago

ikkvix deleted the fix/Optimize-speed-while-truncate_sequences branch 2 years ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone