transformers
feature: Add robust token counting with padding exclusion
#40416
Merged
Merged
feature: Add robust token counting with padding exclusion #40416
ArthurZucker
merged
6 commits into
huggingface:main
from
PrathmeshAdsod:feature-to-count-tokens-exclude-padding
created robust token counting by using existing include_num_input_tok…
1b984718
PrathmeshAdsod
changed the title
some codebase mismatched in my local and remote, commiting to solve i…
42ead73a
Merge branch 'main' into feature-to-count-tokens-exclude-padding
fade9b0d
ci: retrigger tests
fd140287
another attemp to trigger CI for checks
7f87044a
Merge branch 'main' into feature-to-count-tokens-exclude-padding
fdda75f3
ArthurZucker
approved these changes
on 2025-09-11
Login to write a write a comment.
Login via GitHub