PR #757 aggregation strategy and offset mapping

aggregation strategy and offset mapping #757

Th3G33k wants to merge 5 commits into huggingface:main from Th3G33k:ofssets-mapping

aggregation strategy and offset mapping

769adeba

offsets_mapping space delimited tokens

a817b2e8

Update tokenizers.js

56f0b922

get offsets mapping - case sensitive boolean

e73e4e8c

get offsets mapping - case sensitive boolean 2

261cb6fe

Th3G33k marked this pull request as draft 1 year ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone