PR #1567 Bugfix: eos and bos tokens positions are inconsistent

Bugfix: eos and bos tokens positions are inconsistent #1567

OlivierDehaene merged 1 commit into huggingface:main from amihalik:bugfix-eos-bos-token-issue

Bugfix: eos and bos tokens positions are inconsistent

1e51fdc1

OlivierDehaene approved these changes on 2024-02-16

OlivierDehaene merged 142cdabe into main 2 years ago

Reviewers

OlivierDehaene

Assignees

No one assigned

Labels

None yet

Milestone

No milestone