PR #14848 Fix token text splitter duplicates

Fix token text splitter duplicates #14848

hwchase17 merged 2 commits into langchain-ai:master from sweetkane:fix-tokenTextSplitter-overlaps

add break condition and unit test

b6901ab4

formatting

cb885fe9

dosubot added size:S

dosubot added area: models

dosubot added bug

hwchase17 approved these changes on 2023-12-19

dosubot added lgtm

hwchase17 merged ea331f31 into master 2 years ago

Reviewers

hwchase17

Assignees

No one assigned

Labels

bug

Milestone

No milestone