PR #4203 Token-Based Chunking Support

Token-Based Chunking Support #4203

badGarnet merged 12 commits into Unstructured-IO:main from eureka928:feat/token-based-chunking

eureka928 force pushed from 40a820c2 to c0ab0fed 62 days ago

badGarnet commented on 2026-01-22

feat: add tiktoken dependency for token-based chunking

725bd1ae

feat: add token-based chunking support to ChunkingOptions

1a02540e

feat: add token-based parameters to chunk_by_title()

91eb0482

feat: add token-based parameters to chunk_elements()

690236ba

test: add tests for token-based chunking

e0d0e14c

docs: update CHANGELOG for token-based chunking feature

75a1281e

fix: remove duplicate TokenCounter imports in tests

333dad79

style: apply black formatting

bde6a052

fix: use token-based overlap in token chunking mode

6da7ca9c

eureka928 force pushed from 7e161ec6 to 6da7ca9c 61 days ago

fix: align regex version in extra-chunking-tokens.txt with base const…

1c89138f

eureka928 requested a review from

badGarnet 61 days ago

badGarnet commented on 2026-01-22

test: use exact text assertions in token-based chunking tests

7ec3505b

badGarnet commented on 2026-01-22

chore: bump version to 0.18.31-dev1

91f6a24a

badGarnet approved these changes on 2026-01-22

badGarnet merged 01c3f7c2 into main 61 days ago

Reviewers

badGarnet

Assignees

No one assigned

Labels

None yet

Milestone

No milestone