fix(chunking): don't emit a chunk that holds only the overlap when overlap_all splits an element (#4598)
## Summary
With `overlap=N` and `overlap_all=True`, chunking can emit a chunk that
holds
only the last N characters of the previous chunk, and the chunk after it
starts by repeating most of that tail. This happens in
`chunk_elements()`,
`chunk_by_title()` and `partition(..., chunking_strategy=...,
overlap_all=True)`
when an element does not fit next to the overlap prefix (the normal case
for
an element longer than `max_characters`) and the prefix is a full N
characters long, that is, the last N characters of the previous chunk do
not
start with whitespace.
Example (`max_characters=50, overlap=10, overlap_all=True`):
```python
chunk_elements(
[
Text("Lorem ipsum dolor sit amet, consectetur."),
Text("In rhoncus ipsum sed lectus porta volutpat, ut aliquet magna."),
],
max_characters=50, overlap=10, overlap_all=True,
)
```
- On `main`: `"Lorem ipsum dolor sit amet, consectetur."`,
`"nsectetur."`,
`"sectetur. In rhoncus ipsum sed lectus porta"`,
`"tus porta volutpat, ut aliquet magna."`
- With this PR: `"Lorem ipsum dolor sit amet, consectetur."`,
`"nsectetur.\n\nIn rhoncus ipsum sed lectus porta"`,
`"tus porta volutpat, ut aliquet magna."`
A repository fixture shows it too:
`partition_text("example-docs/book-war-and-peace-1p.txt",
chunking_strategy="by_title", max_characters=300, overlap=40,
overlap_all=True)`
gives two chunks that are only the 40-character overlap:
`'you--sit down and tell me all the news."'` and
`'ight between 7 and 10--Annette Scherer."'`.
Cause: `overlap_all` joins the overlap prefix to the next element with a
blank line (`"\n\n"`). When the result is too long, `_TextSplitter`
splits it
with `_split_from_maxlen()`. That function starts its separator search
at
`overlap + 1` so that the split comes out longer than the overlap (its
comment: "A split not longer than overlap will not progress"). But the
search
can match the second `"\n"` of the blank line.
`s[:match_start].rstrip()`
then strips the first `"\n"`, so the fragment is exactly the prefix
(length == overlap).
Fix, in `unstructured/chunking/base.py`: after the fragment is trimmed,
`_split_from_maxlen()` returns `"", s` when `len(fragment) <= overlap`.
The
splitter then falls back to the next separator (`" "`), and the prefix
shares
a chunk with the start of the element. This mirrors the existing guard
on the
token-counting path in `_split_by_tokens()`
(`overlap == 0 or fragment_tokens > overlap`). The fixed output has the
same
shape as the one `main` already gives when the prefix is shorter than
`overlap`: with `overlap=13` the example's stripped prefix is
`"consectetur."` (12 characters), and chunk 2 on `main` is
`"consectetur.\n\nIn rhoncus ipsum sed lectus porta"`.
Related: open PR #4392 also changes overlap splitting, but only
`_split_by_tokens()` (an infinite loop in token-based overlap). This PR
changes
only the character path (`_split_from_maxlen()` is called only from the
character branch of `_TextSplitter.__call__`), so the two do not
overlap.
## Validation
Environment: worktree on `upstream/main` (c74e6c4), Python 3.12.3,
`uv sync --locked --group test --extra docx --extra chunking-tokens`.
New tests:
-
`test_unstructured/chunking/test_base.py::Describe_TextSplitter::but_it_does_not_split_off_a_fragment_no_longer_than_the_overlap`
splits `"nsectetur.\n\nIn rhoncus ipsum sed lectus porta volutpat, ut
aliquet magna."`
with `max_characters=50, overlap=10`.
-
`test_unstructured/chunking/test_basic.py::test_it_does_not_emit_a_chunk_holding_only_the_overlap_when_overlap_all_splits_an_element`
runs the `chunk_elements()` example above.
Results:
- New tests without the fix:
`pytest
"test_unstructured/chunking/test_base.py::Describe_TextSplitter::but_it_does_not_split_off_a_fragment_no_longer_than_the_overlap"
"test_unstructured/chunking/test_basic.py::test_it_does_not_emit_a_chunk_holding_only_the_overlap_when_overlap_all_splits_an_element"`
gives 2 failed. The unit test gets `s == "nsectetur."` (only the
overlap).
The integration test gets `"nsectetur."` as chunk 2 plus one extra
chunk.
- New tests with the fix: 2 passed.
`pytest test_unstructured/chunking/test_base.py::Describe_TextSplitter`:
9 passed.
- `pytest test_unstructured/chunking/test_basic.py`: 23 passed.
- `pytest test_unstructured/chunking/test_base.py`: 334 passed, 16
skipped,
55 failed. All 55 failures are token-mode tests: `tiktoken` cannot
download
`cl100k_base` from `openaipublic.blob.core.windows.net` in my
environment
(proxy 403). They do not reach the changed function.
- `pytest test_unstructured/chunking/`: without the fix (new tests
included)
65 failed, 607 passed, 18 skipped. With the fix 63 failed, 609 passed,
18 skipped. The two failure sets differ only by the 2 new tests. Each of
the
63 remaining failures has "token" in its test id and fails on the
`tiktoken` download above.
- War and Peace fixture call above: 19 chunks before, with 2
overlap-only
chunks. 17 chunks after, none overlap-only, longest chunk 300
characters.
- Randomized check, 3000 `chunk_elements()` runs (`max_characters`
30-300,
`overlap` up to `max_characters // 2`, mixed `NarrativeText` and
`CodeSnippet`, `overlap_all=True` in about 70% of runs). Before: 4166
chunks
that are a suffix of the previous chunk, no longer than `overlap`, and
not an
input element's text. After: 0. No chunk is longer than `max_characters`
in
either run, and every run terminates.
- `make check` (ruff 0.15.10 from the `lint` group): `All checks
passed!`,
`355 files already formatted`.
- `bash scripts/test-check-new-release-version.sh`: `Passed 27 version
guard tests.`
- `VERSION_BASE_REF=upstream/main bash
scripts/check-new-release-version.sh`:
`PASS: unchanged development version 0.27.35-dev (no tag lookup).`
- No `CHANGELOG.md` or `unstructured/__version__.py` changes.
## Release note
With `overlap_all=True`, chunking no longer emits a chunk that contains
only
the overlap copied from the previous chunk when it splits a long
element. The
overlap now shares a chunk with the start of that element, so documents
chunked this way can produce fewer chunks than before.
## Checklist
- [x] The PR title is a user-facing release-note entry with a
conventional commit prefix.
- [ ] Apply a category label (`enhancement`, `bug`, or an appropriate
existing label), or ask a maintainer to apply it. A maintainer needs to
apply `bug`.
- [ ] For breaking changes, use `!` in the title and explain impact and
migration above. (Not a breaking change.)
---------
Co-authored-by: sinnercode228 <sinnercode228@users.noreply.github.com>
Co-authored-by: cragwolfe <crag@unstructured.io>