unstructured
ac932052 - fix(chunking): don't emit a chunk that holds only the overlap when overlap_all splits an element (#4598)

Commit
1 day ago
fix(chunking): don't emit a chunk that holds only the overlap when overlap_all splits an element (#4598) ## Summary With `overlap=N` and `overlap_all=True`, chunking can emit a chunk that holds only the last N characters of the previous chunk, and the chunk after it starts by repeating most of that tail. This happens in `chunk_elements()`, `chunk_by_title()` and `partition(..., chunking_strategy=..., overlap_all=True)` when an element does not fit next to the overlap prefix (the normal case for an element longer than `max_characters`) and the prefix is a full N characters long, that is, the last N characters of the previous chunk do not start with whitespace. Example (`max_characters=50, overlap=10, overlap_all=True`): ```python chunk_elements( [ Text("Lorem ipsum dolor sit amet, consectetur."), Text("In rhoncus ipsum sed lectus porta volutpat, ut aliquet magna."), ], max_characters=50, overlap=10, overlap_all=True, ) ``` - On `main`: `"Lorem ipsum dolor sit amet, consectetur."`, `"nsectetur."`, `"sectetur. In rhoncus ipsum sed lectus porta"`, `"tus porta volutpat, ut aliquet magna."` - With this PR: `"Lorem ipsum dolor sit amet, consectetur."`, `"nsectetur.\n\nIn rhoncus ipsum sed lectus porta"`, `"tus porta volutpat, ut aliquet magna."` A repository fixture shows it too: `partition_text("example-docs/book-war-and-peace-1p.txt", chunking_strategy="by_title", max_characters=300, overlap=40, overlap_all=True)` gives two chunks that are only the 40-character overlap: `'you--sit down and tell me all the news."'` and `'ight between 7 and 10--Annette Scherer."'`. Cause: `overlap_all` joins the overlap prefix to the next element with a blank line (`"\n\n"`). When the result is too long, `_TextSplitter` splits it with `_split_from_maxlen()`. That function starts its separator search at `overlap + 1` so that the split comes out longer than the overlap (its comment: "A split not longer than overlap will not progress"). But the search can match the second `"\n"` of the blank line. `s[:match_start].rstrip()` then strips the first `"\n"`, so the fragment is exactly the prefix (length == overlap). Fix, in `unstructured/chunking/base.py`: after the fragment is trimmed, `_split_from_maxlen()` returns `"", s` when `len(fragment) <= overlap`. The splitter then falls back to the next separator (`" "`), and the prefix shares a chunk with the start of the element. This mirrors the existing guard on the token-counting path in `_split_by_tokens()` (`overlap == 0 or fragment_tokens > overlap`). The fixed output has the same shape as the one `main` already gives when the prefix is shorter than `overlap`: with `overlap=13` the example's stripped prefix is `"consectetur."` (12 characters), and chunk 2 on `main` is `"consectetur.\n\nIn rhoncus ipsum sed lectus porta"`. Related: open PR #4392 also changes overlap splitting, but only `_split_by_tokens()` (an infinite loop in token-based overlap). This PR changes only the character path (`_split_from_maxlen()` is called only from the character branch of `_TextSplitter.__call__`), so the two do not overlap. ## Validation Environment: worktree on `upstream/main` (c74e6c4), Python 3.12.3, `uv sync --locked --group test --extra docx --extra chunking-tokens`. New tests: - `test_unstructured/chunking/test_base.py::Describe_TextSplitter::but_it_does_not_split_off_a_fragment_no_longer_than_the_overlap` splits `"nsectetur.\n\nIn rhoncus ipsum sed lectus porta volutpat, ut aliquet magna."` with `max_characters=50, overlap=10`. - `test_unstructured/chunking/test_basic.py::test_it_does_not_emit_a_chunk_holding_only_the_overlap_when_overlap_all_splits_an_element` runs the `chunk_elements()` example above. Results: - New tests without the fix: `pytest "test_unstructured/chunking/test_base.py::Describe_TextSplitter::but_it_does_not_split_off_a_fragment_no_longer_than_the_overlap" "test_unstructured/chunking/test_basic.py::test_it_does_not_emit_a_chunk_holding_only_the_overlap_when_overlap_all_splits_an_element"` gives 2 failed. The unit test gets `s == "nsectetur."` (only the overlap). The integration test gets `"nsectetur."` as chunk 2 plus one extra chunk. - New tests with the fix: 2 passed. `pytest test_unstructured/chunking/test_base.py::Describe_TextSplitter`: 9 passed. - `pytest test_unstructured/chunking/test_basic.py`: 23 passed. - `pytest test_unstructured/chunking/test_base.py`: 334 passed, 16 skipped, 55 failed. All 55 failures are token-mode tests: `tiktoken` cannot download `cl100k_base` from `openaipublic.blob.core.windows.net` in my environment (proxy 403). They do not reach the changed function. - `pytest test_unstructured/chunking/`: without the fix (new tests included) 65 failed, 607 passed, 18 skipped. With the fix 63 failed, 609 passed, 18 skipped. The two failure sets differ only by the 2 new tests. Each of the 63 remaining failures has "token" in its test id and fails on the `tiktoken` download above. - War and Peace fixture call above: 19 chunks before, with 2 overlap-only chunks. 17 chunks after, none overlap-only, longest chunk 300 characters. - Randomized check, 3000 `chunk_elements()` runs (`max_characters` 30-300, `overlap` up to `max_characters // 2`, mixed `NarrativeText` and `CodeSnippet`, `overlap_all=True` in about 70% of runs). Before: 4166 chunks that are a suffix of the previous chunk, no longer than `overlap`, and not an input element's text. After: 0. No chunk is longer than `max_characters` in either run, and every run terminates. - `make check` (ruff 0.15.10 from the `lint` group): `All checks passed!`, `355 files already formatted`. - `bash scripts/test-check-new-release-version.sh`: `Passed 27 version guard tests.` - `VERSION_BASE_REF=upstream/main bash scripts/check-new-release-version.sh`: `PASS: unchanged development version 0.27.35-dev (no tag lookup).` - No `CHANGELOG.md` or `unstructured/__version__.py` changes. ## Release note With `overlap_all=True`, chunking no longer emits a chunk that contains only the overlap copied from the previous chunk when it splits a long element. The overlap now shares a chunk with the start of that element, so documents chunked this way can produce fewer chunks than before. ## Checklist - [x] The PR title is a user-facing release-note entry with a conventional commit prefix. - [ ] Apply a category label (`enhancement`, `bug`, or an appropriate existing label), or ask a maintainer to apply it. A maintainer needs to apply `bug`. - [ ] For breaking changes, use `!` in the title and explain impact and migration above. (Not a breaking change.) --------- Co-authored-by: sinnercode228 <sinnercode228@users.noreply.github.com> Co-authored-by: cragwolfe <crag@unstructured.io>
Author
Parents
Loading