unstructured

e2732cf0 - fix: stop treating every hyphen as a bullet delimiter in text partitioning (#4458)

fix: stop treating every hyphen as a bullet delimiter in text partitioning (#4458) ## Summary `-` (U+002D) and `–` (U+2013) are bullet glyphs *and* ordinary punctuation — an intra-word hyphen, an intra-number separator, a minus sign. `UNICODE_BULLETS` was reused as an **unanchored split delimiter**, so `partition_text` split at every hyphen in a qualifying line *and consumed the hyphen itself*: ```python partition_text(text="Phone: 555-123-4567\n\nEnd") # ['Phone: 555', '123', '4567', 'End'] partition_text(text="Contact: Jean-Luc Picard\n\nEnd") # ['Contact: Jean', 'Luc Picard', 'End'] ``` The only thing limiting the blast radius was the `all_lines_short` guard, which counts **space-separated tokens** — so long English prose escaped by accident of word count, while short `Field: value` lines, log lines, YAML and all CJK text (no inter-word spaces, so the count is always 1) did not. This happened at partition time, upstream of every transform, on a job that reported success. Nothing downstream recovered it: both chunkers rejoin elements with a blank line, so the fragments never became contiguous again — a regex for a phone number or an SSN can never match across an element boundary. Scope: `FileType.TXT` and its 18 extensions (`.txt .text .c .cc .cpp .cs .cxx .go .java .js .log .php .py .rb .swift .ts .yaml .yml`) plus `.eml`/`.msg`, which delegate to `partition_text`. PDF/DOCX/PPTX/HTML are unaffected — verified by call graph and by round-tripping the same content through DOCX and HTML. ## The fix Two conditions now have to hold before an ambiguous glyph is read as a bullet, and **three** call sites needed changing, not one. **1. Start of line.** `PARAGRAPH_PATTERN_RE` splits only on the unambiguous glyphs. Dash bullets at line start were already handled by the `\n` branch, so nothing is lost. **2. Followed by whitespace** (or end of text). A bullet is separated from the item it introduces; a sign is not. Without this, `-123.45` reads as a bullet and is rewritten to `123.45` — **silently flipping the value's sign**. `E_BULLET_PATTERN` in the same file already required `(?=\s)` for exactly this reason, so this brings dash handling in line with existing precedent rather than inventing a convention. Applied to: - `PARAGRAPH_PATTERN_RE` — the paragraph split. - `BULLET_SPLIT_RE_0W` (new) — used by `group_bullet_paragraph`, which splits on `UNICODE_BULLETS_RE_0W`. This site is reached for any paragraph that *starts* with a bullet, i.e. exactly a dash bullet list, so without it a genuine dash list whose items contain hyphens was still shredded. - `UNICODE_BULLETS_RE` — backs `clean_bullets`, `is_bulleted_text` and `_is_empty_bullet`. Fixing only the split sites preserves the line boundary but lets classification strip the sign anyway. `-` and `–` stay in `UNICODE_BULLETS`; the qualification is scoped to the ambiguous glyphs via a separate alternation, so `•item` and friends need no separator and are unchanged in every respect. ### Before / after ``` 'Phone: 555-123-4567' ['Phone: 555','123','4567'] -> ['Phone: 555-123-4567'] 'Contact: Jean-Luc Picard' ['Contact: Jean','Luc Picard'] -> ['Contact: Jean-Luc Picard'] 'trace_id: 550e8400-e29b…' 5 fragments -> 1 element 'Content-Type: image/jpeg' ['Content','Type: image/jpeg'] -> ['Content-Type: image/jpeg'] '-123.45' ListItem('123.45') # sign flipped -> Text('-123.45') '- readings\n-5\n-10 °C' 3 sign-stripped items -> ListItem('readings -5 °C -10 °C') ``` ## Behavior changes Both are declared in the changelog. 1. An inline dash bullet list on a single line (`- one - two - three`) no longer splits into separate elements. Ambiguous input either way, and a far better trade than destroying every hyphenated identifier. 2. A dash with no separating whitespace (`-item`) is no longer a bullet: it keeps its leading character and is no longer classified as a `ListItem`. Unchanged: `- item` at line start is still a bullet, a lone `-` is still an empty bullet, `----` is still not a bullet, and inline `•`-style bullets still split. ## Verification **1828 tests pass** — cleaners, nlp, text, email, msg, text_type, partition/common, chunking, auto, html, docx, md, documents. Nothing broke. **Corpus audit** over the 66 `.txt`/`.eml` files in `example-docs/`: | | elements | dash chars recovered | docs with text loss | | -- | -- | -- | -- | | before | 26,857 | — | — | | after | 26,524 | **+430** | **0** | The recovered characters are the point: the old split *deleted* the delimiter. Among them, wrapped URL slugs in `norwich-city.txt` were losing their leading dash (`-can-get-hucks-firing-1-648688`). **Adversarial cases** checked against the bullet regex: `-Xmx512m`, `--flag`, `-o`, `-.5`, `-1e9`, `-5`, `-item`, `----` (none are bullets); `- item`, `– item`, `-\titem`, `-`, `-\nnext`, `•item`, `○x`, `* item` (all are). **New regression tests** cover hyphenated values through `partition_text` and a new `.eml` fixture (Japanese phone, US SSN, ISO date, card, IBAN, UUID, hyphenated personal name, YAML value, log line with hostname); dash/en-dash/indented bullet lists; dash list items containing hyphens; inline `•` bullets; the Apache-License short-line case cited in `group_broken_paragraphs`; signs (`-123.45`, `–10 °C`, `-5`); and both declared behavior changes, pinned with a comment saying they are intentional. ## Notes for review Two related defects found during this work, deliberately **not** in scope: 1. `layout_list_to_list_items` (`partition/common/common.py`) has the same unanchored split on `UNICODE_BULLETS_RE`. Currently unreachable — the live `pdf.py` hi_res path passes `infer_list_items=False` and the `ocr_only` path never emits `ElementType.LIST` — but latent. 2. `all_lines_short` in `group_broken_paragraphs` counts space-separated tokens and is structurally blind to CJK, where the count is always 1. This PR makes it moot for dashes; it stays wrong for any other ambiguous delimiter. `paragraph_grouper=False` was the only workaround, and it is not reachable from the platform — the parameter is absent from the partition node's settings schema and from `unstructured_client`'s `PartitionParameters`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4458?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: badGarnet <badGarnet@users.noreply.github.com>