fix: handle text too long for spacy issue (#4353)
This PR addresses the issue that very long text can fail partition
because its length exceeds `spacy`'s character limit. `spacy` is used to
classify text content. For text too long to fit under the limit we now
truncate the text and use the truncated text to represent the full text
for classification purposes.
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Prevents tokenization failures on very long inputs by truncating text
that exceeds `spacy`’s `max_length`, keeping partition/classification
stable for large documents without affecting normal cases.
- **Bug Fixes**
- Guard `_process` for inputs over `nlp.max_length`; truncate at the
last whitespace within budget, log a warning, and avoid `spacy`
ValueError E088.
- Add tests for truncation behavior and for normal processing within the
limit.
- Update version to `0.22.29` and changelog.
<sup>Written for commit 1bfdd91b99789c5333f300f8e10481250a536add.
Summary will update on new commits. <a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4353?utm_source=github">Review
in cubic</a></sup>
<!-- End of auto-generated description by cubic. -->
---------
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
