unstructured

44f9d747 - fix(security): bound quadratic array-stream decoding in is_pdf_too_complex (SEC-146) (#4437)

fix(security): bound quadratic array-stream decoding in is_pdf_too_complex (SEC-146) (#4437) ## Summary Fixes [SEC-146](https://linear.app/unstructured/issue/SEC-146/quadratic-memorycpu-in-is-pdf-too-complex-array-stream-decoding-pypdf). `is_pdf_too_complex()` in `unstructured/partition/pdf.py` re-implemented pypdf's array-based `/Contents` decoding using the same quadratic `raw_data += obj.get_data()` accumulation that pypdf patched under **CVE-2026-33123 / GHSA-qpxp-75px-xjcp** ("Inefficient decoding of array-based streams"). A crafted PDF whose page `/Contents` is an array of many small stream objects could force excessive CPU/memory. This function runs on every partitioned PDF, so it sits on the untrusted-input path (`pdf.py:309`). Per the ticket, this keeps the intentional lightweight raw-bytes approach (added in #4268 to cheaply detect vector-heavy CAD/engineering PDFs) — it does **not** switch to pypdf's expensive `ContentStream`. It just makes the accumulation efficient and bounded. ## Changes - **Accumulate into a `bytearray` with `.extend()`** instead of rebinding a `bytes` object (amortized O(1) vs O(n²)). - **Per-page caps, fail closed** (a single pathological page → treat as too complex, skip PDFMiner): - `max_raw_stream_bytes` (50 MB) — decoded bytes per page, checked *before* each stream is copied so an oversized stream is never accumulated or regex-scanned. - `max_content_stream_array_entries` (10,000, pypdf's number) — bounds an array of many empty/tiny streams that a byte cap alone misses. - **Document-level caps** so total work is a property of the function, not the page count (pages can share one indirect `/Contents` array → tiny file, unbounded scan). Charged incrementally per stream so a mid-page decode error can't discard the accounting. Set far above any plausible real document and logged at `warning`: - `max_total_stream_bytes` (1 GB) — decoded bytes per document. - `max_total_array_entries` (1,000,000) — decoded entries per document (zero-byte entries never advance the byte total but still cost a decode). - **Dereference an indirect `/Contents`** before the array check — `DictionaryObject.get` (unlike `__getitem__`) does not resolve references, so an indirect array of streams was silently skipping the array branch entirely (both the heuristic and the caps). 87 of 1,105 corpus pages reach their content array this way. - **Count operators with `finditer`** instead of `findall`, so counting no longer allocates a match list proportional to stream size. - **Bump `pypdf` to `>=6.9.1`** (from `>=6.6.2`) so the library's own code path is patched too; lock resolves 6.10.0. ## Audit (AC4) Grepped `unstructured/partition/pdf.py` and `pdf_image/` for other `+=`-on-stream-bytes loops — none found; this was the only instance. A separate O(n²) string-accumulation pattern in `unstructured/partition/html/transformations.py` (element-merge loop) is out of SEC-146's scope (not on the PDF path) and is tracked in its own ticket. ## Testing Rewrote the regression tests around real `PdfWriter`/`PdfReader` fixtures (FlateDecode-compressed streams so the file stays small while decoded output is huge — the actual attack shape). Coverage: direct + indirect graphics-heavy arrays, a many-small-streams array (~2 MB file → ~900 MB decoded) that runs for minutes / OOMs on the pre-fix code and returns in ~0.05 s here, the entry cap, the pre-copy byte cap, the cross-page byte and entry budgets, and budget survival across a mid-page decode error. The key tests were confirmed to fail on the pre-fix code. - Full `test_unstructured/partition/pdf_image/test_pdf.py`: **178 passed, 1 skipped** - `ruff check` + `ruff format --check`: clean ## Acceptance criteria - [x] `is_pdf_too_complex` accumulates via `bytearray`, not `bytes +=` - [x] Total-length bounds cap worst-case work on array-based content streams (per-page + per-document, bytes + entries) - [x] Regression test with a many-small-streams array PDF completes in bounded time/memory (fails on the old code) - [x] Repo audited for other `+=`-on-stream-bytes copies; findings fixed or ticketed - [x] `pypdf` dependency confirmed ≥ 6.9.1 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4437?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>