unstructured

445c9573 - refactor: centralize URL fetching with host validation and default timeouts (#4388)

refactor: centralize URL fetching with host validation and default timeouts (#4388) ## Summary Introduces `unstructured/safe_http.py` as a single, shared entry point for outbound URL fetches, and routes the `url=` code paths in `partition`, `partition_html`, and `partition_md` through it. Previously each of these called `requests.get` directly with inconsistent (and in some cases absent) timeout and validation behavior; this centralizes that logic in one place so it stays consistent and testable. ## What changed - **New `unstructured/safe_http.py`** — a `safe_get()` helper that all three partitioners now use instead of calling `requests.get` directly. - The helper applies, consistently: - an `http`/`https` scheme allowlist - a hostname denylist with IDNA normalization - address validation performed at TCP connect time, so the address validated is the address actually connected to - manual redirect following with per-hop re-validation, dropping credential material (`Authorization`/`Cookie`/proxy auth, plus `auth=`/`cookies=`) on cross-origin hops via requests' own `should_strip_auth` - default `(connect, read)` timeouts, and refusal of proxied requests - an opt-out via `allow_private=` / the `UNSTRUCTURED_ALLOW_PRIVATE_URL` environment variable for controlled local usage ## Behavior changes (why this is a minor release) Fetches that resolve to non-routable, loopback, or link-local addresses are now rejected by default. Outbound fetches also now carry default timeouts where some previously had none. Callers that legitimately need to reach a private/internal host can opt out with `UNSTRUCTURED_ALLOW_PRIVATE_URL=1` (or `allow_private=True`). ## Tests Adds `test_unstructured/test_safe_http.py` covering IP/hostname classification, connect-time validation, redirect re-validation, cross-origin credential stripping, and the opt-out. ## Version Bumps to `0.24.0` (see CHANGELOG) — minor, reflecting the behavior changes above.