unstructured

e5dafa41 - fix(filetype): detect the file type when the content type is application/octet-stream (#4625)

fix(filetype): detect the file type when the content type is application/octet-stream (#4625) ## Summary `application/octet-stream` is the generic "arbitrary binary data" type that servers fall back to when they don't know what a file is. GitHub raw, for example, serves PDFs that way. `partition(url=...)` passes the response's `Content-Type` to `detect_filetype()`, and on `main` such a document can't be partitioned: ```python from unstructured.partition.auto import partition url = "https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/example-docs/pdf/layout-parser-paper-fast.pdf" elements = partition(url=url, strategy="fast") # main: UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type. # this PR: 25 elements, each with metadata.filetype == "application/pdf" ``` `_FileTypeDetector._file_type_from_content_type` returns `FileType.from_mime_type(content_type)`, and `application/octet-stream` is the MIME type of `FileType.UNK`. `UNK` is an enum member, so it is truthy and ends the `or` chain in `_file_type`: MIME-type guessing (libmagic, or the `filetype` package) and the filename extension never run. Apart from the CFB/ZIP-based formats that are inspected directly before the content type is read (DOC, DOCX, EPUB, MSG, ODT, PPT, PPTX, XLS, XLSX, ZIP), `detect_filetype(file=..., content_type="application/octet-stream")` returns `UNK` the same way, and `partition(file=..., content_type="application/octet-stream")` raises, even when `metadata_filename` ends in `.pdf`. A content type that `FileType` doesn't know, such as `binary/octet-stream`, maps to `None` and is skipped, so detection already carries on for it. The content-type strategy now returns `None` when the content type maps to `FileType.UNK`, so detection carries on with the other strategies. That is what the `detect_filetype()` docstring describes for a content type that "does not map to a FileType member or is ambiguous", and it is how the MIME-guessing strategy already treats `UNK`. Content types that map to a real file type are trusted exactly as before. Left unchanged: when a caller passes `content_type="application/octet-stream"` to `partition()` itself, `augment_metadata()` still writes that asserted type to `metadata.filetype`, as `main` already does for a DOCX passed that way. `partition(url=...)` without a `content_type` argument records the detected type, as shown above. ## Validation Two new tests fail on `main` and pass with this change: - `test_unstructured/file_utils/test_filetype.py::test_it_ignores_asserted_octet_stream_content_type`: a file-like object asserted as `application/octet-stream` is detected by MIME-type guessing (PDF, HTML) or by the `metadata_file_path` extension (MD). The guessed MIME type is mocked, so the result doesn't depend on libmagic. - `test_unstructured/partition/test_auto.py::test_auto_partition_from_url_detects_file_type_when_served_as_octet_stream`: `safe_get` returns the bytes of `layout-parser-paper-fast.pdf` with `Content-Type: application/octet-stream`, and every element `partition(url=...)` returns has `metadata.filetype == "application/pdf"`. ``` # main source + the new tests FAILED test_filetype.py::test_it_ignores_asserted_octet_stream_content_type[FileType.PDF-...] - assert <FileType.UNK: 'unk'> is <FileType.PDF: 'pdf'> FAILED test_filetype.py::test_it_ignores_asserted_octet_stream_content_type[FileType.HTML-...] - assert <FileType.UNK: 'unk'> is <FileType.HTML: 'html'> FAILED test_filetype.py::test_it_ignores_asserted_octet_stream_content_type[FileType.MD-...] - assert <FileType.UNK: 'unk'> is <FileType.MD: 'md'> FAILED test_auto.py::test_auto_partition_from_url_detects_file_type_when_served_as_octet_stream - UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type. 4 failed # this PR 4 passed ``` Wider runs on Windows (Python 3.12, no libmagic, so MIME guessing uses the `filetype` package): - `pytest test_unstructured/file_utils` (without `test_file_conversion.py`, which needs pypandoc): `main` 22 failed, 539 passed; this PR 22 failed, 542 passed. The 22 failures are the same tests on both, and all of them expect libmagic's detection of text, JSON or empty files. - `pytest test_unstructured/partition/test_auto.py -k "url or filetype or content_type"`: `main` 6 failed, 30 passed; this PR 6 failed, 31 passed. The 6 failures are the same on both: the DOC, EPUB, ODT, PPT, RST and RTF cases of `test_auto_partition_applies_the_correct_filetype_for_all_filetypes`, which need LibreOffice or pypandoc. - The GitHub raw URL above: `main` raises `UnsupportedFileFormatError`; this PR returns 25 elements with `metadata.filetype == "application/pdf"`. `ruff check .` and `ruff format --check .` pass (ruff 0.15.10, the version in `uv.lock`). ## Release note (optional) `partition(url=...)` and `detect_filetype()` no longer reject a document whose content type is the generic `application/octet-stream`, such as a PDF downloaded from raw.githubusercontent.com. The file type is detected from the content or the filename extension instead. ## Checklist - [x] The PR title is a user-facing release-note entry with a conventional commit prefix. - [x] Apply a category label (`enhancement`, `bug`, or an appropriate existing label), or ask a maintainer to apply it. This is a `bug` fix; could a maintainer please apply the `bug` label? I can't set labels on this repository. - [ ] For breaking changes, use `!` in the title and explain impact and migration above. (Not a breaking change.) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4625?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <!-- End of auto-generated description by cubic. -->