unstructured

ceb2e72a - fix(metrics): read evaluation files as UTF-8 on every platform (#4623)

fix(metrics): read evaluation files as UTF-8 on every platform (#4623) ## Summary The metrics package opens ground-truth and output files without an `encoding`, so Python decodes them with the locale encoding: UTF-8 on Linux and macOS, but usually cp1252 on Windows. The files themselves are UTF-8: ground-truth `.txt` files, and JSON saved with `ensure_ascii=False`, such as unstructured-ingest output and the repo's own table fixture, keep raw UTF-8 bytes, and `elements_from_json()` already reads JSON as UTF-8. So on Windows the same evaluation gives different results: - **Text-extraction scores are off.** `TextExtractionMetricsCalculator` reads every ground-truth `.txt` (and, with `document_type="txt"`, the documents too) through `_read_text_file()`. Curly quotes, accents, dashes, bullets and CJK text come back as mojibake (`It’s a café` becomes `It’s a café`) and are compared with correctly decoded output, so `cct-accuracy` drops and `cct-%missing` rises. The quote normalisation in `text_extraction.py` can't help, because the quotes are already mangled when it runs. - **Table documents are dropped.** `TableEvalProcessor.from_json_files()` loads JSON with `open()` and `json.load()`. A UTF-8 file that contains a byte cp1252 leaves undefined (0x81, 0x8D, 0x8F, 0x90 or 0x9D, as in the closing quote `”`) raises `UnicodeDecodeError`. `_try_process_document()` logs it, and `TableStructureMetricsCalculator` leaves the document out of its results. Every `open()` that reads an evaluation file now passes `encoding="utf-8"`: `_read_text_file()` (keeping `errors="ignore"`), `TableEvalProcessor.from_json_files()`, `ObjectDetectionEvalProcessor.from_json_files()` and `ObjectDetectionPerClassMetricsCalculator._set_supported_metrics()`. The object-detection readers had the same pattern. The object-detection dumps unstructured writes are ASCII (`json.dumps` escapes non-ASCII), so nothing changes for those files; the readers are just consistent now. Nothing changes on Linux or macOS with a UTF-8 locale. On Windows, files that are not UTF-8 are now handled the way Linux already handles them: undecodable bytes are dropped from `.txt` files, and a non-UTF-8 JSON file raises `UnicodeDecodeError`. The repo's own fixtures in `example-docs/test_evaluate_files` show the problem. These numbers come from Python 3.12 on Windows with the default cp1252 locale. The "this PR" column is also what `main` prints with `PYTHONUTF8=1`, which is what Linux and macOS report: | calculator: document (output type) | main | this PR | | --- | --- | --- | | text extraction: `Performance-Audit-Discussion.pdf` (json) | accuracy 0.965, missing 0.014 | 0.970, 0.001 | | text extraction: `Performance-Audit-Discussion.pdf` (txt) | 0.970, 0.003 | 0.970, 0.001 | | text extraction: `Bank Good Credit Loan.pptx` (txt) | 0.853, 0.105 | 0.863, 0.092 | | text extraction: `currency.csv` (txt) | 0.918, 0.027 | 0.956, 0.027 | | table structure: `IRS-2023-Form-1095-A.pdf` | not in the results (`'charmap' codec can't decode byte 0x9d in position 24503`) | 2 tables, `table_level_acc` 0.278 | Documents whose files are plain ASCII score the same before and after. To see it on Windows: ```python from pathlib import Path from unstructured.metrics.utils import _read_text_file Path("gt.txt").write_text("It’s a café", encoding="utf-8") print(_read_text_file("gt.txt")) # main: It’s a café this PR: It’s a café ``` ## Validation Two new tests replace `open` in the module under test so that a call without `encoding` decodes cp1252, as it does on Windows. They fail on any platform without the fix: - `test_unstructured/metrics/test_utils.py::test_read_text_file_decodes_utf8_regardless_of_locale` - `test_unstructured/metrics/test_table_structure.py::test_table_eval_processor_from_json_files_decodes_utf8_regardless_of_locale` ``` $ pytest test_unstructured/metrics/test_utils.py test_unstructured/metrics/test_table_structure.py -k "test_utils or decodes_utf8" # main source + the new tests FAILED test_utils.py::test_read_text_file_decodes_utf8_regardless_of_locale - It’s a café — “naïve” résumé. 中文 + It’s a café — “naïve†résumé. 中文 FAILED test_table_structure.py::test_table_eval_processor_from_json_files_decodes_utf8_regardless_of_locale UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 94: character maps to <undefined> 2 failed, 6 passed # this PR 8 passed ``` `pytest test_unstructured/metrics` on Windows (Python 3.12, default cp1252 locale): - `main`: 7 failed, 153 passed. Two of the failures are this bug, and both pass on `main` with `PYTHONUTF8=1`: `test_evaluate.py::test_table_structure_evaluation` (`assert 1 == 2`, the `IRS-2023-Form-1095-A.pdf` row is missing) and `test_process_document_returns_the_correct_amount_of_values[TableStructureMetricsCalculator-...]` (`UnicodeDecodeError ... byte 0x9d`). - this PR: 5 failed, 157 passed. The two tests above and the two new tests pass. The other five fail the same way on `main` on this machine and don't involve the changed readers: three need tesseract or poppler, and the two `fake-email.txt` cases in `test_element_type.py` also fail on `main` with `PYTHONUTF8=1`. Running both calculators on the fixtures in the table with `PYTHONUTF8=0` and with `PYTHONUTF8=1` gives identical output with this PR. On `main` the two runs differ as shown in the table. `ruff check .` and `ruff format --check .` pass (ruff 0.15.10, the version in `uv.lock`). ## Release note (optional) The evaluation metrics now read ground-truth and output files as UTF-8 on every platform. On Windows, text-extraction scores for files with non-ASCII text now match Linux and macOS, and table-structure evaluation no longer skips UTF-8 JSON documents that contain characters such as a closing curly quote. Evaluation inputs are read as UTF-8: convert ground-truth files saved in another encoding (for example cp1252) to UTF-8 before evaluating. ## Checklist - [x] The PR title is a user-facing release-note entry with a conventional commit prefix. - [x] Apply a category label (`enhancement`, `bug`, or an appropriate existing label), or ask a maintainer to apply it. This is a `bug` fix; could a maintainer please apply the `bug` label? I can't set labels on this repository. - [ ] For breaking changes, use `!` in the title and explain impact and migration above. (Not a breaking change.) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4623?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <!-- End of auto-generated description by cubic. -->