unstructured

0b7ef709 - fix: honor encoding= in partition_json and partition_ndjson (#4483)

fix: honor encoding= in partition_json and partition_ndjson (#4483) ## Problem `partition_json()` and `partition_ndjson()` read the source with a hard-coded UTF-8, and neither accepts an `encoding` parameter: ```python if filename is not None: with open(filename, encoding="utf8") as f: # hard-coded file_text = f.read() elif file is not None: file_content = file.read() file_text = file_content if isinstance(file_content, str) else file_content.decode() # UTF-8 ``` Because both signatures end in `**kwargs: Any`, an `encoding=` passed by the caller — directly, or forwarded by `partition()` — is swallowed silently. On `main` (`ee2b3a35`): ```python >>> partition_json(filename="utf16.json", encoding="utf-16") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte ``` The same call succeeds for the sibling text formats, which is where the inconsistency shows: | partitioner | takes `encoding` | reads via | |---|---|---| | `partition_text` | yes | `read_txt_file()` | | `partition_csv` | yes | passed to the reader | | `partition_json` | **no** | `open(..., encoding="utf8")` | | `partition_ndjson` | **no** | `open(..., encoding="utf8")` | UTF-16 JSON is not exotic — it is what several Windows tools emit by default, and `json.dump` to a file opened in a non-UTF-8 encoding produces it too. ## Fix Add `encoding` to both signatures and read through the existing `read_txt_file()` helper, exactly as `partition_text()` does: ```python if filename is not None: _, file_text = read_txt_file(filename=filename, encoding=encoding) elif file is not None: _, file_text = read_txt_file(file=file, encoding=encoding) file.seek(0) ``` `read_txt_file()` uses the given encoding when one is supplied and falls back to `detect_file_encoding()` when it is not, so this also makes a non-UTF-8 document readable *without* naming its encoding. The `text=` path is untouched, and `file.seek(0)` is preserved so the detect-then-partition sequence over one handle still works. Source change is +11/-8 across the two files; no new helper, no behaviour change for UTF-8 input. ## Verification ``` $ pytest test_unstructured/partition/test_json.py test_unstructured/partition/test_ndjson.py 35 failed, 75 passed ``` The 35 failures are pre-existing on `main` in this environment — they come from `unstructured/nlp/tokenize.py` failing to import an optional dependency (`ModuleNotFoundError: No module named 'installer'`), unrelated to these files. I compared the failure sets with `git stash`: **identical before and after**, and the pass count goes 70 → 75, which is exactly the five tests added here. Five regression tests, following the existing naming style in both files: - explicit `encoding="utf-16"` for `filename=` (both partitioners) - explicit `encoding="utf-16"` for `file=` (both partitioners) - no `encoding` given, UTF-16 detected (json) **Rollback proof.** Restoring just the two read blocks to the hard-coded form — leaving the new parameter in place — fails exactly those five and nothing else: ``` FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object FAILED test_json.py::it_detects_the_encoding_of_a_non_utf8_file_when_none_is_specified FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object 5 failed, 105 deselected ``` each with `UnicodeDecodeError`. `ruff check` on the four changed files is clean, and a CHANGELOG entry is added under the current `0.27.8-dev0` section. ## Notes This is the *parsing* side. There are open PRs (#4397, #4398, #4447) addressing non-UTF-8 crashes during *file-type detection* in `file_utils/filetype.py`; I checked all currently open PRs and none touches `partition/json.py` or `partition/ndjson.py`, so this does not overlap with them. I kept the scope to honoring the parameter. I did not change what happens when a document really is undecodable, nor touch the `text=` path. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4483?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->