unstructured
fb2a1d42 - Jj/1798 languages warning (#1805)

Commit
2 years ago
Jj/1798 languages warning (#1805) ### Summary Closes #1798 Fixes language detection of elements with empty strings: This resolves a warning message that was raised by `langdetect` if the language was attempted to be detected on an empty string. Language detection is now skipped for empty strings. ### Testing on the main branch this will log the warning "No features in text", but it will not log anything on this branch. ``` from unstructured.documents.elements import NarrativeText, PageBreak from unstructured.partition.lang import apply_lang_metadata elements = [NarrativeText("Sample text."), PageBreak("")] elements = list( apply_lang_metadata( elements=elements, languages=["auto"], detect_language_per_element=True, ), ) ``` ### Other Also changes imports in test_lang.py so imports are explicit --------- Co-authored-by: cragwolfe <crag@unstructured.io>
Author
Parents
Loading