unstructured
c2853e4a - refactor languages parameter for pdf partition functions (#1334)

Commit
2 years ago
refactor languages parameter for pdf partition functions (#1334) ### Summary In order to support language functionality other than Tesseract OCR, we want to represent languages provided for either partitioning accuracy or OCR as a standard list of langcodes as strings. ### Details Adds `languages` (a list of strings) as a parameter to pdf partitioning functions. Marks `ocr_languages` for deprecation. Adds a new file `lang.py` for language-related helper functions. Coming up: langcode standardization, language detection ### Test Call `partition_pdf` or `partition_pdf_or_image` with a variety of strategies, languages, or `ocr_languages`. - inclusion of `ocr_languages` as a parameter should display a deprecation warning - the other valid call outputs should be no different from the current outputs. ex: ``` from unstructured.partition.pdf import partition_pdf elements = partition_pdf(filename="example-docs/DA-1p.pdf", strategy="hi_res", languages=["eng", "spa"]) print("\n\n".join([str(el) for el in elements])) ```
Author
Parents
Loading