unstructured

a3172f8e - mem: exclude unused spaCy pipeline components to reduce model memory (#4296)

mem: exclude unused spaCy pipeline components to reduce model memory (#4296) Only tok2vec, tagger, and sentence splitting are used (`pos_tag` and `sent_tokenize`). Exclude `ner`, `parser`, `lemmatizer`, `attribute_ruler` when loading `en_core_web_sm`, and add lightweight `sentencizer` to replace the dependency parser for sentence boundary detection. ## Benchmark Measured with [memray](https://github.com/bloomberg/memray) (`memray run` + `memray stats --json`), 3 rounds × 5 texts through `pos_tag()` + `sent_tokenize()` + `word_tokenize()`, Python 3.12. <img width="1400" alt="bench_spacy_exclude" src="https://raw.githubusercontent.com/codeflash-ai/codeflash/pr-assets/images/bench_spacy_exclude.png" /> ``` spaCy en_core_web_sm — component exclusion benchmark pos_tag + sent_tokenize + word_tokenize | 3 rounds x 5 texts | Python 3.12.12 Configuration Peak MB Saved % ---------------------------------------------------------------------- All components (default) 202.1MB 0.0MB 0.0% Exclude ner/parser/lemma/attr_ruler 189.3MB 12.7MB 6.3% ```