unstructured
perf: avoid spaCy statistical pipeline for word tokenization
#4408
Open
Open
perf: avoid spaCy statistical pipeline for word tokenization #4408
KRRT7
wants to merge
7 commits into
Unstructured-IO:main
from
KRRT7:perf-tokenizer-only
perf: tokenize words without running spaCy pipeline
3a88532e
perf: add text partition benchmark
ab5e888b
test: cover tokenizer-only behavior
efe77c7f
docs: add tokenizer performance changelog
722584c0
cubic-dev-ai
commented
on 2026-07-17
cubic-dev-ai
commented
on 2026-07-17
build(version): bump to 0.25.2-dev0
956fadbb
cubic-dev-ai
commented
on 2026-07-17
cragwolfe
approved these changes
on 2026-07-22
fix benchmark aggregate median
7fb89c06
cubic-dev-ai
commented
on 2026-07-23
fix parallel metrics evaluation test
8e06546a
cubic-dev-ai
commented
on 2026-07-23
Login to write a write a comment.
Login via GitHub