PR #3210 Roman/fix ingest async connectors

Run partitioner always in multiprocessing mode

rbiseck3 committed 2 years ago

dynamically use async if fs class supports it for fsspec connectors

rbiseck3 committed 2 years ago

use async support in elasticsearch sdk

rbiseck3 committed 2 years ago

update weaviate to not be async since no async support in sdk yet

rbiseck3 committed 2 years ago

update embedder step to not use async until underlying clients are updated

rbiseck3 committed 2 years ago

fix dataclass issue

rbiseck3 committed 2 years ago

fix exists() call in fsspec connector

rbiseck3 committed 2 years ago

Fix issue with threadlock not pickling by having fsspec client as dataclass field

rbiseck3 committed 2 years ago

bump changelog

rbiseck3 committed 2 years ago

Update partitioner to make api calls async via run_in_executor

rbiseck3 committed 2 years ago

remove post init method on fsspec connectors that don't need to instantiate a client anymore

rbiseck3 committed 2 years ago

Add more info to README about care around async steps

rbiseck3 committed 2 years ago

Fix weaviate and chunker

rbiseck3 committed 2 years ago

Roman/fix ingest async connectors <- Ingest test fixtures update (#3211)

rbiseck3 committed 2 years ago

fix partitioning

rbiseck3 committed 2 years ago

Fix es connector

rbiseck3 committed 2 years ago

fix null checks in pipeline steps

rbiseck3 committed 2 years ago

unstructured Roman/fix ingest async connectors #3210 Merged