Roman/fix ingest async connectors #3210
rbiseck3
merged
17 commits into
main
from
roman/fix-ingest-async-connectors
potter-potter
commented
on 2024-06-14
Run partitioner always in multiprocessing mode
b6c520e5
dynamically use async if fs class supports it for fsspec connectors
553c06ba
use async support in elasticsearch sdk
aa1a280e
update weaviate to not be async since no async support in sdk yet
75a47235
update embedder step to not use async until underlying clients are up…
a58732ae
fix dataclass issue
2d8051bd
fix exists() call in fsspec connector
172bd348
Fix issue with threadlock not pickling by having fsspec client as dat…
2d9ebe62
bump changelog
e907c602
Update partitioner to make api calls async via run_in_executor
64023285
remove post init method on fsspec connectors that don't need to insta…
573d21d5
Add more info to README about care around async steps
d381bfd2
Fix weaviate and chunker
0c223c4f
Roman/fix ingest async connectors <- Ingest test fixtures update (#3211)
ac7a3caf
fix partitioning
e746cf8b
Fix es connector
72601103
potter-potter
commented
on 2024-06-17
potter-potter
approved these changes
on 2024-06-17
fix null checks in pipeline steps
a288ffbc
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub