unstructured
update ingest pipeline to share ingest docs via multiprocessing.manager.dict
#1814

Merged

update ingest pipeline to share ingest docs via multiprocessing.manager.dict #1814

rbiseck3 merged 11 commits into main from roman/ingest-pipeline-fix

rbiseck3 requested a review from

ryannikolaidis 2 years ago

rbiseck3 requested a review from

ahmetmeleq 2 years ago

rbiseck3 force pushed from 4c93470e to d96ef14b 2 years ago

ryannikolaidis commented on 2023-10-20

ryannikolaidis approved these changes on 2023-10-20

ryannikolaidis requested changes on 2023-10-20

rbiseck3 force pushed from 4b50ccd5 to 3e2bfcc6 2 years ago

rbiseck3 changed the title ~~ingest pipeline fixes~~ update ingest pipeline to share ingest docs via multiprocessing.manager.dict 2 years ago

rbiseck3 requested a review from

ryannikolaidis 2 years ago

ryannikolaidis approved these changes on 2023-10-24

fix wikipedia serialization issue

3b855bab

WIP

338cc3b1

refactor to use dict to share data across steps in pipeline

28d37270

refactor pipeline to use dict to pass data around

48d863ba

remove revision var

1bb16ef3

Remove debugging statements

ec2df4aa

update changelog

d1f7834a

change order of chunking and embedding

76f35d68

Fix registry dict method to not change passed in dict

4753b8fb

rbiseck3 force pushed from 9e56f6ae to 4753b8fb 2 years ago

Fix writer

24d2ea9f

rbiseck3 enabled auto-merge 2 years ago

update ingest pipeline to share ingest docs via multiprocessing.manag…

456ce678

rbiseck3 merged 135aa659 into main 2 years ago

rbiseck3 deleted the roman/ingest-pipeline-fix branch 2 years ago

Reviewers

ryannikolaidis

ahmetmeleq

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

unstructured update ingest pipeline to share ingest docs via multiprocessing.manager.dict #1814 Merged

update ingest pipeline to share ingest docs via multiprocessing.manager.dict #1814

unstructured
update ingest pipeline to share ingest docs via multiprocessing.manager.dict
#1814

Merged