unstructured
feat: add pinecone destination connector
#1774
Merged
Overview
Commits
118
Changes
View On GitHub

feat: add pinecone destination connector #1774

ahmetmeleq merged 118 commits into main from ahmet/pinecone-connector
ahmetmeleq
ahmetmeleq ahmetmeleq added ingest
ahmetmeleq ahmetmeleq assigned ahmetmeleq ahmetmeleq 2 years ago
ahmetmeleq ahmetmeleq marked this pull request as ready for review 2 years ago
ahmetmeleq ahmetmeleq requested a review from ryannikolaidis ryannikolaidis 2 years ago
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
ahmetmeleq
ahmetmeleq ahmetmeleq changed the base branch from main to CORE-1503-dont-use-partition 2 years ago
ahmetmeleq ahmetmeleq changed the base branch from CORE-1503-dont-use-partition to main 2 years ago
ahmetmeleq ahmetmeleq force pushed from 003f89ff to a3735890 2 years ago
ahmetmeleq add index creation script
0389b527
ahmetmeleq rebase off main for the changes in ingest cli
f5fe2a2f
ahmetmeleq trials on bugfix
655ffb6d
ahmetmeleq fix dependency name
33f5054f
ahmetmeleq apply roman's updates to pinecone
473f73c6
ahmetmeleq trials on pinecone example
0ee9e6bd
ahmetmeleq serially batched upsert with embeddings issue workaround
c6b1dc59
ahmetmeleq parallelized upsert with session handles
4bb0b1b5
ahmetmeleq skip chunking to avoid missing embeddings, remove zipping (another wo…
22394949
ahmetmeleq fix for logging error
d781698e
ahmetmeleq alphabetic order setup.py
6a96193b
ahmetmeleq add docs
0289dcc0
ahmetmeleq docs
4bec171e
ahmetmeleq docs
10fb5e76
ahmetmeleq rearrange imports
5a3975c9
ahmetmeleq add dependencies
397328f3
ahmetmeleq update example
1f6aacb5
ahmetmeleq add tests
ab73a497
ahmetmeleq add pinecone ingest test
d040010f
ahmetmeleq obfuscate embedding api keys
03b32bc8
ahmetmeleq update pinecone cli based on the new cli rebase
9895077a
ahmetmeleq shellcheck
9b3096e3
ahmetmeleq changelog and version
099fc4fb
ahmetmeleq linting
d1b1045b
ahmetmeleq linting
4fb14b01
ahmetmeleq linting
5c33688c
ahmetmeleq fix chunking node logs
b1069d4f
ahmetmeleq remove redundant secret from test fixtures update pr job
67ccfaf6
ahmetmeleq remove redundant helper script
9dbad768
ahmetmeleq remove redundant comments in test
2470dc7f
ahmetmeleq update example
2e4dda21
ahmetmeleq fix log in pipeline embedding node
ae8598ec
ahmetmeleq change pinecone batching size
15d2459f
ahmetmeleq add debugging tip
0c28c179
ahmetmeleq update ingest test with chunking
5f39a648
ahmetmeleq update example with chunking
f307a0a7
ahmetmeleq organize requirements
daeecf98
ahmetmeleq update expected uploads based on the updates in main
134a8bf9
ahmetmeleq ahmetmeleq force pushed from 3c05cecc to 134a8bf9 2 years ago
ahmetmeleq session handle fix
9c12d3bc
ahmetmeleq ahmetmeleq enabled auto-merge 2 years ago
rbiseck3
rbiseck3 commented on 2023-10-27
rbiseck3
rbiseck3 commented on 2023-10-27
rbiseck3
rbiseck3 commented on 2023-10-27
rbiseck3
rbiseck3 commented on 2023-10-27
rbiseck3
rbiseck3 commented on 2023-10-27
rbiseck3
rbiseck3 commented on 2023-10-27
ahmetmeleq doc, comment and logging updates
6d84efc0
ahmetmeleq test and session creation updates
84e65e53
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
8d7612e3
ahmetmeleq update for cli changes
5a438b8c
rbiseck3
rbiseck3 commented on 2023-11-02
rbiseck3
rbiseck3 commented on 2023-11-02
ahmetmeleq do not exclude metadata
cf773158
ahmetmeleq multiple attempts for testing
17c724bc
ahmetmeleq fix path typos on setup.py
02577697
ryannikolaidis
ryannikolaidis commented on 2023-11-03
ryannikolaidis
ryannikolaidis commented on 2023-11-03
ryannikolaidis
ryannikolaidis commented on 2023-11-03
ryannikolaidis
ryannikolaidis commented on 2023-11-03
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
fec22635
ahmetmeleq ahmetmeleq changed the title feat: pinecone destination (downstream) connector feat: add pinecone destination connector 2 years ago
ahmetmeleq reorder test, update path in test
830e3878
potter-potter
potter-potter commented on 2023-11-05
potter-potter
potter-potter commented on 2023-11-05
potter-potter
potter-potter commented on 2023-11-06
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
5ef236f2
ahmetmeleq setup py changes from main
ca949471
ahmetmeleq ingest test uses huggingface embedder
791bf03d
ahmetmeleq remove comment
f434c3d7
ahmetmeleq add secret to test_ingest_dest job
b7345c76
disabled auto-merge 2 years ago
Manually disabled by user
ahmetmeleq make batch size a parameter
766d4853
ahmetmeleq
ahmetmeleq bugfix on chunking params and implementing related test
69e1949a
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
3fd8c628
ahmetmeleq pass metadata fields individually
f2786ce1
ahmetmeleq Merge branch 'ahmet/pinecone-connector' of https://github.com/Unstruc…
d1b1cd28
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
c945851d
ahmetmeleq implement check_connection
4b49fbd2
ahmetmeleq ahmetmeleq requested a review from ryannikolaidis ryannikolaidis 2 years ago
ahmetmeleq ahmetmeleq requested a review from potter-potter potter-potter 2 years ago
potter-potter
potter-potter commented on 2023-11-09
ryannikolaidis
ryannikolaidis commented on 2023-11-09
ryannikolaidis
ryannikolaidis commented on 2023-11-09
ryannikolaidis
ryannikolaidis commented on 2023-11-09
ryannikolaidis
ryannikolaidis commented on 2023-11-09
ahmetmeleq expose writer num_processes, apply parallelization in ingest test
1ff1fd66
ahmetmeleq fix session handles
007ad367
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
b4b858ab
ahmetmeleq logging updates
3d81cfdb
ahmetmeleq changelog and version
fac751e6
ahmetmeleq random index names to avoid test run collisions
35be64a3
ahmetmeleq re-add --chunk-new-after-n-chars
0440eb25
ahmetmeleq add support for new_after_n_chars
1e8f34e9
ahmetmeleq check existence of num_processes (dest) when logging
e700a75e
ahmetmeleq update docs
80fed3bd
potter-potter
ahmetmeleq update example and docs
00b123e1
ahmetmeleq ahmetmeleq requested a review from ryannikolaidis ryannikolaidis 2 years ago
ahmetmeleq ahmetmeleq requested a review from potter-potter potter-potter 2 years ago
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
1ead5e02
ahmetmeleq changelog
bac73f02
ahmetmeleq fix typo in example
1e6ff4c3
ahmetmeleq index creation retry logic for when another index is being deleted in…
c44ab12a
ahmetmeleq index creation retry logic for when another index is being deleted in…
94e66b3e
ahmetmeleq Merge branch 'ahmet/pinecone-connector' of https://github.com/Unstruc…
be87dd42
ahmetmeleq Merge branch 'ahmet/pinecone-connector' of https://github.com/Unstruc…
567ed4ea
ahmetmeleq Merge branch 'ahmet/pinecone-connector' of https://github.com/Unstruc…
074c1cac
ahmetmeleq
ahmetmeleq update project variables, update sleep amounts
9adefb7e
ahmetmeleq update docs
65fce1c9
ahmetmeleq update docs
3a16a087
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
362eb811
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
rbiseck3
rbiseck3 commented on 2023-11-16
ahmetmeleq update docs
c3266f0b
ahmetmeleq Merge branch 'ahmet/pinecone-connector' of https://github.com/Unstruc…
8a6a0cbc
ahmetmeleq remove download_dir, remove index creation loop
387a7ad9
ahmetmeleq update example
f884123c
ahmetmeleq pythonic approach in docs
cbd734f4
ahmetmeleq update log
fa083ff7
ahmetmeleq move upsert method
29758f88
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
14d4e511
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
ahmetmeleq shellcheck
5723467c
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
992b60d2
rbiseck3
rbiseck3 commented on 2023-11-20
ahmetmeleq Update docs/source/ingest/destination_connectors/pinecone.rst
49391134
rbiseck3
rbiseck3 commented on 2023-11-20
rbiseck3
rbiseck3 commented on 2023-11-20
rbiseck3
rbiseck3 commented on 2023-11-20
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
9812d932
ahmetmeleq version
b6e97733
ahmetmeleq s3 docs pythonic approach and local connector
4f65b496
ahmetmeleq add comment on why we use random rather than uuidgen
738d75c8
ahmetmeleq check if test variables are defined before setting
fe818e47
ahmetmeleq shellcheck double quotes
937bdfa9
ahmetmeleq update parent classes for cliconfig
a2b2fc37
ahmetmeleq different number of processes for processor and writer in test
940f72d5
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
e51b88f2
ahmetmeleq
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
ahmetmeleq add comment, add field selection from element, add list items separat…
35701dcd
potter-potter
potter-potter commented on 2023-11-25
rbiseck3
rbiseck3 commented on 2023-11-27
rbiseck3
rbiseck3 commented on 2023-11-27
rbiseck3
rbiseck3 commented on 2023-11-27
rbiseck3
rbiseck3 commented on 2023-11-27
rbiseck3
rbiseck3 commented on 2023-11-27
ahmetmeleq walrus syntax := instead of if [-z $...] for default parameters
12783796
ahmetmeleq better type checking for session handles
0ec7cae1
potter-potter
potter-potter commented on 2023-11-28
ahmetmeleq implement check_connection
7b9e02b4
potter-potter
potter-potter commented on 2023-11-28
ahmetmeleq move log for number of (upload) processes from pipeline to connector
1cf12903
ahmetmeleq update embedding docs to have embedding prepend for cli args
ca0785e9
ahmetmeleq add potter's flatten lists to flatten dicts
83518b0f
ahmetmeleq make all element fields indexable, add element_serialized
e1a6365f
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
ahmetmeleq ahmetmeleq requested a review from potter-potter potter-potter 2 years ago
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
f8688e5a
rbiseck3
rbiseck3 commented on 2023-11-29
ahmetmeleq unique ids for pinecone entries rather than using element ids
e353c6b2
ahmetmeleq Merge branch 'ahmet/pinecone-connector' of https://github.com/Unstruc…
7e0c7e7a
rbiseck3
rbiseck3 commented on 2023-11-29
ahmetmeleq an additional error wrapper for check connection
b54e5ce3
ahmetmeleq ahmetmeleq requested a review from rbiseck3 rbiseck3 2 years ago
rbiseck3
rbiseck3 approved these changes on 2023-11-29
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
07dfbd85
ahmetmeleq changelog and version
cebfcb67
ahmetmeleq ahmetmeleq enabled auto-merge 2 years ago
ahmetmeleq Merge branch 'main' into ahmet/pinecone-connector
5cfef3c1
ahmetmeleq ahmetmeleq enabled auto-merge 2 years ago
ahmetmeleq ahmetmeleq merged ed08773d into main 2 years ago
ahmetmeleq ahmetmeleq deleted the ahmet/pinecone-connector branch 2 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum