feat: add elasticsearch destination connector #2152
ahmetmeleq
merged
52 commits into
main
from
ahmet/feat/elasticsearch-destination-connector
trials on es destination
61dc14b9
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
61012b43
ahmetmeleq
changed the title
remove file
a8328335
add writer, destination connector classes, cli cmd, test, test helper…
59ad89e0
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
6aa34646
add test, add ingest cmd
21e93e5f
update conform_dict
a0825e3e
fixes on cli params, index creation and ingest test
f66c3732
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
735e16d1
path fixes for the source connector tests
643c8891
cli batch size name fix
9e3bd9b4
fix field's default for source and destination connector, fix integer…
a1de50a4
refactor mapping schema
bbec4172
fix test (success)
2a2a6113
add memory efficient processing
26db6fb9
add parallel bulk upload (3.2Mb * 10(files) -- 1 thread: 51s, 12 thre…
ae4a193e
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
96f58551
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
bab4492a
makefile path fix for pinecone install
9d909676
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
2e23c04d
update arguments, properties and runner based on improve sensitive da…
f45233a1
Merge branch 'ahmet/feat/elasticsearch-destination-connector' of http…
ff33ada0
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
d0c24656
update code based on audit PR, add auth vars for dest connector scrip…
61106bef
test chunking and embedding
148db64d
implement check_connection, tidy log
3d543603
rename var for es entry id
d993dc0c
implement generate_client, fix check_connection, replace prints with …
acffe517
remove redundant conform, reformat check_connection
539e5f55
expose batch size in bytes as a cmd arg to the user
4636e5b6
remove dependency to ingest for helper scripts
c2d2e4ea
assign variables to (actually safe) credentials so that they do not a…
f086a2ea
remove comment, add log on num_processes for upload
c495f107
changelog and version
ac8ee1d3
shellcheck
1a558af8
add examples for source and destination connector
cf8327fd
add docs for destination connector
1a26dc64
potter-potter
commented
on 2023-12-12
fixes on credentials
f8052181
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
82bfb4f9
shellcheck
de172756
Merge branch 'ahmet/feat/elasticsearch-destination-connector' of http…
9d0ddeda
shfmt
0f5e77ce
fixes on es source connector for env vars
95117418
potter-potter
commented
on 2023-12-13
potter-potter
commented
on 2023-12-13
potter-potter
commented
on 2023-12-13
potter-potter
commented
on 2023-12-13
potter-potter
commented
on 2023-12-13
remove kwargs, inherit fields, add num-processes for upload to docs
04fe747b
fix example
52b4ac9b
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
00710a2c
potter-potter
commented
on 2023-12-16
potter-potter
approved these changes
on 2023-12-16
remove unused method, update schema
e2fe2c56
Merge branch 'ahmet/feat/elasticsearch-destination-connector' of http…
569584ab
Merge branch 'main' into ahmet/feat/elasticsearch-destination-connector
e0e697a3
shfmt
842ea2e3
move util function
b2419274
make credentials variables
c2df38d1
Reviewers
Assignees
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub