sentence-transformers
[`feat`] Update mine_hard_negatives to using a full corpus and multiple positives
#2848
Merged
Merged
[`feat`] Update mine_hard_negatives to using a full corpus and multiple positives #2848
tomaarsen
merged
25 commits into
huggingface:master
from
ArthurCamara:hard_negatives_from_corpus
updated mine_hard_negatives method to include a seperate corpus for m…
8e5927c8
Run 'make check'
bcac9632
Update "corpus" to just a list of strings
e63938fa
Prevent duplicate embeddings if no separate corpus
0736190d
Deduplicate corpus
46ae230b
Skip rescoring positive pairs via pos_to_corpus_indices instead
aba1fb77
Add a mine_hard_negatives_from_corpus util
ecbfc3ca
Merge pull request #1 from ChrisGeishauser/mine_hard_negatives_with_c…
1d5c985a
Speedup pos_to_corpus_indices for large corpora
344b6b4e
Fix range_max by number of max_positives in dataset
2e865703
encode in chunks, ensure at least one positive per query always
330ea40b
Hard_negative_mining with corpus and multiple positives is possible
d75282f1
docstring
0962fe32
Fix for random sampling
9f00605b
fix for return_triplets=False
adcd06a5
Merge branch 'master' into hard_negatives_from_corpus
b5ce9391
Typo on list
defbfe50
Fix bug with multiple positives. More efficient creation of some tens…
33c74c68
Fix offset of positives scoring with multiple chunks
83bd3871
fix pytorch copy warning
c15c7a68
tomaarsen
commented
on 2024-09-11
tomaarsen
commented
on 2024-09-11
Only embed each text once; no need for chunking if convert_to_numpy=True
93b4d698
Undo unintended changes
691917e1
tomaarsen
changed the title
Fix mismatch in anchor/positive and negatives if multiple positives p…
5320a23f
Don't repeat positive_scores as it inflates the positive score counts
762c0fe7
Remove the "Count" for Difference as it's rather confusing
0d4d73e3
Login to write a write a comment.
Login via GitHub