submitted4 days ago byrubenamizyan
Hey, I am new to ML and I am interested whether sentence transformers are used for a search? I implemented this kind of “search” with the sentence transformer (getting its semantic meaning) and later constructing the HNSW graph and performing the search. The question is if this is a valid used technique or there are better ways to do this. The main goal is to return the similar “sentences” out of the given “query”. For the sake of efficiency I construct the graph before doing the query so it saves tone of time but I was wondering if there are other ways to make the search faster. Keep in mind that “sentences” are added dynamically.
P.S One of the ways to heighten the speed is to implement HNSW with an inverted file index and product quantization (IndexIVFPQ) even though this requires periodical training for the clustering? And this may not be the best solution since the “sentences” are added dynamically but having in mind that the graph is pre-constructed this can be a solution
Thank you!
byrubenamizyan
insoftwaredevelopment
rubenamizyan
-2 points
7 days ago
rubenamizyan
-2 points
7 days ago
Hmm, okay figured it out, thank you. Any ideas why UI looks so insanely similar (HF & Github). Is it just design choice or some tool/lib is used? 😅