Keyword Search using Dense Vectors & Filter Recommendation using Deep Learning and Haystack

There are several sparse approaches that produce state-of-the-art, however, there are no widespread dense vector-based approaches for achieving the same. So I framed this problem as Fuzzy matching, which is one of the useful features data scientists use for matching strings based on the character similarity most of the implementation underlying use edit distance, Levenshtein distance, however, the computational cost gets quadratic when the corpus size increases. Hence an embedding-based method significantly cut costs leveraging algorithms like Faiss, ANNOY and ANN, etc. Implementation summary, I repurposed the Siamese transformer architecture implementation by sentence transformers and by augmenting the Birkbeck spelling error corpus by Roger Mitton, like generating random n-grams from single tokens, shuffling, random deletion, then I finetuned the model, however, instead of feeding the phrases and words, I first break the sequence on char level since we are not aiming for semantic similarity, the results look promising, however, this is just a baseline, I hope to work and refine it a little further. Metadata is already data that provides further useful information about the data, in other words, it can be used to enhance human or machine's knowledge about the existing dataset. Metadata filtering can have multiple benefits as I can yield better speed for dense retrieval models as well as increasing the accuracy of Reader models.