While instruction-tuned Large Language Models (LLMs) excel at extracting information from text, they are not suitable for locating texts conforming to a given description in a large document collection (semantic retrieval). Similarity search over embedding vectors does allow to perform retrieval by query, but the similarity reflected in the embedding is ill-defined and non-consistent, and is sub-optimal for many use cases. What, then, is a good query representation for effective retrieval? We identify the well defined and consistent task of retrieving sentences based on abstract descriptions of their content. We demonstrate the inadequacy of current text embeddings and propose an alternative model that significantly improves when used in standard nearest neighbor search. The model is trained using positive and negative pairs sourced through prompting a LLM. While it is easy to source the training material from an LLM, the retrieval task cannot be performed by the LLM directly. This demonstrates that data from LLMs can be used not only for distilling more efficient specialized models than the original LLM, but also for creating new capabilities not immediately possible using the original model.

The other answer here just posted text from an article on Medium. It goes over the idea of Bayesian deep networks, and lists three ways of implementing a Bayesian approach to network parameters. The first is to use Monte Carlo -- which means you have to first sample the network parameters (weights and biases), and then sample the network outputs from the inputs. That will never work at scale; you can't train anything practical that way, too slow. The second approach is to use variational inference to approximately find the right weights; but you still have to sample the weights and average in order to get the mean and variance for the network outputs, which still slows down inference, without mentioning that variational inference is approximate and often very computationally expensive. The third approach is the one that was actually proposed, that is, to use DropOut, which is hardly Bayesian in the traditional sense, whatever theoretical justification may be offered.