Domain-specific ChatBots for Science using Embeddings

Artificial intelligence and machine-learning (AI/ML) methods are growing in sophistication and capability. The application of these methods to the physical sciences is correspondingly seeing enormous growth.[1] Recent years have seen the convergence of several new trends. Generative AI seeks to create novel outputs that conform to the structure of training data,[2, 3] for instance enabling image synthesis[4-6] or text generation. Large language models (LLMs) are generative neural networks trained on text completion, but which can be used for a variety of tasks, including sentiment analysis, code completion, document generation, or for interactive chatbots that respond to users in natural language.[7] The most successful implementations of this concept--such as the generative pre-trained transformer (GPT)[8]-- exploit the transformer architecture,[9] which has a self-attention mechanism, allowing the model to weigh the relevance of each input in a sequence and capture the contextual dependencies between words regardless of their distance from each other in the text sequence. LLMs are part of a general trend in ML towards foundation models--extensive training of large deep neural networks on enormous datasets in a task-agnostic manner.[7,