Large language models (LLMs) enable state-of-the-art semantic capabilities to be added to software systems such as semantic search of unstructured documents and text generation. However, these models are computationally expensive. At scale, the cost of serving thousands of users increases massively affecting also user experience. To address this problem, semantic caches are used to check for answers to similar queries (that may have been phrased differently) without hitting the LLM service. Due to the nature of these semantic cache techniques that rely on query embeddings, there is a high chance of errors impacting user confidence in the system. Adopting semantic cache techniques usually requires testing the effectiveness of a semantic cache (accurate cache hits and misses) which requires a labelled test set of similar queries and responses which is often unavailable. In this paper, we present VaryGen, an approach for using LLMs for test input generation that produces similar questions from unstructured text documents. Our novel approach uses the reasoning capabilities of LLMs to 1) adapt queries to the domain, 2) synthesise subtle variations to queries, and 3) evaluate the synthesised test dataset. We evaluated our approach in the domain of a student question and answer system by qualitatively analysing 100 generated queries and result pairs, and conducting an empirical case study with an open source semantic cache. Our results show that query pairs satisfy human expectations of similarity and our generated data demonstrates failure cases of a semantic cache. Additionally, we also evaluate our approach on Qasper dataset. This work is an important first step into test input generation for semantic applications and presents considerations for practitioners when calibrating a semantic cache.

The Stanford CS 224N course - Natural Language Processing with Deep Learning is known to be one of the best courses around for as evident by the title. I recently started doing the course as I wanted to do NLP for a long time now and considering that the 2019 course is going to be publicly available (http://web.stanford.edu/class/cs224n/), I thought that it might be a good idea to write a blog as I go through the assignments. In a series of stories, I'll be releasing the solutions as I go on about the course. Feel free to comment if you find a mistake or a better solution to mine.

Together they contribute 30% to your overall course mark. Both assignments are to be submitted using the submit command on DICE. You should put all of your work into one prolog file (commenting out any written sections) and submit the file before the deadline. Details of how to use the submit command will be provided at the end of each assignment. Both assignments require you to develop complete Prolog programs. You may develop these programs at home on a PC or Mac but you must test that they run under the DICE version of sicstus before submission.