Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologies

Using Large Language Models (LLMs) to generate database queries is an area of active research. In [4], Sequeda et al. argue that knowledge graphs (KGs) with rich ontologies can enable an LLM to answer queries of enterprise complexity, noting that text-to-SQL benchmarks such as Spider [6] are not tailored to such queries. In addition to query complexity, an equally challenging problem in the enterprise setting is schema complexity, where the ontology itself is large and complex. This paper contributes an approach to using off-the-shelf LLMs and enterprise-scale ontologies to answer natural language questions on large data sets. We address the schema complexity problem by incrementally revealing "just enough" of an ontology that is needed to answer a given question.