A Generalized LLM-Augmented BIM Framework: Application to a Speech-to-BIM system

As large language models (LLMs) rapidly evolve into large multimodal models (LMMs), the integration of these technologies into building information modeling (BIM) tasks to enhance work performance is signiLicantly increasing. The use of generative artiLicial intelligence (AI) during the conceptual design phase is particularly becoming a norm in industry and academia. A recent survey by the Royal Institute of British Architects (RIBA) reported that 68% of the responding architects are already using generative AI, such as text-to-image models, for early design visualization While the application of LLMs in BIM tasks beyond the early design phase is still in an early stage, it is foreseeable that BIM systems with natural language interfaces supported by LLMs will supplant BIM tools with traditional user interfaces in the near future. In this paper, we use the term "LLM-augmented BIM" as a general expression to indicate a task or a process of querying, generating, and managing BIM data and/or models via speech or text in natural language. We refer to the former as "speech-to-BIM" and the latter as "text-to-BIM" tasks.