Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning

Artificial intelligence (AI) is widely deployed to solve problems related to marketing attribution and budget optimization. However, AI models can be quite complex, and it can be difficult to understand model workings and insights without extensive implementation teams. In principle, recently developed large language models (LLMs), like GPT-4, can be deployed to provide marketing insights, reducing the time and effort required to make critical decisions. In practice, there are substantial challenges that need to be overcome to reliably use such models. We focus on domain-specific question-answering, SQL generation needed for data retrieval, and tabular analysis and show how a combination of semantic search, prompt engineering, and fine-tuning can be applied to dramatically improve the ability of LLMs to execute these tasks accurately. We compare both proprietary models, like GPT-4, and open-source models, like Llama-2-70b, as well as various embedding methods. These models are tested on sample use cases specific to marketing mix modeling and attribution. NTRODUCTION Marketing is an important function of many businesses. Historically, marketing was done primarily through a limited number of channels, such as print, radio, and linear TV advertising, but as the internet became more widely used, digital marketing increased dramatically. One of the benefits of digital marketing was the trove of additional data that companies could acquire related to the efficacy of their marketing efforts. However, drawing insights from this data requires significant effort from analysts, often utilizing a variety of different marketing software solutions. Machine learning based marketing software further complicates this effort because, while providing more sophisticated and accurate insights, it can be difficult to understand how models work. To address these difficulties, enterprise software companies often have extensive implementation teams that configure applications and explain insights in detail. This situation is not optimal because it makes deployments more expensive, and it can take a nontrivial amount of time to get questions answered as they arise.