As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets - boosting mean ROCAUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our code, a simple demo and a python package.

As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems.

License: Werelease the code used to build our benchmark and perform our experiments under theMITLicense (https://mit-license.org/),whereas werelease datawecreated, including the performance metrics collected by us, the splits used to train, validate and test our surrogate models, and our surrogate models, under the CCBY 4.0 License (https://creativecommons. Compute resources We trained the configurations on a large SLURM-based cluster with approximately 300,000 CPU-cores available in parallel. This ensures that all three data splits retain all or most of the statistical properties, including any biases, of the original performancedataset. Whereas fitting XGBoost used mean-squared-error as a regression metric, quality of fit for hyperparameters was judged using Kendall's tau rank correlation values. Task SpeedupoverHPO-only SpeedupoverNAS-only CIFAR-10 54.7 33.7 Colorectal-Histology 75.2 20.1 Fashion-MNIST 8.5 34.6 Geometricmean 32.7 28.6 resource consumption for our experiments performed on Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHztobe1.75CPU-core-hours.

Neural networks are sensitive to architecture and hyper-parameter choices [3,4]. For example, on the ImageNet dataset [5], we have observed a large increase in accuracy thanks to changes in architectures, hyper-parameters, and training algorithms, from the seminal work of AlexNet [5] to recent state-of-the-art models such as EfficientNet [6]. However, as neural networks become increasingly complex,thepotential number ofarchitecture and hyper-parameter choices becomes numerous.

As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems.We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features.Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets -- boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature.CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our code, a simple demo and a python package .