DIFER: Differentiable Automated Feature Engineering

Feature engineering, a crucial step of machine learning, aims to construct useful features from raw data to improve model performance. In recent years, great efforts have been devoted to Automated Feature Engineering (AutoFE) to replace expensive human labor. However, all existing methods treat AutoFE as an optimization problem over a discrete feature space, leading to the problems of feature explosion and computational inefficiency. Unlike previous work, we perform AutoFE in a continuous vector space and propose a differentiable method called DIFER in this paper. Specifically, we first propose an evolutionary framework to search for better features iteratively. In each feature evolution step, we introduce a feature optimizer based on the encoder-predictor-decoder, which maps features into the continuous vector space via the encoder, optimizes the embedding along the gradient direction induced by the predictor, and recovers better features from the optimized embedding by the decoder. Extensive experiments on classification and regression datasets demonstrate that DIFER can significantly outperform the state-of-the-art AutoFE method in terms of both model performance and computational efficiency.