Unstructured Semantic Model supported Deep Neural Network for Click-Through Rate Prediction

With the rapid development of online advertising and recommendation systems,click-through rate prediction is expected to play an increasingly important role. Recently many DNN-based models which follow a similar Embedding&MLP paradigm have been proposed, andhave achieved good result in image/voice and nlp fields. In these methods the Wide&Deep model announced by Google plays a key role. Most models first map large scale sparse input features into low-dimensional vectors which are transformed to fixed-length vectors, then concatenated together before being fed into a multilayer perceptron (MLP) to learn nonlinear relations among input features. The number of trainable variables normally grow dramatically the number of feature fields and the embedding dimension grow. It is a big challenge to get state-of-the-art result through training deep neural network and embedding together, which falls into local optimal or overfitting easily. In this paper, we propose an Unstructured Semantic Model (USM) to tackles this challenge by designing a orthogonal base convolution and pooling modelwhich adaptively learn the multi-scale base semantic representation between features supervised by the click label. The output of USM are then used in the Wide&Deep for CTR prediction. Experiments on two public datasets as well as real Weibo production datasetwith over 1 billion samples have demonstrated the effectiveness of our proposed approach with superior performance comparing to state-of-the-art methods.