Though they become the most representative intelligent systems with a dominant performance, DNNs are criticized for lacking transparency and interpretability. Better understanding the working mechanism of machine learning systems has become a requested demand, which is not only beneficial to academic research but also significant to many critical industries requiring a high level of safety concerns. In this paper, we propose a simple and interpretable disentanglement form for deep neural networks, which can not only reveal neural network's functional behaviors but also have application improvement in visual explanation task [8] and adversarial example detection [1]. The main idea is that we propose to extract the class-specific sub-network for each semantic category from a pre-trained full Figure 1: Method overview. For each class, we extract a subnetwork from the full model by learning to activate only a fraction of neurons on each layer. The extracted class-specific subnetwork can focus on one class prediction, and maintain comparable performance with the full model.