Deep Component Analysis via Alternating Direction Neural Networks

Deep convolutional neural networks have achieved remarkable success in the field of computer vision. While far from new [1], the increasing availability of extremely large, labeled datasets along with modern advances in computation with specialized hardware have resulted in state-of-the-art performance in many problems, including essentially all visual learning tasks. Examples include image classification [2], object detection [3], and semantic segmentation [4]. Despite a rich history of practical and theoretical insights about these problems, modern deep learning techniques typically rely on task-agnostic models and poorly-understood heuristics. However, recent work [5-7] has shown that specialized architectures incorporating classical domain knowledge can increase parameter efficiency, relax training data requirements, and improve performance. Prior to the advent of modern deep learning, optimization-based methods like component analysis and sparse coding dominated the field of representation learning. These techniques use structured matrix factorization to decompose data into linear combinations of shared components. Latent representations are inferred by minimizing reconstruction error subject to constraints that enforce properties like uniqueness and interpretability. Unlike feed-forward alternatives that construct representations in closed-form via independent feature detectors, this optimization-based approach naturally introduces conditional dependence between features in order to best explain data, a useful phenomenon commonly referred to as "explaining away" within the context of graphical models [8].