Although deep neural networks have demonstrated significant success due to their powerful expressiveness, most models struggle to meet practical requirements for uncertainty estimation. Concurrently, the entangled nature of deep neural networks leads to a multifaceted problem, where various localized explanation techniques reveal that multiple unrelated features influence the decisions, thereby undermining interpretability. To address these challenges, we develop a Bayesian N on-negative D ecision Layer ( BNDL), which reformulates deep neural networks as a conditional Bayesian non-negative factor analysis. By leveraging stochastic latent variables, the BNDL can model complex dependencies and provide robust uncertainty estimation. Moreover, the sparsity and non-negativity of the latent variables encourage the model to learn disentangled representations and decision layers, thereby improving interpretability. We also offer theoretical guarantees that BNDL can achieve effective disentangled learning. In addition, we developed a corresponding variational inference method utilizing a Weibull variational inference network to approximate the posterior distribution of the latent variables. Our experimental results demonstrate that with enhanced disentanglement capabilities, BNDL not only improves the model's accuracy but also provides reliable uncertainty estimation and improved interpretability. Over the last decade, deep neural networks (DNNs) have achieved significant success and have been widely applied across various research domains (LeCun et al., 2015). As these applications expand, quantifying uncertainty in predictions has gained importance, especially for AI safety (Amodei et al., 2016). A key goal of uncertainty estimation is to ensure that neural networks assign low confidence to test cases poorly represented by training data or prior knowledge (Gal & Ghahramani, 2016). One approach to this challenge is Bayesian neural networks, which treat model parameters as random variables.