Layer Sparsity in Neural Networks