Neural networks (NNs) and deep learning (DL) currently provide the best solutions to many problems in image recognition, speech recognition, natural language processing, control and precision health. NN and DL make the artificial intelligence (AI) much closer to human thinking modes. However, there are many open problems related to DL in NN, e.g.: convergence, learning efficiency, optimality, multi-dimensional learning, on-line adaptation. This requires to create new algorithms and analysis methods. Practical applications both require and stimulate this development.