Deep Learning
Unsupervised Learning Algorithms
Well,…Unsupervised learning in itself says not supervised learning. Intuitively speaking,most of human and animal learning is unsupervised learning. We are not given right answer every where. We just make decision we find to be right and try to do less mistakes next time. Just like above, unsupervised learning is the process of learning by machines without having labels (right answer) and making less errors next time(clustering does it by placing data in nearest clusters).
Deep Learning A-Z : Hands-On Artificial Neural Networks
Learn to create Deep Learning Algorithms in Python from two Machine Learning & Data Science experts. Artificial intelligence is growing exponentially. There is no doubt about that. Self-driving cars are clocking up millions of miles, IBM Watson is diagnosing patients better than armies of doctors and Google Deepmind's AlphaGo beat the World champion at Go - a game where intuition plays a key role. But the further AI advances, the more complex become the problems it needs to solve.
Genius Tool to Compare Best Time-Series Models For Multi-step Time Series Modeling
The intensity of the growth of the covid-19 pandemic worldwide has propelled researchers to evaluate the best machine learning model that could the people affected in the distant future by considering the current statistics and predicting the near future terms in subsequent stages. While different univariate models like ARIMA/SARIMA and traditional time-series are capable of predicting Number of Active cases, daily recoveries, Number of deaths, they do not take into consideration the other time-varying factors like Medical Facilities (Volume of Testing, ICU beds, Hospital Admissions, Ventilators, Isolation Units, Quarantine Centres, etc). As these factors become important we build a predictive model that can predict the Number of Active Cases, Deaths, and Recoveries based on the change in Medical Facilities as well as other changes in infrastructure. Here in this blog, we try to model Multi-step Time Series Prediction using Deep learning Models on the basis of Medical Information available for different states of India. A typical multi-step predictive model looks as the below figure, where each of the predicted outcomes from the previous state is treated as next state input to derive the outcome for the second-state and so forth.
Multipole Graph Neural Operator for Parametric Partial Differential Equations
Li, Zongyi, Kovachki, Nikola, Azizzadenesheli, Kamyar, Liu, Burigede, Bhattacharya, Kaushik, Stuart, Andrew, Anandkumar, Anima
One of the main challenges in using deep learning-based methods for simulating physical systems and solving partial differential equations (PDEs) is formulating physics-based data in the desired structure for neural networks. Graph neural networks (GNNs) have gained popularity in this area since graphs offer a natural way of modeling particle interactions and provide a clear way of discretizing the continuum models. However, the graphs constructed for approximating such tasks usually ignore long-range interactions due to unfavorable scaling of the computational complexity with respect to the number of nodes. The errors due to these approximations scale with the discretization of the system, thereby not allowing for generalization under mesh-refinement. Inspired by the classical multipole methods, we propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Our multi-level formulation is equivalent to recursively adding inducing points to the kernel matrix, unifying GNNs with multi-resolution matrix factorization of the kernel. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
Smooth activations and reproducibility in deep networks
Shamir, Gil I., Lin, Dong, Coviello, Lorenzo
Deep networks are gradually penetrating almost every domain in our lives due to their amazing success. However, with substantive performance accuracy improvements comes the price of irreproducibility. Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on individual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems. The popular Rectified Linear Unit (ReLU) activation has been key to recent success of deep networks. We demonstrate, however, that ReLU is also a catalyzer to irreproducibility in deep networks. We show that not only can activations smoother than ReLU provide better accuracy, but they can also provide better accuracy-reproducibility tradeoffs. We propose a new family of activations; Smooth ReLU (SmeLU), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap. SmeLU is monotonic, mimics ReLU, while providing continuous gradients, yielding better reproducibility. We generalize SmeLU to give even more flexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of REctified Smooth Continuous Unit (RESCU) activations. Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular. Recent developments in deep learning leave no question about the advantages of deep networks over classical methods, which relied heavily on linear convex optimization solutions. With their astonishing unprecedented success, deep models are providing solutions to a continuously increasing number of domains in our lives. These solutions, however, while much more accurate than their convex counterparts, are usually irreproducible in the predictions they provide. While average accuracy of deep models on some validation dataset is usually much higher than that of linear convex models, predictions on individual examples of two models, that were trained to be identical, may diverge substantially, exhibiting Prediction Differences that may be as high as non-negligible fractions of the actual predictions (see, e.g., Chen et al. (2020); Dusenberry et al. (2020)). Deep networks express (only) what they learned.
ERIC: Extracting Relations Inferred from Convolutions
Townsend, Joe, Kasioumis, Theodoros, Inakoshi, Hiroya
Our main contribution is to show that the behaviour of kernels across multiple layers of a convolutional neural network can be approximated using a logic program. The extracted logic programs yield accuracies that correlate with those of the original model, though with some information loss in particular as approximations of multiple layers are chained together or as lower layers are quantised. We also show that an extracted program can be used as a framework for further understanding the behaviour of CNNs. Specifically, it can be used to identify key kernels worthy of deeper inspection and also identify relationships with other kernels in the form of the logical rules. Finally, we make a preliminary, qualitative assessment of rules we extract from the last convolutional layer and show that kernels identified are symbolic in that they react strongly to sets of similar images that effectively divide output classes into sub-classes with distinct characteristics.
Softer Pruning, Incremental Regularization
Cai, Linhang, An, Zhulin, Yang, Chuanguang, Xu, Yongjun
Network pruning is widely used to compress Deep Neural Networks (DNNs). The Soft Filter Pruning (SFP) method zeroizes the pruned filters during training while updating them in the next training epoch. Thus the trained information of the pruned filters is completely dropped. To utilize the trained pruned filters, we proposed a SofteR Filter Pruning (SRFP) method and its variant, Asymptotic SofteR Filter Pruning (ASRFP), simply decaying the pruned weights with a monotonic decreasing parameter. Our methods perform well across various networks, datasets and pruning rates, also transferable to weight pruning. On ILSVRC-2012, ASRFP prunes 40% of the parameters on ResNet-34 with 1.63% top-1 and 0.68% top-5 accuracy improvement. In theory, SRFP and ASRFP are an incremental regularization of the pruned filters. Besides, We note that SRFP and ASRFP pursue better results while slowing down the speed of convergence.
A Demonstration of Smart Doorbell Design Using Federated Deep Learning
Patel, Vatsal, Kanani, Sarth, Pathak, Tapan, Patel, Pankesh, Ali, Muhammad Intizar, Breslin, John
Smart doorbells have been playing an important role in protecting Furthermore, the processing and storage of multiple video streams our modern homes. Existing approaches of sending video streams make the subscription more costly. Secondly, this design requires to a centralized server (or Cloud) for video analytics have been a huge amount of reliable bandwidth, which may not always be facing many challenges such as latency, bandwidth cost and more had. Third, even if we assume that we could address latency and importantly users' privacy concerns. To address these challenges, bandwidth issue by empowering a sophisticated infrastructure, a this paper showcases the ability of an intelligent smart doorbell large class of video-based applications may not be suitable because based on Federated Deep Learning, which can deploy and manage of regulations and security concerns of sharing data as there is an video analytics applications such as a smart doorbell across Edge involvement of biometric data of residents.
Meta-learning the Learning Trends Shared Across Tasks
Rajasegaran, Jathushan, Khan, Salman, Hayat, Munawar, Khan, Fahad Shahbaz, Shah, Mubarak
Meta-learning stands for 'learning to learn' such that generalization to new tasks is achieved. Among these methods, Gradient-based meta-learning algorithms are a specific sub-class that excel at quick adaptation to new tasks with limited data. This demonstrates their ability to acquire transferable knowledge, a capability that is central to human learning. However, the existing meta-learning approaches only depend on the current task information during the adaptation, and do not share the meta-knowledge of how a similar task has been adapted before. To address this gap, we propose a 'Path-aware' model-agnostic meta-learning approach. Specifically, our approach not only learns a good initialization for adaptation, it also learns an optimal way to adapt these parameters to a set of task-specific parameters, with learnable update directions, learning rates and, most importantly, the way updates evolve over different time-steps. Compared to the existing meta-learning methods, our approach offers: (a) The ability to learn gradient-preconditioning at different time-steps of the inner-loop, thereby modeling the dynamic learning behavior shared across tasks, and (b) The capability of aggregating the learning context through the provision of direct gradient-skip connections from the old time-steps, thus avoiding overfitting and improving generalization. In essence, our approach not only learns a transferable initialization, but also models the optimal update directions, learning rates, and task-specific learning trends. Specifically, in terms of learning trends, our approach determines the way update directions shape up as the task-specific learning progresses and how the previous update history helps in the current update. Our approach is simple to implement and demonstrates faster convergence. We report significant performance improvements on a number of FSL datasets.
A Learning-based Discretionary Lane-Change Decision-Making Model with Driving Style Awareness
Zhang, Yifan, Xu, Qian, Wang, Jianping, Wu, Kui, Zheng, Zuduo, Lu, Kejie
Discretionary lane change (DLC) is a basic but complex maneuver in driving, which aims at reaching a faster speed or better driving conditions, e.g., further line of sight or better ride quality. Although many DLC decision-making models have been studied in traffic engineering and autonomous driving, the impact of human factors, which is an integral part of current and future traffic flow, is largely ignored in the existing literature. In autonomous driving, the ignorance of human factors of surrounding vehicles will lead to poor interaction between the ego vehicle and the surrounding vehicles, thus, a high risk of accidents. The human factors are also a crucial part to simulate a human-like traffic flow in the traffic engineering area. In this paper, we integrate the human factors that are represented by driving styles to design a new DLC decision-making model. Specifically, our proposed model takes not only the contextual traffic information but also the driving styles of surrounding vehicles into consideration and makes lane-change/keep decisions. Moreover, the model can imitate human drivers' decision-making maneuvers to the greatest extent by learning the driving style of the ego vehicle. Our evaluation results show that the proposed model almost follows the human decision-making maneuvers, which can achieve 98.66% prediction accuracy with respect to human drivers' decisions against the ground truth. Besides, the lane-change impact analysis results demonstrate that our model even performs better than human drivers in terms of improving the safety and speed of traffic.