The RNN transducer is a promising end-to-end model candidate. We compare the original training criterion with the full marginalization over all alignments, to the commonly used maximum approximation, which simplifies, improves and speeds up our training. We also generalize from the original neural network model and study more powerful models, made possible due to the maximum approximation. We further generalize the output label topology to cover RNN-T, RNA and CTC. We perform several studies among all these aspects, including a study on the effect of external alignments. We find that the transducer model generalizes much better on longer sequences than the attention model. Our final transducer model outperforms our attention model on Switchboard 300h by over 6% relative WER.

Spatio-temporal sequence forecasting is one of the fundamental tasks in spatio-temporal data mining. It facilitates many real world applications such as precipitation nowcasting, citywide crowd flow prediction and air pollution forecasting. Recently, a few Seq2Seq based approaches have been proposed, but one of the drawbacks of Seq2Seq models is that, small errors can accumulate quickly along the generated sequence at the inference stage due to the different distributions of training and inference phase. That is because Seq2Seq models minimise single step errors only during training, however the entire sequence has to be generated during the inference phase which generates a discrepancy between training and inference. In this work, we propose a novel curriculum learning based strategy named Temporal Progressive Growing Sampling to effectively bridge the gap between training and inference for spatio-temporal sequence forecasting, by transforming the training process from a fully-supervised manner which utilises all available previous ground-truth values to a less-supervised manner which replaces some of the ground-truth context with generated predictions. To do that we sample the target sequence from midway outputs from intermediate models trained with bigger timescales through a carefully designed decaying strategy. Experimental results demonstrate that our proposed method better models long term dependencies and outperforms baseline approaches on two competitive datasets.

Intrusion detection is one of the primary mechanisms to provide computer networks with security. With an increase in attacks and growing dependence on various fields such as medicine, commercial, and engineering to give services over a network, securing networks have become a significant issue. The purpose of Intrusion Detection Systems (IDS) is to make models which can recognize regular communications from abnormal ones and take necessary actions. Among different methods in this field, Artificial Neural Networks (ANNs) have been widely used. However, ANN-based IDS, has two main disadvantages: 1- Low detection precision. 2- Weak detection stability. To overcome these issues, this paper proposes a new approach based on Deep Neural Network (DNN. The general mechanism of our model is as follows: first, some of the data in dataset is properly ranked, afterwards, dataset is normalized with Min-Max normalizer to fit in the limited domain. Then dimensionality reduction is applied to decrease the amount of both useless dimensions and computational cost. After the preprocessing part, Mean-Shift clustering algorithm is the used to create different subsets and reduce the complexity of dataset. Based on each subset, two models are trained by Support Vector Machine (SVM) and deep learning method. Between two models for each subset, the model with a higher accuracy is chosen. This idea is inspired from philosophy of divide and conquer. Hence, the DNN can learn each subset quickly and robustly. Finally, to reduce the error from the previous step, an ANN model is trained to gain and use the results in order to be able to predict the attacks. We can reach to 95.4 percent of accuracy. Possessing a simple structure and less number of tunable parameters, the proposed model still has a grand generalization with a high level of accuracy in compared to other methods such as SVM, Bayes network, and STL.

This paper surveys deep learning algorithms, IoT cyber security and risk models, and established mathematical formulas to identify the best approach for developing a dynamic and self-adapting system for predictive cyber risk analytics supported with Artificial Intelligence and Machine Learning and real-time intelligence in edge computing. The paper presents a new mathematical approach for integrating concepts for cognition engine design, edge computing and Artificial Intelligence and Machine Learning to automate anomaly detection. This engine instigates a step change by applying Artificial Intelligence and Machine Learning embedded at the edge of IoT networks, to deliver safe and functional real-time intelligence for predictive cyber risk analytics. This will enhance capacities for risk analytics and assists in the creation of a comprehensive and systematic understanding of the opportunities and threats that arise when edge computing nodes are deployed, and when Artificial Intelligence and Machine Learning technologies are migrated to the periphery of the internet and into local IoT networks.

A vast array of Deep Learning (DL) approaches have been proven successful in many different learning tasks in recent years. One of the key strength of DL models is their ability to automatically learn a representation of the features composing a data set. Deep Neural Networks (DNNs) represent the foremost and widely spread class of DL models. Broadly speaking, DNNs are sub-symbolic ML approaches that are very good at extracting the useful information contained in large data sets. One of the advantages of DL techniques is that, in general, they do not rely on stringent assumptions on the distribution of the underlying data and on the function to be learned or approximated. This allows them to be applied in many different areas with very good results, without significant changes to the DNNs' structure and training algorithm. However, there are contexts where purely data-driven models are not an ideal fit, for example when scarce data of good quality and very difficult learning tasks. In such situations, a great boost in the performance of neural networks (and ML models in general) can be obtained through the exploitation of domain knowledge, e.g.

Over the last decade, there has been significant progress in the field of machine learning for de novo drug design, particularly in deep generative models. However, current generative approaches exhibit a significant challenge as they do not ensure that the proposed molecular structures can be feasibly synthesized nor do they provide the synthesis routes of the proposed small molecules, thereby seriously limiting their practical applicability. In this work, we propose a novel forward synthesis framework powered by reinforcement learning (RL) for de novo drug design, Policy Gradient for Forward Synthesis (PGFS), that addresses this challenge by embedding the concept of synthetic accessibility directly into the de novo drug design system. In this setup, the agent learns to navigate through the immense synthetically accessible chemical space by subjecting commercially available small molecule building blocks to valid chemical reactions at every time step of the iterative virtual multi-step synthesis process. The proposed environment for drug discovery provides a highly challenging test-bed for RL algorithms owing to the large state space and high-dimensional continuous action space with hierarchical actions. PGFS achieves state-of-the-art performance in generating structures with high QED and penalized clogP. Moreover, we validate PGFS in an in-silico proof-of-concept associated with three HIV targets. Finally, we describe how the end-to-end training conceptualized in this study represents an important paradigm in radically expanding the synthesizable chemical space and automating the drug discovery process.

Identification of input data points relevant for the classifier (i.e. serve as the support vector) has recently spurred the interest of researchers for both interpretability as well as dataset debugging. This paper presents an in-depth analysis of the methods which attempt to identify the influence of these data points on the resulting classifier. To quantify the quality of the influence, we curated a set of experiments where we debugged and pruned the dataset based on the influence information obtained from different methods. To do so, we provided the classifier with mislabeled examples that hampered the overall performance. Since the classifier is a combination of both the data and the model, therefore, it is essential to also analyze these influences for the interpretability of deep learning models. Analysis of the results shows that some interpretability methods can detect mislabels better than using a random approach, however, contrary to the claim of these methods, the sample selection based on the training loss showed a superior performance.

We propose a theoretical framework for an adaptive learning rate policy for the Mean Absolute Error loss function and Quantile loss function and evaluate its effectiveness for regression tasks. The framework is based on the theory of Lipschitz continuity, specifically utilizing the relationship between learning rate and Lipschitz constant of the loss function. Based on experimentation, we have found that the adaptive learning rate policy enables up to 20x faster convergence compared to a constant learning rate policy.

The most appreciative advancements in statistical AI, the ones with the most meaning and potential to improve data's worth to the enterprise, are deep learning deployments of computer vision and natural language technologies. The distinctions between these applications involve much more than image recognition versus that of speech or language. Horizontal computer vision use cases pertain to some aspects of inter-machine intelligence, e.g., scanning videos or production settings for anomalies and generating alerts to initiate automated procedures to address them. Conversely, natural language technologies provide the most effective cognitive computing application for furthering human intelligence, decision making, and the action required to extract business value from such perceptivity. While the utility derived from image recognition largely varies according to the vertical, the capability for machines to understand natural language--for humans to interact with databases in layperson's terms across sources--strikes at the core of converting the unstructured data of language into informed action.

That being said, which deep learning framework from the above list would best suit your requirements? The answer to that lies on a number of factors, however, if you are looking to just get started, then a Python based deep learning framework like TensorFlow or Chainer should be your choice. If you happen to be seasoned, you need to consider speed, resource requirement, and usage along with the coherence of the trained model before picking out the best deep learning framework.