Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their "clean" correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code will be available at https://github.com/SHI-Labs/Pyramid-Attention-Networks

We examine the applicability of modern neural network architectures to the midterm prediction of earthquakes. Our data-based classification model aims to predict if an earthquake with the magnitude above a threshold takes place at a given area of size $10 \times 10$ kilometers in $10$-$60$ days from a given moment. Our deep neural network model has a recurrent part (LSTM) that accounts for time dependencies between earthquakes and a convolutional part that accounts for spatial dependencies. Obtained results show that neural networks-based models beat baseline feature-based models that also account for spatio-temporal dependencies between different earthquakes. For historical data on Japan earthquakes our model predicts occurrence of an earthquake in $10$ to $60$ days from a given moment with magnitude $M_c > 5$ with quality metrics ROC AUC $0.975$ and PR AUC $0.0890$, making $1.18 \cdot 10^3$ correct predictions, while missing $2.09 \cdot 10^3$ earthquakes and making $192 \cdot 10^3$ false alarms. The baseline approach has similar ROC AUC $0.992$, number of correct predictions $1.19 \cdot 10^3$, and missing $2.07 \cdot 10^3$ earthquakes, but significantly worse PR AUC $0.00911$, and number of false alarms $1004 \cdot 10^3$.

We live in fascinating times, where Deep Learning [DL] is continuously applied in new areas of our life and very often, revolutionizes otherwise stagnated industries. At the same time, open-source frameworks such as Keras and PyTorch level the playing field and give everybody access to state-of-the-art tools and algorithms. Strong community and simple API of these libraries make it possible to have cutting edge models at your fingertips, even without in-depth knowledge of math that makes it all possible. However, the understanding of what is happening inside the Neural Network [NN] helps a lot with tasks like architecture selection, hyperparameters tuning, or performance optimization. Since I believe that nothing teaches you more than getting your hands dirty, I'll show you how to create a Convolutional Neural Network [CNN] capable of classifying MNIST images, with 90% accuracy, using only NumPy.

Then this course is for you! This course is designed in a very simple and easily understandable content. You might have seen lots of buzz on deep learning and you want to figure out where to start and explore. This course is designed exactly for people like you! If basics are strong, we can do bigger things with ease.

A very simple graph that adds two numbers together. In the figure above, two numbers are supposed to be added. Those numbers are stored in two variables, a and b. The two values are flowing through the graph and arrive at the square node, where they are being added. The result of the addition is stored into another variable, c.

Deep learning is still bearing fruits. However, the standard types of networks are exhausting their possibilities, and researchers seek out such extensions to the basic neural network models, which will weaken their inherent limitations. Some extensions such as self-attention layers have enjoyed great practical success. Remarkably, many shortcomings of neural networks mirror the advantages of symbolic systems (and vice versa). Indeed, one can note that both self-attention layers and capsule networks are attempts to work around the notorious variable binding problem described in the Fodor and Pylyshyn's paper, which is easily solved in symbolic systems but is very inconvenient for neural networks.

Wind farms have traditionally made less money for the electricity they produce because they have been unable to predict how windy it will be tomorrow. "The way a lot of power markets work is you have to schedule your assets a day ahead," said Michael Terrell, the head of energy market strategy at Google. "And you tend to get compensated higher when you do that than if you sell into the market real-time. "Well, how do variable assets like wind schedule a day ahead when you don't know the wind is going to blow?" Terrell asked, "and how can you actually reserve your place in line?" Here's how: Google and the Google-owned Artificial Intelligence firm DeepMind combined weather data with power data from 700 megawatts of wind energy that Google sources in the Central United States. Using machine learning, they have been able to better predict wind production, better predict electricity supply and demand, and as a result, reduce operating costs. "What we've been doing is working in partnership with the DeepMind team to use machine learning to take the weather data that's available publicly, actually forecast what we think the wind production will be the next day, and bid that wind into the day-ahead markets," Terrell said in a recent seminar hosted by the Stanford Precourt Institute of Energy. Stanford University posted video of the seminar last week. The result has been a 20 percent increase in revenue for wind farms, Terrell said. The Department of Energy listed improved wind forecasting as a first priority in its 2015 Wind Vision report, largely to improve reliability: "Improve Wind Resource Characterization," the report said at the top of its list of goals. "Collect data and develop models to improve wind forecasting at multiple temporal scales--e.g., minutes, hours, days, months, years." Google's goal has been more sweeping: to scrub carbon entirely from its energy portfolio, which consumes as much power as two San Franciscos. Google achieved an initial milestone by matching its annual energy use with its annual renewable-energy procurement, Terrell said. But the company has not been carbon-free in every location at every hour, which is now its new goal--what Terrell calls its "24x7 carbon-free" goal. "We're really starting to turn our efforts in this direction, and we're finding that it's not something that's easy to do.

Generative Adversarial Networks (GANs) are a trend nowadays in various unsupervised learning applications. They are applied in animation and gaming with a full swing due to their capability to produce new images when trained on a set of similar but different images. This model is basically a deep generative model composed of two networks – a generator and a discriminator. The Deep Convolutional Neural Network is one of the variants of GAN where convolutional layers are added to the generator and discriminator networks. In this article, we will train the Deep Convolutional Generative Adversarial Network on Fashion MNIST training images in order to generate a new set of fashion apparel images.

We present a novel approach for adaptive, differentiable parameterization of large-scale random fields. If the approach is coupled with any gradient-based optimization algorithm, it can be applied to a variety of optimization problems, including history matching. The developed technique is based on principal component analysis (PCA) but modifies a purely data-driven basis of principal components considering objective function behavior. To define an efficient encoding, Gradient-Sensitive PCA uses an objective function gradient with respect to model parameters. We propose computationally efficient implementations of the technique, and two of them are based on stationary perturbation theory (SPT). Optimality, correctness, and low computational costs of the new encoding approach are tested, verified, and discussed. Three algorithms for optimal parameter decomposition are presented and applied to an objective of 2D synthetic history matching. The results demonstrate improvements in encoding quality regarding objective function minimization and distributional patterns of the desired field. Possible applications and extensions are proposed.

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study how the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.