Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, we propose an improved activation function, which we name the natural-logarithm-rectified linear unit (NLReLU). This activation function uses the parametric natural logarithmic transform to improve ReLU and is simply defined as. NLReLU not only retains the sparse activation characteristic of ReLU, but it also alleviates the "dying ReLU" and vanishing gradient problems to some extent. It also reduces the bias shift effect and heteroscedasticity of neuron data distributions among network layers in order to accelerate the learning process. The proposed method was verified across ten convolutional neural networks with different depths for two essential datasets. Experiments illustrate that convolutional neural networks with NLReLU exhibit higher accuracy than those with ReLU, and that NLReLU is comparable to other well-known activation functions. NLReLU provides 0.16% and 2.04% higher classification accuracy on average compared to ReLU when used in shallow convolutional neural networks with the MNIST and CIFAR-10 datasets, respectively. The average accuracy of deep convolutional neural networks with NLReLU is 1.35% higher on average with the CIFAR-10 dataset.

We propose a deep learning approach for discovering kernels tailored to identifying clusters over sample data. Our neural network produces sample embeddings that are motivated by--and are at least as expressive as--spectral clustering. Our training objective, based on the Hilbert Schmidt Information Criterion, can be optimized via gradient adaptations on the Stiefel manifold, leading to significant acceleration over spectral methods relying on eigendecompositions. Finally, our trained embedding can be directly applied to out-of-sample data. We show experimentally that our approach outperforms several state-of-the-art deep clustering methods, as well as traditional approaches such as $k$-means and spectral clustering over a broad array of real-life and synthetic datasets.

Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling. Historically, probabilistic modeling has been constrained to (i) very restricted model classes where exact or approximate probabilistic inference were feasible, and (ii) small or medium-sized data sets which fit within the main memory of the computer. However, developments in variational inference, a general form of approximate probabilistic inference originated in statistical physics, are allowing probabilistic modeling to overcome these restrictions: (i) Approximate probabilistic inference is now possible over a broad class of probabilistic models containing a large number of parameters, and (ii) scalable inference methods based on stochastic gradient descent and distributed computation engines allow to apply probabilistic modeling over massive data sets. One important practical consequence of these advances is the possibility to include deep neural networks within a probabilistic model to capture complex non-linear stochastic relationships between random variables. These advances in conjunction with the release of novel probabilistic modeling toolboxes have greatly expanded the scope of application of probabilistic models, and allow these models to take advantage of the recent strides made by the deep learning community. In this paper we review the main concepts, methods and tools needed to use deep neural networks within a probabilistic modeling framework.

--Additive Manufacturing (AM) is a manufacturing paradigm that builds three-dimensional objects from a computer-aided design model by successively adding material layer by layer . AM has become very popular in the past decade due to its utility for fast prototyping such as 3D printing as well as manufacturing functional parts with complex geometries using processes such as laser metal deposition that would be difficult to create using traditional machining. As the process for creating an intricate part for an expensive metal such as Titanium is prohibitive with respect to cost, computational models are used to simulate the behavior of AM processes before the experimental run. However, as the simulations are computationally costly and time-consuming for predicting multiscale multi-physics phenomena in AM, physics-informed data-driven machine-learning systems for predicting the behavior of AM processes are immensely beneficial. Such models accelerate not only multiscale simulation tools but also empower real-time control systems using in-situ data. In this paper, we design and develop essential components of a scientific framework for developing a data-driven model-based real-time control system. Finite element methods are employed for solving time-dependent heat equations and developing the database. The proposed framework uses extremely randomized trees - an ensemble of bagged decision trees as the regression algorithm iteratively using temperatures of prior voxels and laser information as inputs to predict temperatures of subsequent voxels. The models achieve mean absolute percentage errors below 1% for predicting temperature profiles for AM processes. Additive Manufacturing (AM) is a modern manufacturing approach in which digital 3D design data is used to build parts by sequentially depositing layers of materials [1]. AM techniques are becoming very popular compared to traditional approaches because of their success in building complicated designs, fast prototyping, and low-volume or one-of-a-kind productions across many industries. Direct Metal Deposition (DMD) [2] is an AM technology where various materials such as steel or Titanium are used to develop the finished product.

In September 2018, Hurricane Florence devastated the eastern U.S., forcing more than 1.7 million people to evacuate their homes. Immediately after the storm, federal relief agencies started collecting imagery to identify flooded areas and damaged infrastructure to rescue, relieve, and rebuild affected communities, but current analysis methods can be slow and inefficient. The Department of Defense's Joint Artificial Intelligence Center (JAIC) and the Johns Hopkins Applied Physics Laboratory (APL) teamed to quickly build AI-enabled capabilities that accelerated that analysis; the results immediately increased the effectiveness of relief efforts. Learn how the JAIC/APL team used deep learning algorithms in this work, and preview upcoming tech and new AI-based disaster response tools.

Basically, what we are going to try to do, is compress our image into a latent vector (called the'bottleneck'), and from there on try to reconstruct the original image. The better we do this, the better of a smaller-size representation the latent vector is of our input image. We can use various techniques for the encoder and decoder network. First up, let's start of pretty basic with a simple fully connected auto-encoder, and work our way up from there. I've found this Keras blog post very helpful.

Two years ago Quoc Le of Google Brain conceived of "AutoML," where he used AI to design the neural networks of other AIs. The results were impressive, with AIs beating the hand-built networks of top research teams. DeepMind entered the fray this past week, applying sophisticated techniques that mimic evolution... [Read More]

DeepMind has joined the ranks of those using AI to save fragile wildlife populations, and it's doing that on a grand scale. The company is partnering with conservationists and ecologists on a project that uses machine learning to speedily detect and count animals in "millions" of photos taken over the past nine years in Tanzania's Serengeti National Park. Where it normally takes up to a year for volunteers to return labeled photos, DeepMind has developed a model that can label most animals at least as well as humans while shortening the process by up to nine months That's no small challenge when animals seldom cooperate with motion-sensitive cameras -- the AI can recognize out-of-focus cheetahs or fast-moving ostriches. The technology should also be viable in the wild. DeepMind is developing a pre-trained version of its AI model that would need only "modest" hardware and little internet connectivity -- important when a powerful computer and fast internet access could be disruptive to wildlife and expensive to deploy.

We live in an era of too much information -- an endless stream of status updates, memes, reposts, infographics, quotes and hashtags roll daily through our social media feeds, meant to express viewpoints, drum up solidarity, provide information, change minds or cause controversy. Problem is, the average online browser/social media user doesn't have the time or wherewithal to investigate the legitimacy or provenance of everything that shows up on their feeds. And it is this vulnerability that less scrupulous content generators exploit to spread misinformation, with results that can range from a little egg-on-face embarrassment to downright life-changing or potentially deadly consequences. For UC Santa Barbara computer scientist William Wang, this chaotic morass is fertile grounds for exploration. Wang believes that deep learning techniques, when deployed on the text and hyperlink network of online posts and news articles, can help us with some of the critical thinking heavy lifting.

In traditional ML systems, a human (usually a subject matter expert) selects features that are determined to be useful in classification and given as inputs to an ML algorithm. Then, the algorithm learns how to use these features to maximize classification accuracy. In a DL system, features are learned through a mathematical process like backpropagation at every layer of the network. On the first layer of the network, the DL system would learn rudimentary features that can be calculated from the raw input signals. On the second layer, the network learns more complex features using combinations of the features learned on the first layer, and so on.