We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.

How can prior knowledge on the transformation invariances of a domain be incorporated into the architecture of a neural network? We propose Equivariant Transformers (ETs), a family of differentiable image-to-image mappings that improve the robustness of models towards pre-defined continuous transformation groups. Through the use of specially-derived canonical coordinate systems, ETs incorporate functions that are equivariant by construction with respect to these transformations. We show empirically that ETs can be flexibly composed to improve model robustness towards more complicated transformation groups in several parameters. On a real-world image classification task, ETs improve the sample efficiency of ResNet classifiers, achieving relative improvements in error rate of up to 15% in the limited data regime while increasing model parameter count by less than 1%.

Deep learning has been widely adapted to many different problems, such as image classification [1], speech recognition [2] and natural language processing [3], and has demonstrated state-of-the-art results for these problems. Despite the promises, deep neural networks (DNNs) remain challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Due to the limited computational resources available in such on-device edge scenarios, many recent studies [4, 5, 6, 7] have put greater efforts into designing small, low-footprint deep neural network architectures that are more appropriate for embedded devices. A particularly interesting approach for enabling low-footprint deep neural network architectures is the concept of knowledge distillation [8], where the performance of a smaller network is significantly improved by leveraging a teacher-student strategy where the smaller network is trained to mimic the behaviour of a larger teacher network. With much of the research around distillation focused on distilling knowledge from larger networks to smaller networks, there is little research focused on leveraging the concept of distillation for distilling knowledge encapsulated in the training data itself into a reduced form. By producing data with reduced data dimension, one can achieve input-efficient deep neural networks with significantly reduced computational costs. In this study, we explore a concept we will call progressive label distillation, where a series of teacher-student network pairs are leveraged to progressively generate distilled training data.

Most technology news as of late somehow relates back to artificial intelligence. The seemingly complex and high-brow technology is integrated into mundane items, such as Amazon's Alexa or Google Home. With talk of artificial intelligence comes machine learning and deep machine learning. The three phrases can often be conflated, but do refer to three specific technologies.

At last year's re:Invent 2018 conference in Las Vegas, Amazon took the wraps off SageMaker Neo, a feature that enabled developers to train machine learning models and deploy them virtually anywhere their hearts desired, either in the cloud or on-premises. It worked as advertised, but the benefits were necessarily limited to AWS customers -- Neo was strictly a closed-source, proprietary affair. Amazon yesterday announced that it's publishing Neo's underlying code under the Apache Software License as Neo-AI and making it freely available in a repository on GitHub. This step, it says, will help usher in "new and independent innovations" on a "wide variety" of hardware platforms, from third-party processor vendors and device manufacturers to deep learning practitioners. "Ordinarily, optimizing a machine learning model for multiple hardware platforms is difficult, because developers need to tune models manually for each platform's hardware and software configuration," Sukwon Kim, senior product manager for AWS Deep Learning, and Vin Sharma, engineering leader, wrote in a blog post.

We are looking for outstanding Data Engineers to join our team. This is a great opportunity for a Data Science Consultant to join a consulting firm that offers a variety of projects and a structured learning and development path. You will work alongside a talented team of consultants who all share your passion in building great solutions and learning new skills. Skills in Data Engineering and Machine Learning: as a data science consultant you will have proficiency in one or more of Python, R, Scala, Matlab/Octave, Java, C/C, Go, Javascript, Clojure etc. You will also have either hands on experience or good knowledge of the one of the following concepts such as supervised learning, unsupervised learning, reinforcement learning, deep learning, feature engineering, natural language processing, computer vision, signal processing etc.

While Google previously ran its analysis with data in a linear way, handling each element discretely, it no longer does so. It now uses a type of AI called Deep Learning. The hierarchical function of Deep Learning systems enables Google to process data with a non-linear approach. SO... Rather than considering each factor separately like your typical Human SEO does, and awarding small incremental gains for each, Google now batches multiple factors together. Then, if their output reaches a certain value, then their cumulative value triggers or contributes to the next stage if analysis.

Machine-learning algorithms use statistics to find patterns in massive* amounts of data. And data, here, encompasses a lot of things--numbers, words, images, clicks, what have you. If it can be digitally stored, it can be fed into a machine-learning algorithm. Machine learning is the process that powers many of the services we use today--recommendation systems like those on Netflix, YouTube, and Spotify; search engines like Google and Baidu; social-media feeds like Facebook and Twitter; voice assistants like Siri and Alexa.

Artificial intelligence can play chess, drive a car and diagnose medical issues. Examples include Google DeepMind's AlphaGo, Tesla's self-driving vehicles, and IBM's Watson. This type of artificial intelligence is referred to as Artificial Narrow Intelligence (ANI) – non-human systems that can perform a specific task. We encounter this type on a daily basis, and its use is growing rapidly. But while many impressive capabilities have been demonstrated, we're also beginning to see problems.