Recent strategies achieved ensembling "for free" by fitting concurrently diverse subnetworks inside a single base network. The main idea during training is that each subnetwork learns to classify only one of the multiple inputs simultaneously provided. However, the question of how to best mix these multiple inputs has not been studied so far. In this paper, we introduce MixMo, a new generalized framework for learning multi-input multi-output deep subnetworks. Our key motivation is to replace the suboptimal summing operation hidden in previous approaches by a more appropriate mixing mechanism. For that purpose, we draw inspiration from successful mixed sample data augmentations. We show that binary mixing in features - particularly with rectangular patches from CutMix - enhances results by making subnetworks stronger and more diverse. We improve state of the art for image classification on CIFAR-100 and Tiny ImageNet datasets. Our easy to implement models notably outperform data augmented deep ensembles, without the inference and memory overheads. As we operate in features and simply better leverage the expressiveness of large networks, we open a new line of research complementary to previous works.

The convolutional neural network (CNN) is vulnerable to degraded images with even very small variations (e.g. corrupted and adversarial samples). One of the possible reasons is that CNN pays more attention to the most discriminative regions, but ignores the auxiliary features when learning, leading to the lack of feature diversity for final judgment. In our method, we propose to dynamically suppress significant activation values of CNN by group-wise inhibition, but not fixedly or randomly handle them when training. The feature maps with different activation distribution are then processed separately to take the feature independence into account. CNN is finally guided to learn richer discriminative features hierarchically for robust classification according to the proposed regularization. Our method is comprehensively evaluated under multiple settings, including classification against corruptions, adversarial attacks and low data regime. Extensive experimental results show that the proposed method can achieve significant improvements in terms of both robustness and generalization performances, when compared with the state-of-the-art methods.

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved classification accuracy. CPE can be effortlessly implemented with a simple Position Encoding Generator (PEG), and it can be seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings. Benefit from the conditional positional encoding scheme, we obtain state-of-the-art results on the ImageNet classification task compared with vision Transformers to date. Our code will be made available at https://github.com/Meituan-AutoML/CPVT .

Large Neural Networks are difficult to use in production environments as they are memory intensive and are slow during inference. Most successful Deep Learning Models such as Transformers are being followed by their Lite Versions which dramatically speed up inference trading off accuracy. In this article, let's explore Least Squares Quantization, an algorithm to speed up large neural networks by quantizing them while reducing the accuracy gap from the non-quantized model. Hadi Pouransari, Zhucheng Tu, Oncel Tuzel, researchers at Apple, introduced this approach in a paper- Least Squares Binary Quantization of Neural Networks, on 23rd March 2020. We all agree that smaller models are better for practical purposes in memory usage and inference time.

New funding: Seattle-based startup OctoML raised a $28 million Series B round. The University of Washington spinout aims to help companies deploy machine learning models on various hardware configurations. The technology: OctoML is led by the creators of Apache TVM, an open source "deep learning compiler stack" that started as a research project at the UW's computer science school. The idea is to reduce the amount of cost and time it takes companies to develop and deploy deep learning software for specific hardware such as phones, cars, health devices, etc. -- "using ML to optimize ML," as OctoML CEO Luis Ceze explains. Traction: OctoML is working with Qualcomm, Microsoft, AMD, Bosch, and many others.

This blog post describes the TRIC model -- an architecture for Relative Image Captioning task that was created as a part of my Master Thesis. All of them are described in my thesis in a pretty concise way so I highly recommend it -- you can find a link right below. But if you want to check them from another source it is also covered. To each of the topics listed above, I have attached a link to my personal favorite resource concerning this particular subject. Earlier this month I defended my master's thesis in Computer Science at the Warsaw University of Technology.

In light of the coronavirus (COVID-19) and to protect our candidates' and interviewers' health and wellbeing, we will be conducting all interviews virtually. We are searching for a Deep Learning engineer with experience working on production level image recognition models, who is passionate about developing and applying Deep Learning algorithms to solve a wide range of real world problems, from detecting anomalies in shiny glass bottles to identifying nodules in thyroid ultrasound images and predicting possible landslides from topographical maps - just to name a few. At Incubit, we undertake the challenge of solving problems that haven't been solved alongside the leading companies of Japan. Japanese companies have a long tradition of working with external technical experts to solve their core business issues by applying the latest technology. They count on us as an expert in problem solving, especially in the area of image recognition using Deep Learning technology.

Adversarial examples are a hot research topic in deep learning nowadays. Subtle, often invisible changes in the data can push our networks to make terrible mistakes. We, as human beings, seem to be way more resilient to these perturbations in our sensory inputs (though not totally immune). There is a certain pattern in our deep learning systems. They achieve remarkable things, but they are also at times delicate and brittle.

The Rise of Artificial Intelligence has resulted in the creation of rapid world-changing technologies that can impact the future of all entities, preferably for the better of humanity. The modern generation of the world is technologically advanced, and the pace of advancements and improvements is continuously increasing with no signs of slowing down. The current era we all live in will prove to be the most influential period of all time. The Curve of improvements in humanity has always tended to be an exponential curve. With the discovery of the fire way dating back to thousands of years ago, to the discovery of the wheel, to the inventions of high-quality developments in the 17th and 18th Century, and finally, to the modern era of technological advancements where the majority of the inventions have been made.

We recently launched our book on descriptive statistics with Python. If you haven't checked it out yet, this article or this PDF provides a sample of the first 36 pages of the book. Please don't forget that you can access this work, many more books, and other goodies by becoming a member. Next, if you are currently interested in pursuing a Ph.D. in machine learning or a related field, we recommend checking out this piece by machine learning at Carnegie Mellon, which highlights which questions you should consider asking a prospective Ph.D. advisor to make out the best possible match for your research interests. What properties of datasets drive modeling innovation?