The impressive generalization performance of modern neural networks is attributed in part to their ability to implicitly memorize complex training patterns. Inspired by this, we explore a novel mechanism to improve model generalization via explicit memorization. Specifically, we propose the residual-memorization (ResMem) algorithm, a new method that augments an existing prediction model (e.g., a neural network) by fitting the model's residuals with a k-nearest neighbor based regressor. The final prediction is then the sum of the original model and the fitted residual regressor. By construction, ResMem can explicitly memorize the training labels, even when the base model has low capacity. We start by formulating a stylized linear regression problem and rigorously show that ResMem results in a more favorable test risk over a base linear neural network. Then, we empirically show that ResMem consistently improves the test set generalization of the original prediction model across standard vision and natural language processing benchmarks.

From the perspective of the layer normalization (LN) positions, the architectures of Transformers can be categorized into two types: Post-LN and Pre-LN. Recent Transformers tend to be Pre-LN because, in Post-LN with deep Transformers (e.g., those with ten or more layers), the training is often unstable, resulting in useless models. However, Post-LN has consistently achieved better performance than Pre-LN in relatively shallow Transformers (e.g., those with six or fewer layers). This study first investigates the reason for these discrepant observations empirically and theoretically and made the following discoveries: 1, the LN in Post-LN is the main source of the vanishing gradient problem that leads to unstable training, whereas Pre-LN prevents it, and 2, Post-LN tends to preserve larger gradient norms in higher layers during the back-propagation, which may lead to effective training. Exploiting the new findings, we propose a method that can provide both high stability and effective training by a simple modification of Post-LN. We conduct experiments on a wide range of text generation tasks. The experimental results demonstrate that our method outperforms Pre-LN, and enables stable training regardless of the shallow or deep layer settings. Our code is publicly available at https://github.com/takase/b2t_connection.

As shown by the example in this post, we collected enough images, uploaded them, and annotated them with regions and labels. Then we created datasets and trained a Deepnet to perform Object Detection. We also evaluated the model and used it to predict new images that detected objects accurately. All of these tasks were done on the Dashboard with a few clicks. This is as accessible as it gets in Machine Learning. And just as our motto suggests, BigML has made Object Detection beautifully simple for everyone. Be sure to visit the release page of BigML Object Detection, where you can find more information and documentation.

Continuing with our Object Detection release blog posts series, today, we'll showcase how to automate the training of the object detection models (and their predictions) that anyone will be able to create in BigML in short order. As discussed in previous posts, BigML already offers classification, regression, and unsupervised learning models (e.g., clustering, anomaly detection). They all accept images as just another input data type usable for model training. In fact, when images are uploaded a new Source is created for each and their corresponding IDs are added to a new Composite Source object with a new image field type. In summary, images can be combined with any other data type and can be assigned one or more labels by using the new label fields.

We've seen in the past several blog posts on how you can learn simple image classifiers with BigML, via the interface, the API, and with the BigML Python Bindings, and we've also seen that you can train unsupervised models on the same images. But let's dig a little deeper and explore different approaches to an image processing problem. One of the things we try hard to do at BigML is, to paraphrase Alan Kay, make simple things simple and complex things possible. It's a relatively simple thing to train an image classifier: You have a bunch of images, those images have classes, and you want to train a model that will classify any new image into one of those classes. One great application of this is the area of security and monitoring: You have a system constantly looking at something and you want to be alerted when that thing changes.

BigML's upcoming release on Wednesday, December 15, 2021, will be presenting a new set of Image Processing resources to the BigML platform. In this post, we show you how to build a simple image classifier on the BigML Dashboard. Image classification is a supervised learning technique for images. Image classification models are trained to identify various classes of images and have a tremendous amount of applications as touched on in our prior posts. As such, BigML introduces image data support with the latest Image Processing release.

This directory contains files to build Docker images - encapsulated computational containers which enhance reproducibility for scientific research. They are similar in design philosophy to the excellent Jupyter Docker Stacks but with a focus on making it easy to get up and running with GPU-accelerated deep learning. The base image provides a Jupyter Lab (notebook) environment in a Docker container which has direct access to the host system's GPU(s). Additionally there is a custom directory with instructions and examples for building your own image. These are considered stable but may be moved to their own repositories in future. The instructions below refer to the combined (default) image mmrl/dl (based on the Keras Dockerfile) which contains ALL TEH THINGZ!!!

Learn how BigML Deepnets help you automatically find the best neural network to solve your supervised learning problems.