Bio: Alexander Gude is currently a data scientist at Lab41 working on investigating recommender system algorithms. He holds a BA in physics from University of California, Berkeley, and a PhD in Elementary Particle Physics from University of Minnesota-Twin Cities. About: Lab41 is a "challenge lab" where the U.S. Intelligence Community comes together with their counterparts in academia, industry, and In-Q-Tel to tackle big data. It allows participants from diverse backgrounds to gain access to ideas, talent, and technology to explore what works and what doesn't in data analytics.

Today's paper is by Gao Huang, Yu Sun, et al. It introduces a new way to perturb networks during training in order to improve their performance. Before I continue, let me first state that this paper is a real pleasure to read; it is concise and extremely well written. It gives an excellent overview of the motivating problems, previous solutions, and Huang and Sun's new approach. I highly recommended giving it a read!

The next paper from our reading group is by Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally and Kurt Keutzer. This paper introduces a small CNN architecture called "SqueezeNet" that achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters. As you may have noticed with one of our recent posts we're really interested in learning more about the compression of neural network architectures and this paper really stood out. It's no secret that much of deep learning is tied up in the hell that is parameter tuning. This paper makes a case for increased study into the area of convolutional neural network design in order to drastically reduce the number of parameters you have to deal with.

Continuing the tour of older papers that started with our ResNet blog post, we now take on Skip-Thought Vectors by Kiros et al. Their goal was to come up with a useful embedding for sentences that was not tuned for a single task and did not require labeled data to train. They took inspiration from Word2Vec skip-gram (you can find my explanation of that algorithm here) and attempt to extend it to sentences. Skip-thought vectors are created using an encoder-decoder model. The encoder takes in the training sentence and outputs a vector.

Bio: Alexander Gude is currently a data scientist at Lab41 working on investigating recommender system algorithms. He holds a BA in physics from University of California, Berkeley, and a PhD in Elementary Particle Physics from University of Minnesota-Twin Cities. About: Lab41 is a "challenge lab" where the U.S. Intelligence Community comes together with their counterparts in academia, industry, and In-Q-Tel to tackle big data. It allows participants from diverse backgrounds to gain access to ideas, talent, and technology to explore what works and what doesn't in data analytics.

Their model requires groups of sentences in order to train, and so trained on the BookCorpus Dataset. The dataset consists of novels by unpublished authors and is (unsurprisingly) dominated by romance and fantasy novels. This "bias" in the dataset will become apparent later when discussing some of the sentences used to test the skip-thought model; some of the retrieved sentences are quite exciting! Building a model that accounts for the meaning of an entire sentence is tough because language is remarkably flexible. Changing a single word can either completely change the meaning of a sentence or leave it unaltered.

We talk to a lot of analysts at Lab41. A recurring theme of these conversations is what they frequently refer to as "result provenance." Translation -- "Are these results any good? I don't have a whole lot of time to research them any further, so will these results hold up under scrutiny?" The old adage about lies, damn lies, and statistics has been around for just about forever.

Today's paper is by Gao Huang, Yu Sun, et al. It introduces a new way to perturb networks during training in order to improve their performance. Before I continue, let me first state that this paper is a real pleasure to read; it is concise and extremely well written. It gives an excellent overview of the motivating problems, previous solutions, and Huang and Sun's new approach. I highly recommended giving it a read!

The next paper from our reading group is by Song Han, Huizi Mao, and William J. Dally. It won the best paper award at ICLR 2016. It details three methods of compressing a neural network in order to reduce the size of the network on disk, improve performance, and decrease run time. Pre-trained convolutional neural networks are too large for mobile devices: AlexNet is 240 MB and VGG-16 is over 552 MB. This seems small when compared to a music library or large video, but the difference is that the networks reside in memory when running.

For the longest time, I was a Google Reader guy. I had a list of sites that I wanted to follow, and Reader was the way that I could quickly triage the news that I cared about. I'm almost embarrassed to admit it because RSS feeds are so last decade, but I like being in control of what I read and what I don't read. When Google Reader was unceremoniously shut down, I shuffled back and forth between different RSS readers until I found Feedly. Feedly makes a solid product, but I've never fully bought in.