The neocognitron is a neural network for pattern recognition and feature extraction. An analog CCD parallel processing architecture developed at Lincoln Laboratory is particularly well suited to the computational re(cid:173) quirements of shared-weight networks such as the neocognitron, and imple(cid:173) mentation of the neocognitron using the CCD architecture was simulated. A modification to the neocognitron training procedure, which improves network performance under the limited arithmetic precision that would be imposed by the CCD architecture, is presented.

Now, let's begin simply with the question……. What exactly is a convolutional neural network or CNN in short?. CNN is a type of Artificial Neural Network which is often associated with image classification and processing. Convolutional neural networks are composed of multiple layers of artificial neurons(perceptrons). At its base, it's just a Deep Learning algorithm that can process and assign weights/importance of each pixel of an image to process and differentiate each object in the image.

Convolutional Neural Networks which are the backbones of most of the Computer Vision Applications like Self-Driving Cars, Facial Recognition Systems etc are a special kind of Neural Network architectures in which the basic matrix-multiplication operation is replaced by a convolution operation. They specialize in processing data which has a grid-like topology. Examples include time-series data and image-data which can be thought of as a 2-D grid of pixels. The Convolutional Neural Networks was first introduced by Fukushima by the name Neocognitron in 1980. It was inspired by the hierarchical model of the nervous system as proposed by Hubel and Weisel.

When you look at the following image, you see people, objects, and buildings. It brings up memories of past experiences, similar situations you've encountered. The crowd is facing the same direction and holding up phones, which tells you that this is some kind of event. The person standing near the camera is wearing a T-shirt that hints at what the event might be. As you look at other small details, you can infer much more information from the picture.

The history of Convolutional neural networks have a remote origin. It is actually in 1979, when Professor Kunihiko Fukushima proposed a hierarchical, multilayered artificial neural network called The neocognitron. The neocognitron has been used for solving the problem of handwritten character recognition and some other pattern recognition tasks, and served as the inspiration for convolutional neural networks. But, if you asked about the history of the neocognitron, we simply can tell you that it was inspired by the model proposed by Hubel & Wiesel in 1959. They found two types of cells in the visual primary cortex called simple cells and complex cells, and also proposed a cascading model of these two types of cells for use in pattern recognition tasks.

First of all, apologies for not posting as frequently as I used to. As you might imagine, blogging is not my full time job and I'm currently extremely involved in a very exciting startup (something I'm going to write about soon). On weekends and evening I'm busy with 7mo infant to help care for and altogether that leaves me with very little time. But I'll try to make it better soon, since a lot is going on in the AI space and signs of cooling are visible now all over the place. In this post I'd like to focus on the recent book by Gary Marcus and Ernest Davis, Rebooting AI.

Deep learning also known as deep structured learning or hierarchical learning is part of machine learning based on artificial neural networks. This learning methodology can be supervised, semi-supervised or unsupervised. Deep learning architectures such as neural networks and convolutional neural networks have been applied to fields including computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation where they have produced results comparable to and in some cases superior to human experts. They used a combination of algorithms and mathematics they called "threshold logic" to mimic the thought process. Since that time, Deep Learning has evolved steadily, with only two significant breaks in its development.

Deep Learning, as a branch of Machine Learning, employs algorithms to process data and imitate the thinking process, or to develop abstractions. Deep Learning (DL) uses layers of algorithms to process data, understand human speech, and visually recognize objects. Information is passed through each layer, with the output of the previous layer providing input for the next layer. The first layer in a network is called the input layer, while the last is called an output layer. All the layers between the two are referred to as hidden layers.

What a wonderful treasure trove this paper is! Schmidhuber provides all the background you need to gain an overview of deep learning (as of 2014) and how we got there through the preceding decades. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond. The main part of the paper runs to 35 pages, and then there are 53 pages of references. Now, I know that many of you think I read a lot of papers – just over 200 a year on this blog – but if I did nothing but review these key works in the development of deep learning it would take me about 4.5 years to get through them at that rate! And when I'd finished I'd still be about 6 years behind the then current state of the art!

Many well known neural network techniques for adaptive pattern classification and function approximation are inherently highly parallel, and thus have proven difficult to implement for real-time applications at a reasonable cost.