Neural Networks, Types, and Functional Programming -- colah's blog

When I hear colleagues talk at a high level about their models, it has a very different feeling to it than people talking about more classical models. People talk about things in lots of different ways, of course – there's lots of variance in how people see deep learning – but there's often an undercurrent that feels very similar to functional programming conversations. It feels like a new kind of programming altogether, a kind of differentiable functional programming. One writes a very rough functional program, with these flexible, learnable pieces, and defines the correct behavior of the program with lots of data. Then you apply gradient descent, or some other optimization algorithm. The result is a program capable of doing remarkable things that we have no idea how to create directly, like generating captions describing images. It's the natural intersection of functional programming and optimization, and I think it's beautiful. I find this idea really beautiful. At the same time, this is a pretty strange article and I feel a bit weird posting it.