Backprop is not just the chain rule

Almost everyone I know says that "backprop is just the chain rule." Although that's basically true, there are some subtle and beautiful things about automatic differentiation techniques (including backprop) that will not be appreciated with this dismissive attitude. This leads to a poor understanding. As I have ranted before: people do not understand basic facts about autodiff. Let's try to understand the difference between autodiff and the type of differentiation that you learned in calculus, which is called symbolic differentiation.