[D] Simple Questions Thread July 05, 2020

Okay so I'm gonna get out front that while I've studied artificial neural networks, I don't know a whole lot about them. So I hope that these questions are not extremely noobish. I know a little about activation functions, which are (as I understand) really important in most or all forms of neural networks, as they introduce a meaningful non-linearity into the network. Otherwise the network would just be one complicated linear function. And there are several common activation functions like rectified linear, softplus, and sigmoid.