How To Define A ReLU Layer In PyTorch

Now that we know how to define a sequential container and a 2D convolutional layer, the next step is to learn how to define the activator layers that we will place between our convolutional layers. For this, we want to import torch.nn And for a specific example, we will also want to import the random library for Pi. Two issues that can arise when optimizing a neural network are second order effects in activation functions and saturation of an activated unit. Second order effects cause issues because linear functions are more easily optimized than their non-linear counterparts.