We use ReLU as the activation function.

For each convolutional layer, we would firstly apply a ReLU activation function right after the convolution,andthenapplyamaxpoolingwith kernel_size=2,stride=2toextractthefeature map.

Wetheoretically showour method converges for smooth objectives with square regularizers and the convergence dependence on the projection dimension is mild. We also illustrate the benefits of robustness and fairness on a class of linear problems.

Fordetailedresults, please referto Table 2 and Table 3 in Appendix A.6.

Exponential smoothers are a simple and memory efficient way to compute running averages of time series. Here we define and describe practical properties of exponential smoothers for signals observed at constant and variable intervals.