Formally, the CCA problem is defined as follows.

Theorem 4.2.Thefollowingholdsfor Algorithm 2: withprobabilityatleast1 , forallt T hP Pt,Ci 32 log ( 3e / ) ( C)2 t+ 1 1 , where = (C) Theempirical implementation condition allowsusCt, with specified components, 7 1: Experimentsonsyntheticdata.

These forachievingT requires onthepriorB> 1. 2.1 Generalizing Choosing Follow [Kalaiand worst-case FPLalgorithm Follow (FPML).

The rectified linear unit (ReLU) [Fukushima, 1980, Nair and Hinton, 2010] activation has been by far the most widely used nonlinearity and successful building block in deep neural networks (DNNs).

Amotivating question behind the results inthis paper isto understand the hierarchyoffunction classes exactly represented by neural networks ofincreasing depth.