Neural Information Processing Systems http://nips.cc/

In order to help clarify our contributions and or-2 ganize them for readers, we provide the following table to summarize the differences between regrets.3 REVIEWER 4 Thank you for your comments. Concept drift occurs when the optimal model attimetmay no longer bethe optimal model10 at timet+1. Consider an online learning problem with concept drift withT = 3 time periods and loss functions:11 f1(x) = (x 1)2,f2(x) = (x 2)2,f3(x) = (x 3)2. Figure 1: SGD online with momentum Theoretical motivation via Calibration: A more formal motivation of our regret23 can be related to the concept of calibration [1]. The comment on line 110 can be24 rewritten as: If the updates{x1,,xT} are well-calibrated, then perturbingxt by25 anyucannot substantially reduce the cumulative loss.Hence, itcan besaid that the26 sequence {x1,,xT} is asymptotically calibrated with respect to{f1,,fT} if:27 Weindeedranexperiments usingSGDwithmomentum forvariousdecayparameters andconcluded thatSGDwith36 momentum is not even as stable as SGD-online (standard SGD without momentum) as shown in Figure 1.

Each AvehicleC starts sumofcityC, after again.

Word Sense Disambiguation (WSD)4 [36] is a dataset to identify the most appropriate sense (out of three given senses) of the word "president" in a given paragraph.

In the supplementary material, we first introduce the pseudo algorithm of DA W . Then we clarify the Then, we provide a more detailed explanation of Figures 1, 2, 4, and 5, which are slightly abbreviated due to the limited space of the main paper. In the naive pseudo-labeling method, all pseudo labels are enrolled into training, i.e., E 1 + E 2, which is guaranteed by theoretical functional analysis in the next section. Inequality 45 holds true at all times. In this section, we provide more qualitative results between ours and other competitors.