Standard FL approaches train a unique model for all clients [47, 32, 38, 29, 48].

The first algorithm based on multi-armed bandit (MAB) choosesk < K arms (neighbors) repeatedly.

By transferring both features and gradients between different layers, shortcut connections explored by ResNets allow us to effectively train very deep neural networks up to hundreds of layers. However, the additional computation costs induced by those shortcuts are often overlooked. For example, during online inference, the shortcuts in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, because the features in the preceding layers cannot be released until the subsequent calculation is completed. In this work, for the first time, we consider training the CNN models with shortcuts and deploying them without. In particular, we propose a novel joint-training framework to train plain CNN by leveraging the gradients of the ResNet counterpart.

By transferring both features and gradients between different layers, shortcut connections explored by ResNets allow us to effectively train very deep neural networks up to hundreds of layers. However, the additional computation costs induced by those shortcuts are often overlooked. For example, during online inference, the shortcuts in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, because the features in the preceding layers cannot be released until the subsequent calculation is completed. In this work, for the first time, we consider training the CNN models with shortcuts and deploying them without. In particular, we propose a novel joint-training framework to train plain CNN by leveraging the gradients of the ResNet counterpart.

In this post we conclude our summaries of the NeurIPS invited talks from the 2020 meeting. In this final instalment, we cover the talks by Marloes Maathuis (ETH Zurich) and Anthony M Zador (Cold Spring Harbor Laboratory). Marloes began her talk on causal learning with a simple example of the phenomenon known as Simpson's paradox, in which a trend appears in several different groups of data but disappears or reverses when these groups are combined. She also talked about the importance of considering causality when making decisions based on such data. Marloes went on to explain the difference between causal and non-causal questions. Non-causal questions are about predictions in the same system, for example, predicting the cancer rate among smokers.

As part of the workshop programme at NeurIPS2020, Climate Change AI (CCAI) held an all-day session on "Tackling climate change with machine learning". You can watch the talks from this side event in full in a recording provided by CCAI. In this workshop, the speakers, from both industry and academia, discuss how artificial intelligence and remote sensing can be used to monitor global carbon impact. They also consider trust and accountability issues relating to governments, companies, and international projects. You can find out more about this event, and the main workshop, here.

In this post we continue our summaries of the NeurIPS invited talks from the 2020 meeting. Here, we cover the talks by Chris Bishop (Microsoft Research) and Saiph Savage (Carnegie Mellon University). Chris began his talk by suggesting that now is a particularly exciting time to be involved in AI. What he termed "the real AI revolution" has nothing to do with artificial general intelligence (AGI), but is driven by the way we create software, and hence new technology. Machine learning is becoming ubiquitous and can be used to solve many problems that cannot, yet, be solved using other methods.

There were seven interesting and varied invited talks at NeurIPS this year. Here, we summarise the first three, which were given by Charles Isbell (Georgia Tech), Jeff Shamma (King Abdullah University of Science and Technology) and Shafi Goldwasser (UC Berkeley, MIT and Weizmann Institute of Science). The invited talks kicked off in style with a presentation from Charles Isbell. He had posted a teaser on Twitter indicating that he was trying something new with the format, and it certainly did not disappoint. The talk received rave reviews during both the live chat channel and afterwards on social media.

It's been a busy few days at NeurIPS 2020 so far with all manner of talks, workshops, tutorials and socials on offer. This selection of tweets gives a flavour of the various events and discussions taking place. Go watch it right now, you won't regret it! Interesting talk by Chris Bishop at #NeurIPS2020 Basic or Applied research is not a 1D space. Next up at #NeurIPS2020: Shafi Goldwasser presenting on three works about privacy, verifiability, and robustness in machine learning.