Machine learning on mobile: on the device or in the cloud?

So you've decided it's time to add some of this hot new machine learning or deep learning stuff into your app… Great! But what are your options? Let's say you want to make a "celebrity match" app that tells people which famous person they most look alike. You need to first gather a lot of photos of celebrities' faces. Then you train a deep learning network on these photos to teach it what each celebrity looks like. The model you're using would be some kind of convolutional neural network and you've trained it for the specific purpose of comparing people's faces with the faces of celebrities. Training is a difficult and expensive process.