Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks

In spite of the great success of deep learning technologies, training Imagine a realtime AI-as-a-Service (AIaaS) system that can instantly and delivery of a practically serviceable model is still a highly deliver resource-efficient models for any on-demand tasks to multiple time-consuming process. Furthermore, a resulting model is usually users who can be non-expertise in AI. For example, a mobile too generic and heavyweight, and hence essentially goes through user should quickly adapt to a dynamically changing environment another expensive model compression phase to fit in a resourcelimited (e.g., entering a restaurant in an animal theme park and returning to device like embedded systems. Inspired by the fact that see animals having lunch) preferably without continually training a machine learning task specifically requested by mobile users is a large generic model due to the resource constraints. In this case, often much simpler than it is supported by a massive generic model, such a realtime AIaaS system would be an adequate solution so that this paper proposes a framework, called Pool of Experts (PoE), that a user can immediately be given a properly functioning model by instantly builds a lightweight and task-specific model without any the system without any training overhead. Although AIaaS is getting training process. For a realtime model querying service, PoE first popular to the point that global vendors have initiated code-free extracts a pool of primitive components, called experts, from a machine learning platforms, it still takes a while to train a specific well-trained and sufficiently generic network by exploiting a novel neural network that achieves a practical level of the accuracy even conditional knowledge distillation method, and then performs our with the help of transfer learning from a massive pretrained model, train-free knowledge consolidation to quickly combine necessary often referred to as oracle. To realize a realtime AIaaS system that experts into a lightweight network for a target task. Thanks to this immediately builds a task-specific and lightweight model, would it train-free property, in our thorough empirical study, PoE can build be possible to preprocess such an oracle neural network so that any a fairly accurate yet compact model in a realtime manner, whereas queried knowledge is efficiently extracted with no training at all? it takes a few minutes per query for the other training methods to The fundamental problem of extracting knowledge from a pretrained achieve a similar level of the accuracy.