Calibrating BERT-based Intent Classification Models: Part-1

At Expedia Group, we strive to improve our customer satisfaction by providing a frictionless shopping and support experience. A core part of this are machine learning models that power Expedia Group's virtual agents, that are available 24x7 to guide customers through any changes in their travel plans, pull-up information about their upcoming trips, book new trips and much more. Each capability has an associated'intent', which is identified by an intent classification model. However, natural language is messy, and not all customer utterances are actionable -- perhaps not even relevant. Therefore, the predicted probabilities of a classification model -- intent classifier in this case -- should be reliable: a misclassification should not have a high probability.