open questions like, lower bounds, private information, and real-valued feedback, pointed out by reviewers

We thank reviewers for detailed comments and suggestions. We will address all comments in the revision. AIStats'19) considered the problem of learning an optimal action but ignored the contextual information. In this work, we incorporated the contextual information, which is readily available in many applications. The idea might look incremental.