Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs