Towards leveraging LLMs for Conditional QA