Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism