BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models