To Trust or Not To Trust Prediction Scores for Membership Inference Attacks