Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationality