An Analysis for Reasoning Bias of Language Models with Small Initialization