Large language models (LLMs) have shown great potential to solve varieties of natural language processing (NLP) tasks, including mathematical reasoning. By applying self-compare fine-tuning, we have enhanced mathematical reasoning abilities of Skywork-13B-Base remarkably. On GSM8K, SkyMath outperforms all known open-source models of similar size and has established a new SOTA performance. On dataset MATH and out-of-domain dataset CMath, SkyMath also achieves a high accuracy rate, showing remarkable generalizability to varieties of math problems. Moreover, compared to traditional AI methods, LLMs gain unparalleled advantages in these landscapes.