Machine learning leads mathematicians to unsolvable problem