The unreasonable effectiveness of the forget gate