Puzzle Solving without Search or Human Knowledge: An Unnatural Language Approach

For natural language generation (NLG), the transformer architecture provides a scalable mechanism to encode long-range dependencies needed to output plausible text narratives. Transformers (Vaswani, et al., 2017) have rapidly advanced to rival or overtake other deep learning architectures such as convolutional neural networks (CNN). Initially developed to handle long-term language dependencies, this approach over-weights important relations via the "attention" method rather than attempting to localize dependencies (CNN) or grow dense networks for all weights. While the resulting sparse network extends available long-term connections needed to relate distant parts-of-speech or sentence context, the net effect has grown to massive models now in the trillions of connection weights (Child, et al., 2019). This approach has since found application in other fields unrelated to the original language modeling, such as non-local effects needed for visual context problems.