Reading this article, one can notice small nuances that show the whole text was written via software. My personal two cents on the topic of automated content creation is that AI can be used as a source of inspiration, or as a basic structure idea. AI can be a great tool for brainstorming, and getting an essential outline of the issues you want to talk or write about, but it will not become the main source of creation. Nothing beats creativity and experience, AI models react from input to create an output. Essentially, AI can be a great and useful tool for overcoming writer's block and helping with content creation.

Called RETRO (for "Retrieval-Enhanced Transformer"), the AI matches the performance of neural networks 25 times its size, cutting the time and cost needed to train very large models. The researchers also claim that the database makes it easier to analyze what the AI has learned, which could help with filtering out bias and toxic language. "Being able to look things up on the fly instead of having to memorize everything can often be useful, in the same way as it is for humans," says Jack Rae at DeepMind, who leads the firm's research in large language models. Language models generate text by predicting what words come next in a sentence or conversation. The larger a model, the more information about the world it can learn during training, which makes its predictions better.

Over the past few weeks, we made several improvements to our transformers and tokenizers libraries, with the goal of making it way easier to train a new language model from scratch. In this post we'll demo how to train a "small" model (84 M parameters 6 layers, 768 hidden size, 12 attention heads) – that's the same number of layers & heads as DistilBERT – on Esperanto. Esperanto is a constructed language with a goal of being easy to learn. You won't need to understand Esperanto to understand this post, but if you do want to learn it, Duolingo has a nice course with 280k active learners. First, let us find a corpus of text in Esperanto.