This year, we saw a dazzling application of machine learning. The OpenAI GPT-2 exhibited impressive ability of writing coherent and passionate essays that exceed what we anticipated current language models are able to produce. The GPT-2 wasn't a particularly novel architecture – it's architecture is very similar to the decoder-only transformer. The GPT2 was, however, a very large, transformer-based language model trained on a massive dataset. In this post, we'll look at the architecture that enabled the model to produce its results. We will go into the depths of its self-attention layer. My goal here is to also supplement my earlier post, The Illustrated Transformer, with more visuals explaining the inner-workings of transformers, and how they've evolved since the original paper. My hope is that this visual language will hopefully make it easier to explain later Transformer-based models as their inner-workings continue to evolve.
LOS ANGELES, CA, Oct. 20, 2019 (GLOBE NEWSWIRE) -- via NEWMEDIAWIRE – iPR Software, the leader in Online Newsrooms, Digital Publishing, Digital Asset Management (DAM) solutions, and customized integrated solutions, announced its largest technology rollout to date at Public Relations Society of America's International Conference in San Diego, California. With the launch of "Metatron," iPR Software's new application empowers Artificial Intelligence (AI) cloud capabilities as well as integrating the power of machine learning into DAM and customized software platforms to increase productivity and corporate asset sharing across multiple customer ecosystems. This latest software release further advances the company's vision for clients to publish their news and information to Traditional and Social media channels and better engage their B2B & B2C audiences while increasing traffic to their branded media and corporate assets. Leading organization's today are utilizing cloud applications to access the latest technology with encryption algorithms they can securely manage, publish, and share rich branded media content. Metatron introduces core, cloud-based software features that enable customers to securely publish and share key digital media and corporate assets, target practical enterprise use cases, increase workflow efficiencies, and automate mundane tasks to reduce data and storage errors.
Since our recent release of Transformers (previously known as pytorch-pretrained-BERT and pytorch-transformers), we've been working on a comparison between the implementation of our models in PyTorch and in TensorFlow. We've released a detailed report where we benchmark each of the architectures hosted on our repository (BERT, GPT-2, DistilBERT, ...) in PyTorch with and without TorchScript, and in TensorFlow with and without XLA. We benchmark them for inference and the results are visible in the following spreadsheet. We would love to hear your thoughts on the process.
About a year ago, top deepfake artist Hao Li came to a disturbing realization: Deepfakes, i.e. the technique of human-image synthesis based on artificial intelligence (AI) to create fake content, is rapidly evolving. In fact, Li believes that in as soon as six months, deepfake videos will be completely undetectable. And that's spurring security and privacy concerns as the AI behind the technology becomes commercialized – and gets in the hands of malicious actors. Li, for his part, has seen the positives of the technology as a pioneering computer graphics and vision researcher, particularly for entertainment. He has worked his magic on various high-profile deepfake applications – from leading the charge in putting Paul Walker into Furious 7 after the actor died before the film finished production, to creating the facial-animation technology that Apple now uses in its Animoji feature in the iPhone X.
Since time immemorial, sound has been a key source of communication within nature. Birds, bees, whistle of the foliage, thunder, the whisper of air, dolphins speaking underwater and many more such forms of sound are some of the visible examples created by nature, with well-defined purposes. If one were to meander through a dense, human uninhabited forestland, and tip toe in silence, the various forms of the above sounds can be experienced in its raw form, and is a fascinating raw, musical experience! A music that does not have a well-defined structure, but nevertheless, an experience worth savouring! We all know that sound and its associated emotions has been a primary source of human communication.