When designing reinforcement learning (RL) agents, a designer communicates the desired agent behavior through the definition of reward functions - numerical feedback given to the agent as reward or punishment for its actions. However, mapping desired behaviors to reward functions can be a difficult process, especially in complex environments such as autonomous racing. In this paper, we demonstrate how current foundation models can effectively search over a space of reward functions to produce desirable RL agents for the Gran Turismo 7 racing game, given only text-based instructions. Through a combination of LLM-based reward generation, VLM preference-based evaluation, and human feedback we demonstrate how our system can be used to produce racing agents competitive with GT Sophy, a champion-level RL racing agent, as well as generate novel behaviors, paving the way for practical automated reward design in real world applications.

Racing autonomous cars faster than the best human drivers has been a longstanding grand challenge for the fields of Artificial Intelligence and robotics. Recently, an end-to-end deep reinforcement learning agent met this challenge in a high-fidelity racing simulator, Gran Turismo. However, this agent relied on global features that require instrumentation external to the car. This paper introduces, to the best of our knowledge, the first super-human car racing agent whose sensor input is purely local to the car, namely pixels from an ego-centric camera view and quantities that can be sensed from on-board the car, such as the car's velocity. By leveraging global features only at training time, the learned agent is able to outperform the best human drivers in time trial (one car on the track at a time) races using only local input features. The resulting agent is evaluated in Gran Turismo 7 on multiple tracks and cars. Detailed ablation experiments demonstrate the agent's strong reliance on visual inputs, making it the first vision-based super-human car racing agent.

Nearly two years after its prototype debut and eight months after its public beta, Sony's GT Sophy racing AI for Gran Turismo 7 is back, and going by Gran Turismo Sophy 2.0 now. It will be available to all PlayStation 5 users as part of the GT7 Spec II Update (Patch Update 1.40) being released on Wednesday, November 2 at 2 a.m. We got our first look at the Sophy system back in February 2022. At that point it was already handily beating professional Gran Turismo players. "Gran Turismo Sophy is a significant development in AI whose purpose is not simply to be better than human players, but to offer players a stimulating opponent that can accelerate and elevate the players' techniques and creativity to the next level," Sony AI CEO, Hiroaki Kitano, said at the time.

The world's leading Gran Turismo 7 players have had their shot at beating Sony's AI racer, GT Sophy, and they lost spectacularly. Now, the bot has added drifting to its move set to rub salt on the wounds, as seen in a video from the Gran Turismo World Series 2023 event. The video clearly shows an AI-controlled vehicle drifting around the track like an absolute maniac. Meanwhile, most people can't pull off successful drifts in Mario Kart, let alone in realistic racers like Gran Turismo. GT Sophy, you can guess what the GT stands for, is the result of more than six years of development between Sony AI and Sony Interactive Entertainment, and utilizes deep reinforcement learning methods.

A solid six percent of Americans think they can out-punch a Grizzly bear, another one in eight men think they can beat 23-time grand slam champion Serena Williams at tennis. On February 21st, this proud internet tradition of being very loud and very wrong about your physical prowess continues! On Tuesday, gamers around the world will get their shot at racing Sony AI's GT Sophy -- the one that's already wiping the floor with folks who get paid to play this game professionally -- when it arrives in the rev1.29 update for Gran Turismo 7 on the Playstation 5. GT7 players will be able to access a special "Gran Turismo Sophy Race Together" mode from February 21st at 1am ET, when the update arrives. Players will face off against four separate GT Sophy AI opponents, all of whom's vehicles are specced slightly differently so you're not going up against a quartet of clones, in a four-circuit series striated by difficulty (beginner-intermediate-expert). "The difference [between racers] is that, it's essentially the power you have versus the other cars on the track," Michael Spranger told Engadget.

A pivotal moment in my early career as a Ph.D. student at Carnegie Mellon University was when I saw a demonstration of the first soccer-playing robots in the summer of 1994. They were from Alan Mackworth's lab at the University of British Columbia, and I became immediately inspired to try to understand the intelligence required to play soccer. At the time, most AI researchers were focused on much more abstract planning tasks, or short-time-duration skills for individual robots. I saw the opportunity to use the game of soccer to, for the first time, investigate new methods for enabling collaborative (with teammates) and adversarial (against opponents) multi-robot planning in relatively complex domains. It so happened that one of the few other people in the world at the time who was thinking about robot soccer as a challenge domain for AI was Hiroaki Kitano at Sony (currently the CTO of Sony and CEO of Sony AI).

In Early February 2022, Sony's "first AI breakthrough", GT Sophy, made its appearance on the cover page of Nature magazine [2]. GT Sophy is a racing AI built to match with world-class level players in Gran Turismo Sport, the latest installation of the legendary game series on PlayStation 4. GT7 is famous for its extremely realistic simulation of real-life racing experience, which largely complicates the production of GT Sophy at the early stage. Every tiny decision that GT Sophy makes may change the result of the race entirely. Thus, there is little simplification can be done to the training process. Sony's AI team needs to take all possible factors, like drifting effects caused by the passage of nearby cars, to perform any estimation.

I got my undergraduate degree from MIT in mechanical engineering. I didn't feel ready to be an engineer, so I went to the University of Michigan to get a Masters degree in mechanical engineering. Along the way, I got a job programming at the university, and eventually decided to go back to school and get a Ph.D. in computer science. From there, I became a professor in the Computer Science Department at North Carolina State. In 2004, as I went up for tenure, my roommate from my undergraduate days at MIT came up with an idea for a robotic warehouse system, and convinced me to help him start what became Kiva Systems.

Last week OpenAI moved DALL-E 2, the image generation tool, into Beta (the company hopes to expand its current user base to 1 million) while granting users the "the right to reprint, sell, and merchandise" images they generate with DALL-E. This is useful for users who wish to use the generated images for commercial purposes, like making illustrations for children's books. Other openly available AI image generation models face similar problems. Also, it's not clear if OpenAI violated any IP laws for just training on these Internet images and then commercializing their model. While the UK is exploring allowing commercial use of models trained on public but trademarked data, the U.S. may not follow suit.

As part of a special industry session at IJCAI-ECAI 2022, the 2022 ACM SIGAI Industry Award for Excellence in Artificial Intelligence was presented to the team behind Sony's Gran Turismo Sophy project. This project was developed by Sony AI, Sony Interactive Entertainment and Polyphony Digital. Gran Turismo (GT) Sophy is a collection of agents trained using reinforcement learning (RL) techniques to race in Gran Turismo, a hyper-realistic, physics-based automotive racing simulator. The GT Sophy team developed novel, state-of-the-art RL methods for this purpose. Racing against some of the world's best e-sports drivers, GT Sophy has not only performed at world-class levels, it also won a team event in October 2021 by an impressive margin.