In this contributed article, Anupam Datta, Co-Founder, President, and Chief Scientist of TruEra, discusses Zillow and what went wrong with the AI models. For AI and ML models to perform for profitable outcomes, especially for high stakes models like Zillow’s, it is crucial to have serious AI governance supported by tools for monitoring and debugging, which includes having qualified humans-in-the-loop to adjust to major market shifts that can arise during unexpected events.

The first quarter delivered home-sale profits that were more than twice as high as anticipated, the company said. Zillow expected to make money primarily from transaction fees and from services such as title insurance--not from making a killing on the flip. The company's algorithm, which was supposed to predict housing prices, didn't seem to understand the market. Zillow was also behind on its target for home purchases. By the summer, it had the opposite problem, the company later acknowledged.

In February, Zillow appeared so confident in its ability to use artificial intelligence to estimate the value of homes that it announced a new option: for certain homes, its so-called "Zestimate" would also represent an initial cash offer from the company to purchase the property. The move, touted by a company exec at the time as "an exciting advancement," was intended to streamline the process for homeowners considering selling to Zillow as part of its home-flipping business. Zillow promoted this option as a way to make it convenient to sell a home while minimizing interactions with others during the pandemic. Just eight months later, however, the company is shutting down that business, Zillow Offers, entirely. The decision, announced last week, marks a stunning defeat for Zillow.

In 1997, IBM supercomputer Deep Blue made a move against chess champion Garry Kasparov that left him stunned. The computer's choice to sacrifice one of its pieces seemed so inexplicable to Kasparov that he assumed it was a sign of the machine's superior intelligence. Shaken, he went on to resign his series against the computer, even though he had the upper hand. Fifteen years later, however, one of Deep Blue's designers revealed that fateful move wasn't the sign of advanced machine intelligence -- it was the result of a bug. Today, no human can beat a computer at chess, but the story still underscores just how easy it is to blindly trust AI when you don't know what's going on.

Zillow, an online marketplace that facilitates the buying, selling, renting, financing, and remodeling of homes, employs lots of AI technologies to do things like estimate home prices. But the output of AI systems like these can be opaque, creating a "black box" problem where practitioners and customers can't audit the systems properly. Without transparency, serious problems like algorithmic bias can persist undetected, and trust in the models becomes impossible. For obvious ethical reasons, this is why explainable AI (XAI) is so crucial to the creation and deployment of AI systems, but pragmatically, it's also key to the success of AI-powered products and services from companies like Zillow. David Fagnan, director of applied science on the Zillow Offers team, discussed with VentureBeat how and why XAI is indispensable for the company.