Opening Up the Black Box: Auditing Google's Top Stories Algorithm

Auditing algorithms has emerged as a methodology for holding algorithms accountable by testing whether they are fair. This process often relies on the repeated use of a platform to record inputs and their corresponding outputs. For example, to audit Google search, one repeatedly inputs queries and captures the received search pages. The goal is then to discover, in the collected data, patterns that will reveal the ``secrets'' of algorithmic decision making. This knowledge discovery process makes some algorithm auditing tasks great applications for data mining techniques. In this paper, we introduce one particular algorithm audit, that of Google's Top stories. We describe the process of data collection, exploration, and analysis for this application and share some of the gleaned insights. Concretely, our analysis suggests that Google might be trying to burst the famous ``filter bubble'' by choosing less known publishers for the 3rd position in the Top stories.