Drowning in Documents: Consequences of Scaling Reranker Inference