Statistical and Computational Guarantees for Influence Diagnostics