On the Condition Number Dependency in Bilevel Optimization