Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents