Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos