What's in Your "Safe" Data?: Identifying Benign Data that Breaks Safety