Sharp Structure-Agnostic Lower Bounds for General Functional Estimation