Enforcing Fairness Where It Matters: An Approach Based on Difference-of-Convex Constraints