Back to the Drawing Board for Fair Representation Learning