Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations