Learning Signal-Agnostic Manifolds of Neural Fields