Robust Field-level Likelihood-free Inference with Galaxies