Field-level simulation-based inference with galaxy catalogs: the impact of systematic effects