Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks