Efficient Spatially Sparse Inferencefor Conditional GANsand Diffusion Models