Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models