Multi-modal graph neural networks for localized off-grid weather forecasting