LEGO: Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion