Accelerating Particle and Fluid Simulations with Differentiable Graph Networks for Solving Forward and Inverse Problems