Ergodicity of the underdamped mean-field Langevin dynamics