TAMI: Taming Heterogeneity in Temporal Interactions for Temporal Graph Link Prediction