State Space Models on Temporal Graphs: A First-Principles Study