The Structural Complexity of Matrix-Vector Multiplication