Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment