Safe and Efficient Navigation in Extreme Environments using Semantic Belief Graphs