Toward Inference-optimal Mixture-of-Expert Large Language Models