Learning in Discounted-cost and Average-cost Mean-field Games