Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs