Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer