Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts