One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning