Small transformer architectures for task switching