Towards Extreme Pruning of LLMs with Plug-and-Play Mixed Sparsity