Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications