Should LLM Safety Be More Than Refusing Harmful Instructions?