Are Language Models Worse than Humans at Following Prompts? It's Complicated