Large Language Models Know What To Say But Not When To Speak