Expressive TTS Driven by Natural Language Prompts Using Few Human Annotations