How to select a threshold for acting using confidence scores

Many Watson Developer Cloud services and other cognitive software systems provide a numerical score for how confident the system is in some result. For example, the IBM Watson Retrieve and Rank service provides a Ranker confidence score. The IBM Watson Natural Language Classifier also provides a confidence score as do Watson services for language identification, speech processing, entity and relation detection, etc. Some of these confidence scores may have some meaningful interpretation as a probability (e.g., a probability that some result is "correct" or "relevant"). You cannot assume that all results with a very high confidence score are very good and all results with a very low confidence score are very bad.