Explanations play an important role in AI systems in general and case-based reasoning (CBR) in particular. They can be used for reasoning by the system itself or presented to the user to explain solutions proposed by the system. In our work we investigate the approach where causal explanations are automatically extracted from textual incident reports and reused in a CBR system for incident analysis. The focus of this paper is evaluation of such explanations. We propose an automatic evaluation measure based on the ability of explanations to provide an explicit connection between the problem description and the solution parts of a case.