Comparing the Performance of Graphical Structure Learning Algorithms with TETRAD

Often researchers are faced with the problem of choosing an algorithm from among possibly dozens of relevant algorithms for a particular task. This can be time-consuming and errorprone; one must try each algorithm in turn, vary the parameters for that algorithm, run it in simulation on common data sets that hopefully reflect the properties of the real data of interest, and somehow try to discern which algorithm has the best performance over the range of cases under study. Reading research papers for descriptions and evaluations of algorithms is often unhelpful, since papers tend to compare only one or two algorithms at a time, on performance statistics that may not be of interest to the user, using simulations that are not appropriate for the domain. Ideally the user could directly compare a range of algorithms, on data of their choosing, and on performance statistics of interest to them, so that they could make an informed decision as to which algorithm(s) may be best suited to the user's particular purpose. It is a task we feel is best automated and used early and often. We focus on the structure learning algorithms in the TETRAD freeware (http://www.phil.cmu.edu/tetrad). Within TETRAD, we have created a tool for comparing algorithms, both "basic" algorithms with