When do random forests fail?