A typical result of competitive on-line prediction says that, for a giv en benchmark class of prediction strategies, there is a prediction strategy that performs almost as well as the best prediction strategies in the benchmark cla ss. For simplicity, in this paper the performance of a prediction strategy will be measured by the cumulative squared distance between its predictions a nd the true observations, assumed to be real (occasionally complex) numbers . Different methods of competitive on-line predictions (such as Gradient Desce nt, following the perturbed leader, strong and weak aggregating algorithms, d efensive forecasting, etc.) tend to have their narrow "area of expertise": eac h works well for benchmark classes of a specific "size" but is not readily applicable to c lasses of a different size. In this paper we will apply a simple general method based on metric ent ropy to benchmark classes of a wide range of sizes. Typically, this method does not give optimal results, but its results are often not much worse than those given by specialized methods, especially for benchmark classes that are not too massive.

This paper is based on my invited talk at the 19th Annual Conference on Learning Theory (Pittsburgh, PA, June 24, 2006). In recent years COL T invited talks have tended to aim at establishing connections between the traditio nal concerns of the learning community and the work done by other communities (s uch as game theory, statistics, information theory, and optimization). F ollowing this tradition, I will argue that some ideas from the foundations of prob ability can be fruitfully applied in competitive on-line learning. In this paper I will use the following informal taxonomy of predictions (reminiscent of Shafer's [36], Figure 2, taxonomy of probabilities): D-predictions are mere Decisions. They can never be true or false but can be good or bad.

Artificial Intelligence (AI) has recently become a real formal science: the new millennium brought the first mathematically sound, asymptotically optimal, universal problem solvers, providing a new, rigorous foundation for the previously largely heuristic field of General AI and embedded agents. At the same time there has been rapid progress in practical methods for learning true sequence-processing programs, as opposed to traditional methods limited to stationary pattern association. Here we will briefly review some of the new results, and speculate about future developments, pointing out that the time intervals between the most notable events in over 40,000 years or 2^9 lifetimes of human history have sped up exponentially, apparently converging to zero within the next few decades. Or is this impression just a by-product of the way humans allocate memory space to past events?

Predictive learning has traditionally been a standard indu ctive learning, where different sub-problem formulations have been identified. One of the most re presentative is classification, consisting on the estimation of a mapping from the feature sp ace into a finite class space. Depending on the cardinality of the finite class space we are l eft with binary or multiclass classification problems. Finally, the presence or absence o r a "natural" order among classes will separate nominal from ordinal problems. Although two-class and nominal classification problems hav e been dissected in the literature, the ordinal sibling has not yet received a lot of attention, e ven with many learning problems involving classifying examples into classes which have a na tural order. Scenarios in which it is natural to rank instances occur in many fields, such as info rmation retrieval, collaborative filtering, econometric modeling and natural sciences. Conventional methods for nominal classes or for regression problems could be employed to solve ordinal data problems; however, the use of techniques designed specifically for ordered classes yields simpler classifiers, making it easier to inte rpret the factors that are being used to discriminate among classes, and generalises better. Alt hough the ordinal formulation seems conceptually simpler than nominal, some technical di fficulties to incorporate in the algorithms this piece of additional information - the order - may explain the widespread use of conventional methods to tackle the ordinal data problem. This dissertation addresses this void by proposing a nonpar ametric procedure for the classification of ordinal data based on the extension of the original dataset with additional variables, reducing the classification task to the well-known two-clas s problem.

This paper presents a novel approach for using clickthrough data to learn ranked retrieval functions for web search results. We observe that users searching the web often perform a sequence, or chain, of queries with a similar information need. Using query chains, we generate new types of preference judgments from search engine logs, thus taking advantage of user intelligence in reformulating queries. To validate our method we perform a controlled user study comparing generated preference judgments to explicit relevance judgments. We also implemented a real-world search engine to test our approach, using a modified ranking SVM to learn an improved ranking function from preference data. Our results demonstrate significant improvements in the ranking given by the search engine. The learned rankings outperform both a static ranking function, as well as one trained without considering query chains.

The adaptation rule for Vector Quantization algorithms, and consequently the convergence of the generated sequence, depends on the existence and properties of a function called the energy function, defined on a topological manifold. Our aim is to investigate the conditions of existence of such a function for a class of algorithms examplified by the initial ''K-means'' and Kohonen algorithms. The results presented here supplement previous studies and show that the energy function is not always a potential but at least the uniform limit of a series of potential functions which we call a pseudo-potential. Our work also shows that a large number of existing vector quantization algorithms developped by the Artificial Neural Networks community fall into this category. The framework we define opens the way to study the convergence of all the corresponding adaptation rules at once, and a theorem gives promising insights in that direction. We also demonstrate that the ''K-means'' energy function is a pseudo-potential but not a potential in general. Consequently, the energy function associated to the ''Neural-Gas'' is not a potential in general.

We present a novel approach to semi-supervised learning which is based on statistical physics. Most of the former work in the field of semi-supervised learning classifies the points by minimizing a certain energy function, which corresponds to a minimal k-way cut solution. In contrast to these methods, we estimate the distribution of classifications, instead of the sole minimal k-way cut, which yields more accurate and robust results. Our approach may be applied to all energy functions used for semi-supervised learning. The method is based on sampling using a Mul-ticanonical Markov chain Monte-Carlo algorithm, and has a straightforward probabilistic interpretation, which allows for soft assignments of points to classes, and also to cope with yet unseen class types. The suggested approach is demonstrated on a toy data set and on two real-life data sets of gene expression.

In this paper, we study clustering with respect to the k-modes objective function, a natural formulation of clustering for categorical data. One of the main contributions of this paper is to establish the connection between k-modes and k-median, i.e., the optimum of k-median is at most twice the optimum of k-modes for the same categorical data clustering problem. Based on this observation, we derive a deterministic algorithm that achieves an approximation factor of 2. Furthermore, we prove that the distance measure in k-modes defines a metric. Hence, we are able to extend existing approximation algorithms for metric k-median to k-modes. Empirical results verify the superiority of our method.

A method of {\it topological grammars} is proposed for multidimensional data approximation. For data with complex topology we define a {\it principal cubic complex} of low dimension and given complexity that gives the best approximation for the dataset. This complex is a generalization of linear and non-linear principal manifolds and includes them as particular cases. The problem of optimal principal complex construction is transformed into a series of minimization problems for quadratic functionals. These quadratic functionals have a physically transparent interpretation in terms of elastic energy. For the energy computation, the whole complex is represented as a system of nodes and springs. Topologically, the principal complex is a product of one-dimensional continuums (represented by graphs), and the grammars describe how these continuums transform during the process of optimal complex construction. This factorization of the whole process onto one-dimensional transformations using minimization of quadratic energy functionals allow us to construct efficient algorithms.

Experimental physics, along with many other fields in applied and basic research, uses experiments, physical tests, and observations to gain insight into various phenomena and to validate hypotheses and models. Shock p hysics is a field that explores the response of materials to the extremes of p ressure, deformation, and temperature which are present when shock waves interact with those materials [17]. High explosive (HE) or propellant guns are often used to generate these strong shock waves. Many different diagnostic ap proaches have been used to probe these phenomena [8]. Because of the energetic nature of the shock wave drive, often a large amount of experimental equipment is destroyed during the test.