In the previous courses, we have introduced linear and logistic regression, to model a Y variable which is discrete or continuous from one or more Xi variables, in all the examples used to illustrate this technique the modeling was relatively simple, the variable Y was generally modeled by a line parameterized by the variables Xi, but this modeling cannot be applied every time, an aquatic model must be chosen w.r.t to our data, in order to have the best fit. In this course we will study the effect of the choice of this modeling, we will see two cases, the first when the modeling is too weak to model our data, and the second is when the modeling is over-parameterized and that it will over-fit our data. Let's take a simple example and see what different modeling choices will produce in the fit of the data, we will use the following python code to generate and visualize the data, The figure above shows different fits for different choices of modeling assumptions, the first figure shows the simplest choice, modeling by a straight line of our data, in this case, we can notice that the modeling is very weak and we do not end with a good fit to our data, in this case, we are talking about underfitting, that is, the starting hypothesis is too weak for our data set. In this case, we notice that the modeling is over-parameterized, which gives an over-adjustment of our data without having a correct trajectory, we can notice that at the edge, we have a significant oscillation, which can mislead us if we want to predict the value of a new point which is at the edge, in this case, we speak of overfitting, that is to say, that our starting hypothesis is over-parameterized for our data. To sum up, when modeling data we can face two problems, first we can have a hypothesis that fails to model our data, and second, we can have a hypothesis that is over-parameterized and which will over-fit our data without the power to generalize to new examples, a trade-off must be made between the desired level of fit and the ability to generalize to new cases in order to have the best fit to the data.

Under the hood of every cryptocurrency protocol, we will always find that Blockchain Technology is the engine that allows it to keep running. If we trace back the technologies that made its application possible, we will find that the science behind it has been around for decades, and only just recently became ubiquitous. Gradual changes over the last couple of decades contributed to the recent uptake of cryptocurrencies. More and more companies are now able to collect increasingly larger amounts of data. All that data was just lying there, like gasoline waiting for the spark that would transform it into valuable, usable information with real-world applications.

The recent COVID-19 pandemic has promoted vigorous scientific activity in an effort to understand, advice and control the pandemic. Data is now freely available at a staggering rate worldwide. Unfortunately, this unprecedented level of information contains a variety of data sources and formats, and the models do not always conform to the description of the data. Health officials have recognized the need for more accurate models that can adjust to sudden changes, such as produced by changes in behavior or social restrictions. In this work we formulate a model that fits a ``SIR''-type model concurrently with a statistical change detection test on the data. The result is a piece wise autonomous ordinary differential equation, whose parameters change at various points in time (automatically learned from the data). The main contributions of our model are: (a) providing interpretation of the parameters, (b) determining which parameters of the model are more important to produce changes in the spread of the disease, and (c) using data-driven discovery of sudden changes in the evolution of the pandemic. Together, these characteristics provide a new model that better describes the situation and thus, provides better quality of information for decision making.

In this study, we propose using a neural embedding model-graph neural network (GNN)- that leverages the heterogeneous features of urban areas and their interactions captured by human mobility network to obtain vector representations of these areas. Using large-scale high-resolution mobility data sets from millions of aggregated and anonymized mobile phone users in 16 metropolitan counties in the United States, we demonstrate that our embeddings encode complex relationships among features related to urban components (such as distribution of facilities) and population attributes and activities. The spatial gradient in each direction from city center to suburbs is measured using clustered representations and the shared characteristics among urban areas in the same cluster. Furthermore, we show that embeddings generated by a model trained on a different county can capture 50% to 60% of the emergent spatial structure in another county, allowing us to make cross-county comparisons in a quantitative way. Our GNN-based framework overcomes the limitations of previous methods used for examining spatial structures and is highly scalable. The findings reveal non-linear relationships among urban components and anisotropic spatial gradients in cities. Since the identified spatial structures and gradients capture the combined effects of various mechanisms, such as segregation, disparate facility distribution, and human mobility, the findings could help identify the limitations of the current city structure to inform planning decisions and policies. Also, the model and findings set the stage for a variety of research in urban planning, engineering and social science through integrated understanding of how the complex interactions between urban components and population activities and attributes shape the spatial structures in cities.

It is crucially important to estimate unknown parameters in earth system models by integrating observation and numerical simulation. For many applications in earth system sciences, the optimization method which allows parameters to temporally change is required. Here I present an efficient and practical method to estimate the time-varying parameters of relatively low dimensional models. I propose combining offline batch optimization and online data assimilation. In the newly proposed method, called Hybrid Offline Online Parameter Estimation with Particle Filtering (HOOPE-PF), I constrain the estimated model parameters in sequential data assimilation to the result of offline batch optimization in which the posterior distribution of model parameters is obtained by comparing the simulated and observed climatology. The HOOPE-PF outperforms the original sampling-importance-resampling particle filter in the synthetic experiment with the toy model and the real-data experiment with the conceptual hydrological model. The advantage of HOOPE-PF is that the performance of the online data assimilation is not greatly affected by the hyperparameter of ensemble data assimilation which contributes to inflating the ensemble variance of estimated parameters.

Machine learning (ML) models serve critical functions, such as classifying loan applicants as good or bad risks. Each model is trained under the assumption that the data used in training, and the data used in field come from the same underlying unknown distribution. Often this assumption is broken in practice. It is desirable to identify when this occurs in order to minimize the impact on model performance. We suggest a new approach to detect change in the data distribution by identifying polynomial relations between the data features. We measure the strength of each identified relation using its R-square value. A strong polynomial relation captures a significant trait of the data which should remain stable if the data distribution does not change. We thus use a set of learned strong polynomial relations to identify drift. For a set of polynomial relations that are stronger than a given desired threshold, we calculate the amount of drift observed for that relation. The amount of drift is estimated by calculating the Bayes Factor for the polynomial relation likelihood of the baseline data versus field data. We empirically validate the approach by simulating a range of changes in three publicly-available data sets, and demonstrate the ability to identify drift using the Bayes Factor of the polynomial relation likelihood change.

Route recommendation is significant in navigation service. Two major challenges for route recommendation are route representation and user representation. Different from items that can be identified by unique IDs in traditional recommendation, routes are combinations of links (i.e., a road segment and its following action like turning left) and the number of combinations could be close to infinite. Besides, the representation of a route changes under different scenarios. These facts result in severe sparsity of routes, which increases the difficulty of route representation. Moreover, link attribute deficiencies and errors affect preciseness of route representation. Because of the sparsity of routes, the interaction data between users and routes are also sparse. This makes it not easy to acquire user representation from historical user-item interactions as traditional recommendations do. To address these issues, we propose a novel learning framework R4. In R4, we design a sparse & dense network to obtain representations of routes. The sparse unit learns link ID embeddings and aggregates them to represent a route, which captures implicit route characteristics and subsequently alleviates problems caused by link attribute deficiencies and errors. The dense unit extracts implicit local features of routes from link attributes. For user representation, we utilize a series of historical navigation to extract user preference. R4 achieves remarkable performance in both offline and online experiments.

We propose a fast non-gradient based method of rank-1 non-negative matrix factorization (NMF) for missing data, called A1GM, that minimizes the KL divergence from an input matrix to the reconstructed rank-1 matrix. Our method is based on our new finding of an analytical closed-formula of the best rank-1 non-negative multiple matrix factorization (NMMF), a variety of NMF. NMMF is known to exactly solve NMF for missing data if positions of missing values satisfy a certain condition, and A1GM transforms a given matrix so that the analytical solution to NMMF can be applied. We empirically show that A1GM is more efficient than a gradient method with competitive reconstruction errors.

Ordinary differential equation (ODE) is an important tool to study the dynamics of a system of biological and physical processes. A central question in ODE modeling is to infer the significance of individual regulatory effect of one signal variable on another. However, building confidence band for ODE with unknown regulatory relations is challenging, and it remains largely an open question. In this article, we construct post-regularization confidence band for individual regulatory function in ODE with unknown functionals and noisy data observations. Our proposal is the first of its kind, and is built on two novel ingredients. The first is a new localized kernel learning approach that combines reproducing kernel learning with local Taylor approximation, and the second is a new de-biasing method that tackles infinite-dimensional functionals and additional measurement errors. We show that the constructed confidence band has the desired asymptotic coverage probability, and the recovered regulatory network approaches the truth with probability tending to one. We establish the theoretical properties when the number of variables in the system can be either smaller or larger than the number of sampling time points, and we study the regime-switching phenomenon. We demonstrate the efficacy of the proposed method through both simulations and illustrations with two data applications.

We consider the use of deep learning for parameter estimation. We propose Bias Constrained Estimators (BCE) that add a squared bias term to the standard mean squared error (MSE) loss. The main motivation to BCE is learning to estimate deterministic unknown parameters with no Bayesian prior. Unlike standard learning based estimators that are optimal on average, we prove that BCEs converge to Minimum Variance Unbiased Estimators (MVUEs). We derive closed form solutions to linear BCEs. These provide a flexible bridge between linear regrssion and the least squares method. In non-linear settings, we demonstrate that BCEs perform similarly to MVUEs even when the latter are computationally intractable. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance. Examples include distributed sensor networks and data augmentation in test-time. In such applications, unbiasedness is a necessary condition for asymptotic consistency.