How can we enable machines to make sense of the world, and become better at learning? To approach this goal, I believe viewing intelligence in terms of many integral aspects, and also a universal two-term tradeoff between task performance and complexity, provides two feasible perspectives. In this thesis, I address several key questions in some aspects of intelligence, and study the phase transitions in the two-term tradeoff, using strategies and tools from physics and information. Firstly, how can we make the learning models more flexible and efficient, so that agents can learn quickly with fewer examples? Inspired by how physicists model the world, we introduce a paradigm and an AI Physicist agent for simultaneously learning many small specialized models (theories) and the domain they are accurate, which can then be simplified, unified and stored, facilitating few-shot learning in a continual way. Secondly, for representation learning, when can we learn a good representation, and how does learning depend on the structure of the dataset? We approach this question by studying phase transitions when tuning the tradeoff hyperparameter. In the information bottleneck, we theoretically show that these phase transitions are predictable and reveal structure in the relationships between the data, the model, the learned representation and the loss landscape. Thirdly, how can agents discover causality from observations? We address part of this question by introducing an algorithm that combines prediction and minimizing information from the input, for exploratory causal discovery from observational time series. Fourthly, to make models more robust to label noise, we introduce Rank Pruning, a robust algorithm for classification with noisy labels. I believe that building on the work of my thesis we will be one step closer to enable more intelligent machines that can make sense of the world.

Which variables does an agent have an incentive to control with its decision, and which variables does it have an incentive to respond to? We formalize these incentives, and demonstrate unique graphical criteria for detecting them in any single-decision causal influence diagram. To this end, we introduce structural causal influence models, a hybrid of the influence diagram and structural causal model frameworks. Finally, we illustrate how these incentives predict agent incentives in both fairness and AI safety applications.

Online Social Networks(OSNs) have established virtual platforms enabling people to express their opinions, interests and thoughts in a variety of contexts and domains, allowing legitimate users as well as spammers and other untrustworthy users to publish and spread their content. Hence, the concept of social trust has attracted the attention of information processors/data scientists and information consumers/business firms. One of the main reasons for acquiring the value of Social Big Data (SBD) is to provide frameworks and methodologies using which the credibility of OSNs users can be evaluated. These approaches should be scalable to accommodate large-scale social data. Hence, there is a need for well comprehending of social trust to improve and expand the analysis process and inferring the credibility of SBD. Given the exposed environment's settings and fewer limitations related to OSNs, the medium allows legitimate and genuine users as well as spammers and other low trustworthy users to publish and spread their content. Hence, this paper presents an approach incorporates semantic analysis and machine learning modules to measure and predict users' trustworthiness in numerous domains in different time periods. The evaluation of the conducted experiment validates the applicability of the incorporated machine learning techniques to predict highly trustworthy domain-based users.

The Mixed-Membership Stochastic Blockmodel~(MMSB) is proposed as one of the state-of-the-art Bayesian relational methods suitable for learning the complex hidden structure underlying the network data. However, the current formulation of MMSB suffers from the following two issues: (1), the prior information~(e.g. entities' community structural information) can not be well embedded in the modelling; (2), community evolution can not be well described in the literature. Therefore, we propose a non-parametric fragmentation coagulation based Mixed Membership Stochastic Blockmodel (fcMMSB). Our model performs entity-based clustering to capture the community information for entities and linkage-based clustering to derive the group information for links simultaneously. Besides, the proposed model infers the network structure and models community evolution, manifested by appearances and disappearances of communities, using the discrete fragmentation coagulation process (DFCP). By integrating the community structure with the group compatibility matrix we derive a generalized version of MMSB. An efficient Gibbs sampling scheme with Polya Gamma (PG) approach is implemented for posterior inference. We validate our model on synthetic and real world data.

The reconstruction from observations of high-dimensional chaotic dynamics such as geophysical flows is hampered by (i) the partial and noisy observations that can realistically be obtained, (ii) the need to learn from long time series of data, and (iii) the unstable nature of the dynamics. To achieve such inference from the observations over long time series, it has been suggested to combine data assimilation and machine learning in several ways. We show how to unify these approaches from a Bayesian perspective using expectation-maximization and coordinate descents. Implementations and approximations of these methods are also discussed. Finally, we numerically and successfully test the approach on two relevant low-order chaotic models with distinct identifiability.

Distributed statistical inference has recently attracted immense attention. Herein, we study the asymptotic efficiency of the maximum likelihood estimator (MLE), the one-step MLE, and the aggregated estimating equation estimator for generalized linear models with a diverging number of covariates. Then a novel method is proposed to obtain an asymptotically efficient estimator for large-scale distributed data by two rounds of communication between local machines and the central server. The assumption on the number of machines in this paper is more relaxed and thus practical for real-world applications. Simulations and a case study demonstrate the satisfactory finite-sample performance of the proposed estimators. Keywords: Generalized linear models, Large-scale distributed data, Asymptotic efficiency, One-step MLE, Diverging p MSC: 62J12 1 . Introduction In modern times, large-scale data sets have become increasingly common, and they are often stored across multiple machines. Since communication cost between machines is considerably higher than the cost of conducting statistical analysis on a single machine (Jaggi et al., 2014; Smith et al., 2018), it is inefficient to calculate a global estimator by the transmission of the local data to a central machine. Further, the application of the traditional iterative algorithms in a distributed system, such as the Fisher-scoring algorithm for maximum likelihood estimator (MLE) in generalized linear models (GLMs), cannot avoid multiple rounds of communication that incurs exorbitant costs. Therefore, communication-efficient distributed algorithms must be developed to accommodate the new features of modern data sets.

After long arguments between positivism and falsificationism, the verification of universal hypotheses was replaced with the confirmation of uncertain major premises. Unfortunately, Hemple discovered the Raven Paradox (RP). Then, Carnap used the logical probability increment as the confirmation measure. So far, many confirmation measures have been proposed. Measure F among them proposed by Kemeny and Oppenheim possesses symmetries and asymmetries proposed by Elles and Fitelson, monotonicity proposed by Greco et al., and normalizing property suggested by many researchers. Based on the semantic information theory, a measure b* similar to F is derived from the medical test. Like the likelihood ratio, b* and F can only indicate the quality of channels or the testing means instead of the quality of probability predictions. And, it is still not easy to use b*, F, or another measure to clarify the RP. For this reason, measure c* similar to the correct rate is derived. The c* has the simple form: (a-c)/max(a, c); it supports the Nicod Criterion and undermines the Equivalence Condition, and hence, can be used to eliminate the RP. Some examples are provided to show why it is difficult to use one of popular confirmation measures to eliminate the RP. Measure F, b*, and c* indicate that fewer counterexamples' existence is more essential than more positive examples' existence, and hence, are compatible with Popper's falsification thought.

Feature selection is important for modeling high-dimensional data, where the number of variables can be much larger than the sample size. In this paper, we develop a support detection and root finding procedure to learn the high dimensional sparse generalized linear models and denote this method by GSDAR. Based on the KKT condition for $\ell_0$-penalized maximum likelihood estimations, GSDAR generates a sequence of estimators iteratively. Under some restricted invertibility conditions on the maximum likelihood function and sparsity assumption on the target coefficients, the errors of the proposed estimate decays exponentially to the optimal order. Moreover, the oracle estimator can be recovered if the target signal is stronger than the detectable level. We conduct simulations and real data analysis to illustrate the advantages of our proposed method over several existing methods, including Lasso and MCP.

Since data collection and analysis are difficult, time consuming and costly, we are always looking for a way to optimum use of data to achieve the correct decision that can be referred to diagnose and experiment of diseases in healthcare organizations [3]. In addition, common method such as angiography [5,6] in experimenting and diagnosing diseases is costly and have adverse effects for patients as healthcare resear chers are trying to utilize methods that avoid the high cost as well as the adverse effects of previous methods, which can be performed by using computer - aided disease diagnose methods means machine learning. Whereas, da ta mining process by utilizing machine learning science and database management knowledge [1] has become a robust tool for data analysis and management of health industry data which ultimately leads to knowledge extraction. It should be noted that, with the progress of technology in t he healthcare especially, healthcare industry 4.0, human lifetime has become progressive and more comfortable [ 7 ] . In this new generation, with the development of new medical devices, equipment and tools, new knowledge can be gained in the field of disease diagnosis.

The examples are considered to be of the form ( x i,y i), where x i is the feature vector of the i-th example and y i { 1, 1} is its class label. Extension to the multiclass case is often handled by breaking down the problem into multiple binary ones, so our analysis and its main results can carry over to the multiclass case. We consider the online setting where examples are presented to the learner in M minibatches 2 of size b. On the n -th iteration the learner performs the following steps: 1. Receive new examples x i, x i minibatch n 2. Predict the label ˆ y i and/or the probability estimate ˆ p(y i 1 x i), i minibatch n 3. Get true labels y i f ( x i), x i minibatch n, where f is the labelling function 4. Update learner parameters accordingly The steps above are intentionally left general enough to describe all learning components encountered in the paper. Our goal is to study the quality of the probability estimates generated by online boosting ensembles and strategies for improving it. Online boosting ensembles consist of multiple base learners, themselves also trained in an online fashion and -as we will seethe techniques used for improving the probability estimates (both the calibrator and the reward models of the bandits) are also learners trained in an online fashion. All follow the same general approach defined above: they maintain a model with a fixed number of parameters (i.e.