The goal of the research is to discover and summarize data from the emerging social media into information of interests. Specifically, leveraging user-contributed data from cross-domain social media, the idea is to perform multi-modal learning for a given photo, aiming to present people’s description or comments, geographical information, and events of interest, closely related to the photo. These information then can be used for various purposes, such as being a real-time guide for the tourists to improve the quality of tourism. As a result, this research investigates modern challenges of image annotation, image retrieval, and cross-media mining, followed by presenting promising ways to conquer the challenges.

In online social networks (OSNs), users are allowed to create and share content about themselves and others. When multiple entities start distributing content, information can reach unintended individuals and inference can reveal more information about the user. Existing applications do not focus on detecting privacy violations before they occur in the system. This thesis proposes an agent-based representation of a social network, where the agents manage users' privacy requirements and create privacy agreements with agents. The privacy context, such as the relations among users, various content types in the system, and so on are represented with a formal language. By reasoning with this formal language, an agent checks the current state of the system to resolve privacy violations before they occur. We argue that commonsense reasoning could be useful to solve some of privacy examples reported in the literature. We will develop new methods to automatically identify private information using commonsense reasoning, which has never been applied to privacy context. Moreover, agents may have conflicting privacy requirements. We will study how to use agreement technologies in privacy settings for agents to resolve conflicts automatically.

In many real-world domains, the presence of machines is becoming more ubiquitous to the point that they are usually more than simple automation tools. As part of the environment amongst human users, it is necessary for these computers and robots to be able to interact with them reasonably by either working independently around them or participating in a task, especially one with which a person needs help. This interactive procedure requires several steps: recognizing the user and environment from sensor data, interpreting the user’s activity and motives, determining a responsive behavior, performing the behavior, and then recognizing everything again to confirm the behavior choice and replan if necessary. At the moment, the research areas addressing these steps, activity recognition, plan recognition, intent recognition, and planning, have all been primarily studied independently. However, pipelining each independent process can be risky in real-time situations where there may be enough time to only run a few steps. This leads to a critical question: how do we perform everything under time constraints? In this thesis summary, I propose a framework that integrates these processes by taking advantage of features shared between them.

We present an algorithm (LsNet2Vec) that, given a large-scale network (millions of nodes), embeds the structural features of node into a lower and fixed dimensions of vector in the set of real numbers. We experiment and evaluate our proposed approach with twelve datasets collected from SNAP. Results show that our model performs comparably with state-of-the-art methods, such as Katz method and Random Walk Restart method, in various experiment settings.

We propose to predict the personalized emotion perceptions of images for each viewer. Different factors that may influence emotion perceptions, including visual content, social context, temporal evolution, and location influence are jointly investigated via the presented rolling multi-task hypergraph learning. For evaluation, we set up a large scale image emotion dataset from Flickr, named Image-Emotion-Social-Net, with over 1 million images and about 8,000 users. Experiments conducted on this dataset demonstrate the superiority of the proposed method, as compared to state-of-the-art.

For many researchers, one of the biggest issues is the lack of an efficient method to obtain latest academic progresses in related research fields. We notice that many researchers tend to share their research progresses or recommend scholarly information they have known on their microblogs. In order to exploit microblogging to benefit scientific research, we build a system called MicroScholar to automatically collecting and mining scholarly information from Chinese microblogs. In this paper, we briefly introduce the system framework and focus on the component of scholarly microblog categorization. Several kinds of features have been used in the component and experimental results demonstrate their usefulness.

The principle of counter-transitivity plays a vital role in argumentation. It states that an argument is strong when its attackers are weak, and is weak when its attackers are strong. In this work, we develop a formal theory about the argument ranking semantics based on this principle. Three approaches, quantity-based, quality-based and the unity of them, are defined to implement the principle. Then, we show an iterative refinement algorithm for capturing the ranking on arguments based on the recursive nature of the principle.

We propose Bayesian AutoEncoder (BAE) in order to construct a recognition system which uses feedback information. BAE constructs a generative model of input data as a Bayes Net. The network trained by BAE obtains its hidden variables as the features of given data. It can execute inference for each variable through belief propagation, using both feedforward and feedback information. We confirmed that BAE can construct small networks with one hidden layer and extract features as hidden variables from 3x3 and 5x5 pixel input data.

The restricted Boltzmann machine (RBM) has been used as building blocks for many successful deep learning models, e.g., deep belief networks (DBN) and deep Boltzmann machine (DBM) etc. The training of RBM can be extremely slow in pathological regions. The second order optimization methods, such as quasi-Newton methods, were proposed to deal with this problem. However, the non-convexity results in many obstructions for training RBM, including the infeasibility of applying second order optimization methods. In order to overcome this obstruction, we introduce an em-like iterative project quasi-Newton (IPQN) algorithm. Specifically, we iteratively perform the sampling procedure where it is not necessary to update parameters, and the sub-training procedure that is convex. In sub-training procedures, we apply quasi-Newton methods to deal with the pathological problem. We further show that Newton's method turns out to be a good approximation of the natural gradient (NG) method in RBM training. We evaluate IPQN in a series of density estimation experiments on the artificial dataset and the MNIST digit dataset. Experimental results indicate that IPQN achieves an improved convergent performance over the traditional CD method.

Human's cognition system prompts that context information provides potentially powerful clue while recognizing objects. However, for fine-grained image classification, the contribution of context may vary over different images, and sometimes the context even confuses the classification result. To alleviate this problem, in our work, we develop a novel approach, two-stream contextualized Convolutional Neural Network, which provides a simple but efficient context-content joint classification model under deep learning framework. The network merely requires the raw image and a coarse segmentation as input to extract both content and context features without need of human interaction. Moreover, our network adopts a weighted fusion scheme to combine the content and the context classifiers, while a subnetwork is introduced to adaptively determine the weight for each image. According to our experiments on public datasets, our approach achieves considerable high recognition accuracy without any tedious human's involvements, as compared with the state-of-the-art approaches.