Reviewing contracts is a time-consuming procedure that incurs large expenses to companies and social inequality to those who cannot afford it. In this work, we propose "document-level natural language inference (NLI) for contracts", a novel, real-world application of NLI that addresses such problems. In this task, a system is given a set of hypotheses (such as "Some obligations of Agreement may survive termination.") and a contract, and it is asked to classify whether each hypothesis is "entailed by", "contradicting to" or "not mentioned by" (neutral to) the contract as well as identifying "evidence" for the decision as spans in the contract. We annotated and release the largest corpus to date consisting of 607 annotated contracts. We then show that existing models fail badly on our task and introduce a strong baseline, which (1) models evidence identification as multi-label classification over spans instead of trying to predict start and end tokens, and (2) employs more sophisticated context segmentation for dealing with long documents. We also show that linguistic characteristics of contracts, such as negations by exceptions, are contributing to the difficulty of this task and that there is much room for improvement.

The above quotation becomes the basis of what is present in this article, studying natural language processing in individual personality. Personality is defined as the characteristic set of behaviours, cognitions, and emotional patterns [1] as well as thinking patterns [2], and its external appearance can be seen in writing, speech, decision and other aspects of the social and personal lives of people. Language is the most prominent and the most available aspects of individuals' personality. Meanwhile, written text is one of the most utilized appearance of language. Developing the Internet based infrastructure such as social media, e-mails, and different texting contexts, have made the language appearance of people more available. Consequently, considering the increasing of internet based communications, it would be so exciting to became aware of individuals' personality, inspite of their absence. Therefore, the involvement of computers in determining the personality of people seems necessary and turned into a study field in computer science. Automatic Personality Prediction (or Perception) (APP) is the automatic prediction of the personality of individuals and usually done by computers.

This PhD thesis contains several contributions to the field of statistical causal modeling. Statistical causal models are statistical models embedded with causal assumptions that allow for the inference and reasoning about the behavior of stochastic systems affected by external manipulation (interventions). This thesis contributes to the research areas concerning the estimation of causal effects, causal structure learning, and distributionally robust (out-of-distribution generalizing) prediction methods. We present novel and consistent linear and non-linear causal effects estimators in instrumental variable settings that employ data-dependent mean squared prediction error regularization. Our proposed estimators show, in certain settings, mean squared error improvements compared to both canonical and state-of-the-art estimators. We show that recent research on distributionally robust prediction methods has connections to well-studied estimators from econometrics. This connection leads us to prove that general K-class estimators possess distributional robustness properties. We, furthermore, propose a general framework for distributional robustness with respect to intervention-induced distributions. In this framework, we derive sufficient conditions for the identifiability of distributionally robust prediction methods and present impossibility results that show the necessity of several of these conditions. We present a new structure learning method applicable in additive noise models with directed trees as causal graphs. We prove consistency in a vanishing identifiability setup and provide a method for testing substructure hypotheses with asymptotic family-wise error control that remains valid post-selection. Finally, we present heuristic ideas for learning summary graphs of nonlinear time-series models.

Imagine if you could control a robot or play a video game using your mind alone. It sounds like sci-fi, but this is exactly what brain-machine interfaces (BMIs) are already being used for. With applications from entertainment to medicine, BMIs are set to change the world of technology as we know it. But what exactly are they? And how do they work?

A meaningful understanding of clinical protocols and patient pathways helps improve healthcare outcomes. Electronic health records (EHR) reflect real-world treatment behaviours that are used to enhance healthcare management but present challenges; protocols and pathways are often loosely defined and with elements frequently not recorded in EHRs, complicating the enhancement. To solve this challenge, healthcare objectives associated with healthcare management activities can be indirectly observed in EHRs as latent topics. Topic models, such as Latent Dirichlet Allocation (LDA), are used to identify latent patterns in EHR data. However, they do not examine the ordered nature of EHR sequences, nor do they appraise individual events in isolation. Our novel approach, the Categorical Sequence Encoder (CaSE) addresses these shortcomings. The sequential nature of EHRs is captured by CaSE's event-level representations, revealing latent healthcare objectives. In synthetic EHR sequences, CaSE outperforms LDA by up to 37% at identifying healthcare objectives. In the real-world MIMIC-III dataset, CaSE identifies meaningful representations that could critically enhance protocol and pathway development.

Researchers from the Curtin Institute for Computation (CIC) will use the latest in data science to develop an automated fish detection and counting solution that offers exciting economic and ecological benefits. The CIC is part of a consortium that has been awarded $1 million in Federal funding to continue developing the AFID (Automated Fish Identification) system, which uses machine learning and Artificial Intelligence (AI) to automatically gather information about fish, including species and size. Project lead and CIC Lead Data Scientist Dr Daniel Marrable said the technology aimed to accurately, efficiently and more cost-effectively gather data in order to gauge marine and coastal ecosystem health, which would benefit Australia's multi-billion dollar fisheries and aquaculture industries. "AFID operates via a remote underwater video station and runs machine learning methods over video footage to count, classify and calculate the length of all visible fish," Dr Marrable said. "Fish biodiversity and biomass are the best non-invasive indicators of marine and coastal ecosystem health, however the current methods of measuring these are manual and very labour intensive. "Working closely with the Australian Institute of Marine Science (AIMS) and Curtin's own Fish Ecology Lab, by using machine learning and AI we can speed up the process of data collection and analysis, which will allow policy decisions that affect fish stocks and quotas, environmental impact assessment and ecological protection to be better informed." CIC Director Professor Melanie Johnston-Hollitt said AFID will use data science to help reduce the cost and manual labour required to monitor Australia's sensitive marine ecosystem. "The value of the project for fish ecology and the $2.7 billion fisheries industry highlights the important real-world, industry-aligned outcomes of the work being done in the area of data science at Curtin University," Professor Johnston-Hollitt said. "The CIC has been working in this domain for some time now, with Dr Marrable having done the vast majority of the technical work devising a system to identify, count, and measure fish from underwater imagery.

We release EDGAR-CORPUS, a novel corpus comprising annual reports from all the publicly traded companies in the US spanning a period of more than 25 years. To the best of our knowledge, EDGAR-CORPUS is the largest financial NLP corpus available to date. All the reports are downloaded, split into their corresponding items (sections), and provided in a clean, easy-to-use JSON format. We use EDGAR-CORPUS to train and release EDGAR-W2V, which are WORD2VEC embeddings for the financial domain. We employ these embeddings in a battery of financial NLP tasks and showcase their superiority over generic GloVe embeddings and other existing financial word embeddings. We also open-source EDGAR-CRAWLER, a toolkit that facilitates downloading and extracting future annual reports.

A central goal in deep learning is to learn compact representations of features at every layer of a neural network, which is useful for both unsupervised representation learning and structured network pruning. While there is a growing body of work in structured pruning, current state-of-the-art methods suffer from two key limitations: (i) instability during training, and (ii) need for an additional step of fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve state-of-the-art performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training process, while gradually masking out other neurons. We show that our proposed DAM approach has remarkably good performance over various applications, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective of DAM is directly related to minimizing the L0 norm of the masking layer.

We use a dataset of U.S. first names with labels based on predominant gender and racial group to examine the effect of training corpus frequency on tokenization, contextualization, similarity to initial representation, and bias in BERT, GPT-2, T5, and XLNet. We show that predominantly female and non-white names are less frequent in the training corpora of these four language models. We find that infrequent names are more self-similar across contexts, with Spearman's r between frequency and self-similarity as low as -.763. Infrequent names are also less similar to initial representation, with Spearman's r between frequency and linear centered kernel alignment (CKA) similarity to initial representation as high as .702. Moreover, we find Spearman's r between racial bias and name frequency in BERT of .492, indicating that lower-frequency minority group names are more associated with unpleasantness. Representations of infrequent names undergo more processing, but are more self-similar, indicating that models rely on less context-informed representations of uncommon and minority names which are overfit to a lower number of observed contexts.

Pre-trained models learn contextualized word representations on large-scale text corpus through a self-supervised learning method, which has achieved promising performance after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. Pre-trained models with knowledge injection, which we call knowledge enhanced pre-trained models (KEPTMs), possess deep understanding and logical reasoning and introduce interpretability to some extent. In this survey, we provide a comprehensive overview of KEPTMs for natural language processing. We first introduce the progress of pre-trained models and knowledge representation learning. Then we systematically categorize existing KEPTMs from three different perspectives. Finally, we outline some potential directions of KEPTMs for future research.