Goto

Collaborating Authors

Neural Automated Essay Scoring and Coherence Modeling for Adversarially Crafted Input

arXiv.org Artificial Intelligence

We demonstrate that current state-of-the-art approaches to Automated Essay Scoring (AES) are not well-suited to capturing adversarially crafted input of grammatical but incoherent sequences of sentences. We develop a neural model of local coherence that can effectively learn connectedness features between sentences, and propose a framework for integrating and jointly training the local coherence model with a state-of-the-art AES model. We evaluate our approach against a number of baselines and experimentally demonstrate its effectiveness on both the AES task and the task of flagging adversarial input, further contributing to the development of an approach that strengthens the validity of neural essay scoring models.


Sentence Ordering and Coherence Modeling using Recurrent Neural Networks

arXiv.org Artificial Intelligence

Modeling the structure of coherent texts is a key NLP problem. The task of coherently organizing a given set of sentences has been commonly used to build and evaluate models that understand such structure. We propose an end-to-end unsupervised deep learning approach based on the set-to-sequence framework to address this problem. Our model strongly outperforms prior methods in the order discrimination task and a novel task of ordering abstracts from scientific articles. Furthermore, our work shows that useful text representations can be obtained by learning to order sentences. Visualizing the learned sentence representations shows that the model captures high-level logical structure in paragraphs. Our representations perform comparably to state-of-the-art pre-training methods on sentence similarity and paraphrase detection tasks.


Sentence Ordering and Coherence Modeling using Recurrent Neural Networks

AAAI Conferences

Modeling the structure of coherent texts is a key NLP problem. The task of coherently organizing a given set of sentences has been commonly used to build and evaluate models that understand such structure. We propose an end-to-end unsupervised deep learning approach based on the set-to-sequence framework to address this problem. Our model strongly outperforms prior methods in the order discrimination task and a novel task of ordering abstracts from scientific articles. Furthermore, our work shows that useful text representations can be obtained by learning to order sentences. Visualizing the learned sentence representations shows that the model captures high-level logical structure in paragraphs. Our representations perform comparably to state-of-the-art pre-training methods on sentence similarity and paraphrase detection tasks.


Ranking Summaries for Informativeness and Coherence without Reference Summaries

AAAI Conferences

There are numerous applications of automatic summarization systems currently and evaluating the quality of the summary is an important task. Current summary evaluation methods are limited in their scope since they rely on a reference summary, i.e., a human written summary. In this paper, we present a new summary evaluation technique without the use of reference summaries. The framework consists of two sequential steps: feature extraction and rank learning and generation. The former extracts effective features reflecting generic aspects, coherence, topical relevance, and informativeness of summaries and the latter uses features to train a learning model that provides the capability of generating a pair wise ranking for input summaries automatically. Our proposed framework is evaluated on the DUC multi-document summarization dataset and results indicate that this is a promising direction for automatic evaluation of the summaries without the use of a reference summary.


Automated Assessment of Paragraph Quality: Introduction, Body, and Conclusion Paragraphs

AAAI Conferences

Natural language processing and statistical methods were used to identify linguistic features associated with the quality of student-generated paragraphs. Linguistic features were assessed using Coh-Metrix. The resulting computational models demonstrated small to medium effect sizes for predicting paragraph quality: introduction quality r2 = .25, body quality r2 = .10, and conclusion quality r2 = .11. Although the variance explained was somewhat low, the linguistic features identified were consistent with the rhetorical goals of paragraph types. Avenues for bolstering this approach by considering individual writing styles and techniques are considered.