This blog post summarizes our paper Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots, which was published in the proceedings of UIST 2021. Machines that understand and operate user interfaces (UIs) on behalf of users could offer many benefits. For example, a screen reader (e.g., VoiceOver and TalkBack) could facilitate access to UIs for blind and visually impaired users, and task automation agents (e.g., Siri Shortcuts and IFTTT) could allow users to automate repetitive or complex tasks with their devices more efficiently. These benefits are gated on how well these systems can understand an underlying app's UI by reasoning about 1) the functionality present, 2) how its different components work together, and 3) how it can be operated to accomplish some goal. Many rely on the availability of UI metadata (e.g., the view hierarchy and the accessibility hierarchy), which provide some information about what elements are present and their properties.

A frequency count of all the names in a document; think of a dictionary containing the frequency of all words in a document. Often use for autocompleting and resolving word ambiguity (e.g. is watch in the document a verb or a noun?) "Also called shallow parsing, chunking aims at labeling segments of a sentence with syntactic constituents such as noun or verb phrases (NP or VP). Each word is assigned only one unique tag, often encoded as a begin-chunk (e.g., B-NP) or inside-chunk tag (e.g., I-NP)" (Collobert et al.). An entity is a thing that belongs to a noun family (e.g. the car, the chef, the plant). A named entity is an entity that has a proper name (e.g.

Mississippi mother Marie Sanderson and former New Jersey school nurse Erin Pein joined'Fox & Friends' to discuss the surfaced emails. A Virginia teachers union has faced ridicule online after it sent a letter riddled with grammatical and typographical errors. The union president explained that the union mistakenly published a draft, rather than the completed letter. "On behalf of the members of the Arlington Education Association, this dire expression lends great concerns for Arlington Public Schools (APS) return plan for January 3rd, 2022," begins the letter, which Arlington Education Association (AEA) President Ingrid Grant sent to Arlington Public Schools Superintendent Francisco Duran on Dec. 30, 2021. The letter raised concerns about the omicron variant of the coronavirus behind the COVID-19 pandemic, a more contagious variant that has spread rapidly throughout the U.S. Ellen Gallery, a homeschooling mother of three who lives in the area, claimed that she and her kids had gone through the letter, correcting mistakes.

Many tasks in Natural Language Processing (NLP) and Computer Vision (CV) offer evidence that humans disagree, from objective tasks such as part-of-speech tagging to more subjective tasks such as classifying an image or deciding whether a proposition follows from certain premises. While most learning in artificial intelligence (AI) still relies on the assumption that a single (gold) interpretation exists for each item, a growing body of research aims to develop learning methods that do not rely on this assumption. In this survey, we review the evidence for disagreements on NLP and CV tasks, focusing on tasks for which substantial datasets containing this information have been created. We discuss the most popular approaches to training models from datasets containing multiple judgments potentially in disagreement. We systematically compare these different approaches by training them with each of the available datasets, considering several ways to evaluate the resulting models. Finally, we discuss the results in depth, focusing on four key research questions, and assess how the type of evaluation and the characteristics of a dataset determine the answers to these questions. Our results suggest, first of all, that even if we abandon the assumption of a gold standard, it is still essential to reach a consensus on how to evaluate models. This is because the relative performance of the various training methods is critically affected by the chosen form of evaluation. Secondly, we observed a strong dataset effect. With substantial datasets, providing many judgments by high-quality coders for each item, training directly with soft labels achieved better results than training from aggregated or even gold labels. This result holds for both hard and soft evaluation. But when the above conditions do not hold, leveraging both gold and soft labels generally achieved the best results in the hard evaluation. All datasets and models employed in this paper are freely available as supplementary materials.

For many years, research in Artificial Intelligence (AI) has directed efforts towards automating processes to perform specific academic, industrial or economic tasks for society. However, the investigation and development of procedures for the automation of human artistic and creative processes has not had as much attention due to the complexities involved in these activities. Procedures developed for these purposes involve mathematical-computational methods designed to process and learn from a large quantity of digital data, so as to detect patterns in order to simulate the creative process (CP), as explained by Boden in [3]. In this paper, we introduce a model for the generation of rhymes with literary components. Our proposal is based on findings detailed in [11], where Automatic Text Generation (ATG) techniques are combined with neural network (NN) based models, such as the Word2vec algorithm [9], for the generation of literary texts.

Many Natural Language Processing (NLP) techniques exist for detecting "fake news". Multi-phase algorithms with Determined Decision Trees, Gradient Enlargement, and others have been used by various researchers and organizations with varying results. One study from researchers at Rensselaer Polytechnic Institute reported 83% accuracy in predicting whether a news article is from a reliable or unreliable source [1], while Facebook's 2019 attempt at developing an algorithm failed miserably, with some users experiencing a "maelstrom" of fake news [2]. A new study, published in the November 2021 issue of the Journal of Emerging Technologies and Innovative Research [3] performs an analysis of a wide range of AI models for efficacy, finding that models generally perform poorly, ranging from 60% to 77% accuracy. Separating fake news from real news is a challenge even for the most sophisticated AI. Simple content-related programs and shallow marking of the speech part (POS) fail to consider contextual information and are unable to accurately classify news stories as fact or fake unless combined with more sophisticated algorithms.

Data augmentation has been an important ingredient for boosting performances of learned models. Prior data augmentation methods for few-shot text classification have led to great performance boosts. However, they have not been designed to capture the intricate compositional structure of natural language. As a result, they fail to generate samples with plausible and diverse sentence structures. Motivated by this, we present the data Augmentation using Lexicalized Probabilistic context-free grammars (ALP) that generates augmented samples with diverse syntactic structures with plausible grammar. The lexicalized PCFG parse trees consider both the constituents and dependencies to produce a syntactic frame that maximizes a variety of word choices in a syntactically preservable manner without specific domain experts. Experiments on few-shot text classification tasks demonstrate that ALP enhances many state-of-the-art classification methods. As a second contribution, we delve into the train-val splitting methodologies when a data augmentation method comes into play. We argue empirically that the traditional splitting of training and validation sets is sub-optimal compared to our novel augmentation-based splitting strategies that further expand the training split with the same number of labeled data. Taken together, our contributions on the data augmentation strategies yield a strong training recipe for few-shot text classification tasks.

To this end, we propose a new KG embedding model, named A knowledge graph (KG) is a network of entity nodes and WGE, to leverage GNNs to capture entity-focused graph structure relationship edges, which can be represented as a collection and relation-focused graph structure for KG completion. of triples in the form of (h, r, t), wherein each triple (h, r, In particular, WGE transforms a given KG into two views. The t) represents a relation r between a head entity h and a tail first view--a single undirected entity-focused graph--only entity t. Here, entities are real-world things or objects such includes entities as nodes to provide the entity neighborhood as music tracks, movies persons, organizations, places and the information. The second view--a single undirected relationfocused like, while each relation type determines a certain relationship graph--considers both entities and relations as nodes, between entities. KGs are used in a number of commercial applications, constructed from constraints (subjective relation, predicate e.g. in such search engines as Google, Microsoft's entity, objective relation), to attain the potential dependence Bing and Facebook's Graph search. They also are useful between two neighborhood relations. Then WGE introduces a resources for many natural language processing tasks such as new encoder module of adopting two vanilla GNNs directly co-reference resolution ([1], [2]), semantic parsing ([3], [4]) on these two graph views to better update entity and relation and question answering ([5], [6]). However, an issue is that embeddings, followed by the decoder module using a weighted KGs are often incomplete, i.e., missing a lot of valid triples score function. In summary, our contributions are as follows: [7].

Conversational text-to-SQL aims at converting multi-turn natural language queries into their corresponding SQL representations. One of the most intractable problem of conversational text-to-SQL is modeling the semantics of multi-turn queries and gathering proper information required for the current query. This paper shows that explicit modeling the semantic changes by adding each turn and the summarization of the whole context can bring better performance on converting conversational queries into SQLs. In particular, we propose two conversational modeling tasks in both turn grain and conversation grain. These two tasks simply work as auxiliary training tasks to help with multi-turn conversational semantic parsing. We conducted empirical studies and achieve new state-of-the-art results on large-scale open-domain conversational text-to-SQL dataset. The results demonstrate that the proposed mechanism significantly improves the performance of multi-turn semantic parsing.

There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. In this paper, we study whether linguistic knowledge is a necessary condition for the good performance of modern language models, which we call the rediscovery hypothesis. In the first place, we show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures. This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objectives with linguistic information. This framework also provides a metric to measure the impact of linguistic information on the word prediction task. We reinforce our analytical results with various experiments, both on synthetic and on real NLP tasks in English.