Several pieces of work have uncovered performance disparities by conducting "disaggregated evaluations" of AI systems. We build on these efforts by focusing on the choices that must be made when designing a disaggregated evaluation, as well as some of the key considerations that underlie these design choices and the tradeoffs between these considerations. We argue that a deeper understanding of the choices, considerations, and tradeoffs involved in designing disaggregated evaluations will better enable researchers, practitioners, and the public to understand the ways in which AI systems may be underperforming for particular groups of people.

We combine deep learning and Conditional Probabilistic Context Free Grammars (CPCFG) to create an end-to-end system for extracting structured information from complex documents. For each class of documents, we create a CPCFG that describes the structure of the information to be extracted. Conditional probabilities are modeled by deep neural networks. We use this grammar to parse 2-D documents to directly produce structured records containing the extracted information. This system is trained end-to-end with (Document, Record) pairs. We apply this approach to extract information from scanned invoices achieving state-of-the-art results.

After more than a decade of intense focus on automated vehicles, we are still facing huge challenges for the vision of fully autonomous driving to become a reality. The same "disillusionment" is true in many other domains, in which autonomous Cyber-Physical Systems (CPS) could considerably help to overcome societal challenges and be highly beneficial to society and individuals. Taking the automotive domain, i.e. highly automated vehicles (HAV), as an example, this paper sets out to summarize the major challenges that are still to overcome for achieving safe, secure, reliable and trustworthy highly automated resp. autonomous CPS. We constrain ourselves to technical challenges, acknowledging the importance of (legal) regulations, certification, standardization, ethics, and societal acceptance, to name but a few, without delving deeper into them as this is beyond the scope of this paper. Four challenges have been identified as being the main obstacles to realizing HAV: Realization of continuous, post-deployment systems improvement, handling of uncertainties and incomplete information, verification of HAV with machine learning components, and prediction. Each of these challenges is described in detail, including sub-challenges and, where appropriate, possible approaches to overcome them. By working together in a common effort between industry and academy and focusing on these challenges, the authors hope to contribute to overcome the "disillusionment" for realizing HAV.

But the big data industry has significant inertia moving into 2021. In order to give our valued readers a pulse on important new trends leading into next year, we here at insideBIGDATA heard from all our friends across the vendor ecosystem to get their insights, reflections and predictions for what may be coming. We were very encouraged to hear such exciting perspectives. Even if only half actually come true, Big Data in the next year is destined to be quite an exciting ride. The "analytic divide" is going to get worse. Like the much-publicized "digital divide" we're also seeing the emergence of an "analytic divide." Many companies were driven to invest in analytics due to the pandemic, while others have been forced to cut anything they didn't view as critical to keep the lights on – and a proper investment in analytics was, for these organizations, analytics was on the chopping block. This means that the analytic divide will further widen in 2021, and this trend will continue for ...

An academic and a lawyer have teamed up to develop a robot lawyer, which, if successful, will make legal advice affordable to people from all backgrounds, while revolutionizing the legal sector. Robots could take on significant parts of a lawyer's work, reducing the costs and barriers to access to legal services for everyone, rather than just those who can afford the high costs. The project, at the University of Bradford, is initially working on a machine learning-based application to provide immigration-related legal advice, but if successful, it could be replicated across the legal sector. The project was devised by Yash Dubal, immigration lawyer and director at AY&J, and Dhaval Thakker, associate professor at the faculty of engineering and informatics at the University of Bradford. It will harness complex knowledge graph technology and deep learning algorithms to analyse case law and learn from it.

AI is a hot buzzword right now, but with buzz always comes a whole host of misconceptions about a technology's capabilities. There's considerable confusion about what artificial intelligence can do and widespread misinformation about how it works, particularly in the area of managing legal contracts and if AI will replace lawyers. Onit recently hosted a webinar to debunk these common myths. Nick Whitehouse, General Manager of Onit's AI Center of Excellence, and Jean Yang, Vice President of Onit's AI Center of Excellence, dispelled common misconceptions about everything from will AI replace lawyers to who can benefit from AI. The goal is to help legal professionals decipher marketing-speak to determine what's genuinely AI and what's just software.

The Korean Intellectual Property Office (KIPO) announced Patent Examination Guidelines for key technology areas related to the Fourth Industrial Revolution, including machine learning based artificial intelligence ("AI"), on January 18, 2021. In the Examination Guidelines for AI, KIPO outlines specific guidelines on description and novelty/inventiveness requirements for different categories of AI inventions (e.g., AI model training invention and AI application invention, as depicted below), in addition to eligibility requirements which correspond to that of computer-related inventions. In particular, KIPO's Examination Guidelines provide examples of various AI inventions with practical drafting tips on enablement (Article 42(3)(i) of Patent Act) and inventiveness requirements (Article 29(2)). Under Article 42(3)(i), the description of an invention shall be written clearly and fully so that a person with ordinary skill in the art (POSITA) to which the invention pertains can easily practice the claimed invention. For an AI invention, KIPO suggests that the description of the technical problem, solution, and specific technical configuration (e.g., training data, data preprocessing, trained model, and loss function, etc.) be included to enable a POSITA to practice the claimed invention, unless the technical configuration is well known in the art.

Say you're a job-seeker who's got a pretty good idea of what employers want to hear. Like many companies these days, your potential new workplace will give you a personality test as part of the hiring process. You plan to give answers that show you're enthusiastic, a hard worker and a real people person. Then they put you on camera while you take the test verbally, and you frown slightly during one of your answers, and their facial-analysis program decides you're "difficult." This is just one of many problems with the increasing use of artificial intelligence in hiring, contends the new documentary "Persona: The Dark Truth Behind Personality Tests," premiering Thursday on HBO Max. The film, from director Tim Travers Hawkins, begins with the origins of the Myers-Briggs Type Indicator personality test.

Traditional computer scientists and engineers are trained to develop solutions for specific needs, but aren't always trained to consider their broader implications. Each new technology generation, and particularly the rise of artificial intelligence, leads to new kinds of systems, new ways of creating tools, and new forms of data, for which norms, rules, and laws frequently have yet to catch up. The kinds of impact that such innovations have in the world has often not been apparent until many years later. As part of the efforts in Social and Ethical Responsibilities of Computing (SERC) within the MIT Stephen A. Schwarzman College of Computing, a new case studies series examines social, ethical, and policy challenges of present-day efforts in computing with the aim of facilitating the development of responsible "habits of mind and action" for those who create and deploy computing technologies. "Advances in computing have undeniably changed much of how we live and work. Understanding and incorporating broader social context is becoming ever more critical," says Daniel Huttenlocher, dean of the MIT Schwarzman College of Computing.

Artificial intelligence (AI) is considered one of the most revolutionary developments in human history, and the world has already witnessed its transformative capabilities. Not surprisingly, AI-based innovations are powering some of the most cutting-edge solutions we use in our daily lives. Today, AI empowers organizations, governments and communities to build a high-performing ecosystem to serve the entire world. Its profound impact on human lives is solving some of the most critical challenges faced by society. Here are a few innovations for social causes that I find most notable.