Personal
Quality Issues in Machine Learning Software Systems
Côté, Pierre-Olivier, Nikanjam, Amin, Bouchoucha, Rached, Basta, Ilan, Abidi, Mouna, Khomh, Foutse
Context: An increasing demand is observed in various domains to employ Machine Learning (ML) for solving complex problems. ML models are implemented as software components and deployed in Machine Learning Software Systems (MLSSs). Problem: There is a strong need for ensuring the serving quality of MLSSs. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threats to human life. The quality assurance of MLSSs is considered a challenging task and currently is a hot research topic. Objective: This paper aims to investigate the characteristics of real quality issues in MLSSs from the viewpoint of practitioners. This empirical study aims to identify a catalog of quality issues in MLSSs. Method: We conduct a set of interviews with practitioners/experts, to gather insights about their experience and practices when dealing with quality issues. We validate the identified quality issues via a survey with ML practitioners. Results: Based on the content of 37 interviews, we identified 18 recurring quality issues and 24 strategies to mitigate them. For each identified issue, we describe the causes and consequences according to the practitioners' experience. Conclusion: We believe the catalog of issues developed in this study will allow the community to develop efficient quality assurance tools for ML models and MLSSs. A replication package of our study is available on our public GitHub repository.
Practitioner Motives to Select Hyperparameter Optimization Methods
Hasebrook, Niklas, Morsbach, Felix, Kannengießer, Niclas, Zöller, Marc, Franke, Jörg, Lindauer, Marius, Hutter, Frank, Sunyaev, Ali
Advanced programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sample efficiency in reproducibly finding optimal hyperparameter values of machine learning (ML) models. Yet, ML practitioners often apply less sample-efficient HPO methods, such as grid search, which often results in under-optimized ML models. As a reason for this behavior, we suspect practitioners choose HPO methods based on individual motives, consisting of contextual factors and individual goals. However, practitioners' motives still need to be clarified, hindering the evaluation of HPO methods for achieving specific goals and the user-centered development of HPO tools. To understand practitioners' motives for using specific HPO methods, we used a mixed-methods approach involving 20 semi-structured interviews and a survey study with 71 ML experts to gather evidence of the external validity of the interview results. By presenting six main goals (e.g., improving model understanding) and 14 contextual factors affecting practitioners' selection of HPO methods (e.g., available computer resources), our study explains why practitioners use HPO methods that seem inappropriate at first glance. This study lays a foundation for designing user-centered and context-adaptive HPO tools and, thus, linking social and technical research on HPO.
Can a Chatbot Publish an "Original" Novel?
This story is part of Future Tense Fiction, a monthly series of short stories from Future Tense and Arizona State University's Center for Science and the Imagination about how technology and science will change our lives. THE COURT: Please be seated. Let's try to keep the temperature down in here. We don't need a repeat of yesterday. It'll just be Mr. Blatz and myself today. Sorry, it's hard to tell with … are you with us? ORWELL: Omni-dimensional Recursively Written Entity for Language Learning present and ready, Your Honor. THE COURT: You can just say ORWELL. Are we ready to proceed? LIU: Your Honor, we'd like to call the Defendant to the stand. Mr. Blatz will handle examination. THE COURT: We have the wiring sorted out? Please refrain from using the monitor on the Defendant's table until you're off the stand.
Read TIME's Interview With OpenAI CEO Sam Altman
For this week's TIME100 Most Influential Companies cover story about OpenAI and its CEO Sam Altman, TIME's former editor-in-chief Edward Felsenthal sat down with a number of company executives in early May, including two sessions with Altman, transcribed below. The conversations have been condensed and edited for clarity. Sam Altman: One thing I use it for every day is help with summarization. I can't really keep up on my inbox anymore, but I made a little thing to help it summarize for me and pull out important stuff from unknown senders, and that's very helpful. I used it to translate an article for someone I'm meeting next week, to prepare for that. This is sort of a funny thing, I used it to help me draft a tweet that I was having a hard time with. Not as much as it might have seemed from the outside.
AI-generated music won't win a Grammy anytime soon
It looks like Fake Drake won't be taking home a Grammy. Recording Academy CEO Harvey Mason Jr. said this week that although the organization will consider music with limited AI-generated voices or instrumentation for award recognition, it will only honor songs written and performed "mostly by a human." "At this point, we are going to allow AI music and content to be submitted, but the Grammys will only be allowed to go to human creators who have contributed creatively in the appropriate categories," Mason said in an interview with Grammy.com. "If there's an AI voice singing the song or AI instrumentation, we'll consider it. But in a songwriting-based category, it has to have been written mostly by a human. Same goes for performance categories – only a human performer can be considered for a Grammy. If AI did the songwriting or created the music, that's a different consideration. But the Grammy will go to human creators at this point."
OceanGate Titan submarine operated by video game controller, CEO says
A tourist submersible taking passengers down to the Titanic wreck site has gone missing with a search currently underway. Former Navy submariner Bryan Clark joins'Fox & Friends' to discuss. The Titan submarine OceanGate has been charging tourists around $250,000 each to ride in is operated by an inexpensive video game controller, its CEO revealed in a video interview last year. Stockton Rush, during a segment aired by "CBS Sunday Morning," said "we run the whole thing with this game controller" while holding up what appears to be a modified Logitech F710 wireless gamepad. The device first debuted in 2011, according to the gaming website Dexerto, and a refurbished version of it currently retails for $30 on Amazon.
CATS: A Pragmatic Chinese Answer-to-Sequence Dataset with Large Scale and High Quality
Li, Liang, Geng, Ruiying, Fang, Chengyang, Li, Bing, Ma, Can, Cao, Rongyu, Li, Binhua, Huang, Fei, Li, Yongbin
There are three problems existing in the popular data-to-text datasets. First, the large-scale datasets either contain noise or lack real application scenarios. Second, the datasets close to real applications are relatively small in size. Last, current datasets bias in the English language while leaving other languages underexplored. To alleviate these limitations, in this paper, we present CATS, a pragmatic Chinese answer-to-sequence dataset with large scale and high quality. The dataset aims to generate textual descriptions for the answer in the practical TableQA system. Further, to bridge the structural gap between the input SQL and table and establish better semantic alignments, we propose a Unified Graph Transformation approach to establish a joint encoding space for the two hybrid knowledge resources and convert this task to a graph-to-text problem. The experiment results demonstrate the effectiveness of our proposed method. Further analysis on CATS attests to both the high quality and challenges of the dataset.
Cases of EFL Secondary Students' Prompt Engineering Pathways to Complete a Writing Task with ChatGPT
Woo, David James, Guo, Kai, Susanto, Hengky
Although it has potential to support English as a foreign language (EFL) students' writing, to effectively collaborate with it, a student must learn to engineer prompts, that is, the skill of crafting appropriate instructions so that ChatGPT produces desired outputs. However, writing an appropriate prompt for ChatGPT is not straightforward for non-technical users who suffer a trial-and-error process. This paper examines the content of EFL students' ChatGPT prompts when completing a writing task and explores patterns in the quality and quantity of the prompts. The data come from iPad screen recordings of secondary school EFL students who used ChatGPT and other SOTA chatbots for the first time to complete the same writing task. The paper presents a case study of four distinct pathways that illustrate the trial-and-error process and show different combinations of prompt content and quantity. The cases contribute evidence for the need to provide prompt engineering education in the context of the EFL writing classroom, if students are to move beyond an individual trial-anderror process, learning a greater variety of prompt content and more sophisticated prompts to support their writing. Keywords: artificial intelligence; chatbots; prompt engineering; writing; case study; ChatGPT 1. Introduction ChatGPT's incredible popularity indicates many people's desire to transform their world of education, work and leisure through chatbots. Previously, chatbots followed a rule-based design with limited capabilities to respond accurately to user queries, especially with unfamiliar inputs.
On this day in history, June 18, 1983, astronaut Sally Ride becomes first American woman in space
Astronaut Sally Ride became the first American woman in space on this day in history, June 18, 1983. Born on May 26, 1951, in Los Angeles, Ride earned bachelor's degrees in English and physics from Stanford University in California before staying at Stanford and earning a PhD in physics in 1978. Shortly before earning her doctorate, Ride saw an ad for a newspaper that piqued her interest. NASA was recruiting for astronauts -- and, for the first time, the agency would include women in its astronaut class. "Over 8,000 men and women applied to the space program that year. Of the 35 individuals accepted, six were women, and I was one of them. This was in January 1978," said Ride in quotes listed on a tribute page on NASA's website.
Going public: the role of public participation approaches in commercial AI labs
Groves, Lara, Peppin, Aidan, Strait, Andrew, Brennan, Jenny
In recent years, discussions of responsible AI practices have seen growing support for "participatory AI" approaches, intended to involve members of the public in the design and development of AI systems. Prior research has identified a lack of standardised methods or approaches for how to use participatory approaches in the AI development process. At present, there is a dearth of evidence on attitudes to and approaches for participation in the sites driving major AI developments: commercial AI labs. Through 12 semi-structured interviews with industry practitioners and subject-matter experts, this paper explores how commercial AI labs understand participatory AI approaches and the obstacles they have faced implementing these practices in the development of AI systems and research. We find that while interviewees view participation as a normative project that helps achieve "societally beneficial" AI systems, practitioners face numerous barriers to embedding participatory approaches in their companies: participation is expensive and resource intensive, it is "atomised" within companies, there is concern about exploitation, there is no incentive to be transparent about its adoption, and it is complicated by a lack of clear context. These barriers result in a piecemeal approach to participation that confers no decision-making power to participants and has little ongoing impact for AI labs. This papers contribution is to provide novel empirical research on the implementation of public participation in commercial AI labs, and shed light on the current challenges of using participatory approaches in this context.