Scientific Discovery
Controlling false discoveries in large-scale experimentation: Challenges and solutions
"Scientific research has changed the world. Now it needs to change itself. There has been a growing concern about the validity of scientific findings. A multitude of journals, papers and reports have recognized the ever smaller number of replicable scientific studies. In 2016, one of the giants of scientific publishing, Nature, surveyed about 1,500 researchers across many different disciplines, asking for their stand on the status of reproducibility in their area of research. One of the many takeaways to the worrisome results of this survey is the following: 90% of the respondents agreed that there is a reproducibility crisis, and the overall top answer to boosting reproducibility was "better understanding of statistics". Indeed, many factors contributing to the explosion of irreproducible research stem from the neglect of the fact that statistics is no longer as static as it was in the first half of the 20th century, when statistical hypothesis testing came into prominence as a ...
Can we trust scientific discoveries made using machine learning?
Allen, associate professor of statistics, computer science and electrical and computer engineering at Rice and of pediatrics-neurology at Baylor College of Medicine, will address the topic in both a press briefing and a general session today at the 2019 Annual Meeting of the American Association for the Advancement of Science (AAAS). "The question is, 'Can we really trust the discoveries that are currently being made using machine-learning techniques applied to large data sets?'" "The answer in many situations is probably, 'Not without checking,' but work is underway on next-generation machine-learning systems that will assess the uncertainty and reproducibility of their predictions." Machine learning (ML) is a branch of statistics and computer science concerned with building computational systems that learn from data rather than following explicit instructions. Allen said much attention in the ML field has focused on developing predictive models that allow ML to make predictions about future data based on its understanding of data it has studied. "A lot of these techniques are designed to always make a prediction," she said.
Can we trust scientific discoveries made using machine learning?
Rice University statistician Genevera Allen says scientists must keep questioning the accuracy and reproducibility of scientific discoveries made by machine-learning techniques until researchers develop new computational systems that can critique themselves. Allen, associate professor of statistics, computer science and electrical and computer engineering at Rice and of pediatrics-neurology at Baylor College of Medicine, will address the topic in both a press briefing and a general session today at the 2019 Annual Meeting of the American Association for the Advancement of Science (AAAS). "The question is, 'Can we really trust the discoveries that are currently being made using machine-learning techniques applied to large data sets?'" "The answer in many situations is probably, 'Not without checking,' but work is underway on next-generation machine-learning systems that will assess the uncertainty and reproducibility of their predictions." Machine learning (ML) is a branch of statistics and computer science concerned with building computational systems that learn from data rather than following explicit instructions. Allen said much attention in the ML field has focused on developing predictive models that allow ML to make predictions about future data based on its understanding of data it has studied.
Readings in Medical Artificial Intelligence: The First Decade
A survey of early work exploring how AI can be used in medicine, with somewhat more technical expositions than in the complementary volume Artificial Intelligence in Medicine."Each chapter is preceded by a brief introduction that outlines our view of its contribution to the field, the reason it was selected for inclusion in this volume, an overview of its content, and a discussion of how the work evolved after the article appeared and how it relates to other chapters in the book.
Readings in Medical Artificial Intelligence
JANICE S. AIKINS Dr. Aikins received her Ph.D. in computer science from Stanford University in 1980. She is currently a research computer scientist at IBM's Palo Alto Scientific Center. She specializes in designing systems with an emphasis on the explicit representation of control knowledge in expert systems. ROBERT L. BLUM Dr. Blum received his M.D. from the University of California Medical School at San Francisco in 1973. From 1973 to 1976 he did an internship and residency in the Department of Internal Medicine at the Kaiser Foundation Hospital in Oakland, California, where he was chief resident in 1976.
Retail as we know it has moved into a new paradigm TechNative
It offered'queue-less shopping' โ a self-service concept that allowed consumers to shop the store by pushing a metal trolley around the aisles rather than waiting in line at a counter to be served. Today, retailers are becoming increasingly reliant on customer experience innovations such as this to ensure their continuity, as the industry is entering the most transformational period of its experience in response to the current crisis hitting the UK high street. Already a disruptor with its convenience-focused online retail service, Amazon redoubled its efforts to disrupt brick-and-mortar retail outlets by launching its own physical store, Amazon Go, in 2016. By and large, Amazon Go resembled any other supermarket: products on shelves, arranged by aisles; an assortment of baskets and trolleys for transporting goods; and a bright, fresh, welcoming atmosphere to attract customers. It's revolutionary move, however, was to use intelligent innovations in IoT technology to provide the most convenient shopping experience yet.
The Everlasting Database: Statistical Validity at a Fair Price
Woodworth, Blake E., Feldman, Vitaly, Rosset, Saharon, Srebro, Nati
The problem of handling adaptivity in data analysis, intentional or not, permeates a variety of fields, including test-set overfitting in ML challenges and the accumulation of invalid scientific discoveries. We propose a mechanism for answering an arbitrarily long sequence of potentially adaptive statistical queries, by charging a price for each query and using the proceeds to collect additional samples. Crucially, we guarantee statistical validity without any assumptions on how the queries are generated. We also ensure with high probability that the cost for $M$ non-adaptive queries is $O(\log M)$, while the cost to a potentially adaptive user who makes $M$ queries that do not depend on any others is $O(\sqrt{M})$.
Robust Hypothesis Testing Using Wasserstein Uncertainty Sets
GAO, RUI, Xie, Liyan, Xie, Yao, Xu, Huan
We develop a novel computationally efficient and general framework for robust hypothesis testing. The new framework features a new way to construct uncertainty sets under the null and the alternative distributions, which are sets centered around the empirical distribution defined via Wasserstein metric, thus our approach is data-driven and free of distributional assumptions. We develop a convex safe approximation of the minimax formulation and show that such approximation renders a nearly-optimal detector among the family of all possible tests. By exploiting the structure of the least favorable distribution, we also develop a tractable reformulation of such approximation, with complexity independent of the dimension of observation space and can be nearly sample-size-independent in general. Real-data example using human activity data demonstrated the excellent performance of the new robust detector.
Robust Hypothesis Testing Using Wasserstein Uncertainty Sets
GAO, RUI, Xie, Liyan, Xie, Yao, Xu, Huan
We develop a novel computationally efficient and general framework for robust hypothesis testing. The new framework features a new way to construct uncertainty sets under the null and the alternative distributions, which are sets centered around the empirical distribution defined via Wasserstein metric, thus our approach is data-driven and free of distributional assumptions. We develop a convex safe approximation of the minimax formulation and show that such approximation renders a nearly-optimal detector among the family of all possible tests. By exploiting the structure of the least favorable distribution, we also develop a tractable reformulation of such approximation, with complexity independent of the dimension of observation space and can be nearly sample-size-independent in general. Real-data example using human activity data demonstrated the excellent performance of the new robust detector.
The Everlasting Database: Statistical Validity at a Fair Price
Woodworth, Blake E., Feldman, Vitaly, Rosset, Saharon, Srebro, Nati
The problem of handling adaptivity in data analysis, intentional or not, permeates a variety of fields, including test-set overfitting in ML challenges and the accumulation of invalid scientific discoveries. We propose a mechanism for answering an arbitrarily long sequence of potentially adaptive statistical queries, by charging a price for each query and using the proceeds to collect additional samples. Crucially, we guarantee statistical validity without any assumptions on how the queries are generated. We also ensure with high probability that the cost for $M$ non-adaptive queries is $O(\log M)$, while the cost to a potentially adaptive user who makes $M$ queries that do not depend on any others is $O(\sqrt{M})$.