Collaborating Authors


IBM advanced Project Debater effort with Key Point Analysis


IBM has developed a natural language processing advance via its Project Debater effort called Key Point Analysis that aims to use artificial intelligence to sum up crowd-sourced arguments. The technology, led by IBM Research, is being showcased on Bloomberg TV's "That's Debatable" show. The show aired Friday and featured a debate on wealth distribution with US Secretary of Labor Robert Reich, former Greece finance minister Yanis Varoufakis, former US Treasury Secretary Larry Summers and Manhattan Institute's Allison Schrager. Noam Slonim, lead researcher for IBM's Project Debater effort, said the goal of Key Point Analysis is to "enable AI systems to manage the human language." "There's a significant opportunity for using national language processing," he said.

Learning Compact Visual Descriptors for Low Bit Rate Mobile Landmark Search

AI Magazine

Coming with the ever growing computational power of mobile devices, mobile visual search have undergone an evolution in techniques and applications. A significant trend is low bit rate visual search, where compact visual descriptors are extracted directly over a mobile and delivered as queries rather than raw images to reduce the query transmission latency. In this article, we introduce our work on low bit rate mobile landmark search, in which a compact yet discriminative landmark image descriptor is extracted by using location context such as GPS, crowd-sourced hotspot WLAN, and cell tower locations. The compactness originates from the bag-of-words image representation, with an offline learning from geotagged photos from online photo sharing websites including Flickr and Panoramio. The learning process involves segmenting the landmark photo collection by discrete geographical regions using Gaussian mixture model, and then boosting a ranking sensitive vocabulary within each region, with an "entropy" based descriptor compactness feedback to refine both phases iteratively.

NASA launches $5M competition to bring power to the moon


Nights on the moon can last up to 350 hours. That creates some big technical challenges as NASA's Artemis program gears up to send people back to the moon. In addition to issues like extreme temperature changes, one of the biggest difficulties presented by lunar night is the loss of solar power. For long-term habitation to be viable, NASA needs to find a sustainable solution. To that, it's launching a $5 million prize in its Watts on the Moon Challenge, which is being launched in collaboration with crowdsourcing platform HeroX.

NIST is crowdsourcing differential privacy techniques for public safety datasets


The National Institute of Standards and Technology (NIST) is launching the Differential Privacy Temporal Map Challenge. It's a set of contests, with cash prizes attached, that's intended to crowdsource new ways of handling personally identifiable information (PII) in public safety datasets. The problem is that although rich, detailed data is valuable for researchers and for building AI models -- in this case, in the areas of emergency planning and epidemiology -- it raises serious and potentially dangerous data privacy and rights issues. Even if datasets are kept under proverbial lock and key, malicious actors can, based on just a few data points, re-infer sensitive information about people. The solution is to de-identify the data such that it remains useful without compromising individuals' privacy.

Quiet Anthropocene, quiet Earth


Our planet vibrates incessantly, sometimes with notable but more often with imperceptible intensity. Conventional seismology attempts to decipher vibrational sources and path effects by studying seismograms—records of vibrations measured with seismometers. In doing so, scientists seek either to understand the tectonic processes that lead to strong ground motions and earthquake failure ([ 1 ][1]) or to probe otherwise inaccessible planetary interiors ([ 2 ][2]). Progress in these areas of research typically has relied on the rare and geographically irregular occurrence of large earthquakes. However, anthropogenic (human) activities at Earth's surface also generate seismic waves that instruments can detect over great distances. On page 1338 of this issue, Lecocq et al. ([ 3 ][3]) report on a quieting of anthropogenic vibrations since the start of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic. Seismology has benefited from a surge in seismic data volume, computational power, and corresponding methodological development. These advances have enabled seismologists to branch away from traditional source and subsurface characterization of the energy from earthquakes and human-made blasts. The expansion of seismic networks has allowed the observation of previously unseen natural processes as diverse as wildlife activity ([ 4 ][4]), bed load transport in rivers, glacier sliding ([ 5 ][5]), and surface-mass wasting ([ 6 ][6]). In particular, scientists use continuous, ambient seismic vibrations to probe volcanic activities ([ 7 ][7]) and groundwater resources ([ 8 ][8]), to track storms ([ 9 ][9]), and to decipher ice sheet processes ([ 10 ][10]). Human cultural noise carries seismic signatures mostly at frequencies above 1 Hz, whether the source is transient (entertainment; individual cars, trains, or planes), harmonic (wind turbines, machinery), or diffuse (railroads, highways) ([ 11 ][11], [ 12 ][12]) (see the figure). Overall, anthropogenic seismic noise levels have increased over the past few decades, and there is a clear positive correlation between this increase and gross domestic product ([ 13 ][13]). But when the SARS-CoV-2 pandemic began to ravage the planet, humans—and Earth—went quiet. Through a global analysis of seismic noise levels, Lecocq et al. found that most sites experienced a drastic reduction in noise levels in the 4- to 14-Hz frequency band. This reduction was much greater than those observed during the annual noise-level cycles of national or religious holidays. Daily CO2 emissions fell only 11 to 25% ([ 14 ][14]), whereas anthropogenic vibrations dropped by 75% in most countries that imposed lockdown measures. Among countries with the greatest noise reductions were China, Italy, and France—all densely populated places with strong government responses (that is, with high virus-containment indices) ([ 15 ][15]). Lecocq et al. also detected a correlation between seismic data and new types of time series, such as urban audible sound from acoustics data and cell phone mobility data. The authors observed the greatest correlations between seismic noise levels and two common types of pandemic mitigation: surface transportation and nonessential business activities. Lecocq et al. did not detect a strong correlation between lockdown and seismic noise reduction at other frequency bands, which might be explained by certain uninterrupted human activities such as power generation ([ 14 ][14]). For all its hardships, the lockdown has unlocked a door to scientific inquiry into environmental noise and global collaboration. At a fundamental level, low noise benefits traditional seismology, hence the recent noise decrease might open new windows of opportunity; study areas hindered by urban noise might now be targets for detecting microseismicity or for improved subsurface imaging. The crucial next step, as ever in seismology, is to determine the causative nature of these signals beyond their correlation—thus turning anthropogenic noise into informative signals that allow scientists to address new questions. For example: Is there feedback between anthropogenic vibrations and Earth processes? And will seismic monitoring of anthropogenic and environmental activities become complementary, economically valuable alternatives to conventional techniques? To achieve these advances, seismologists must develop new ways of processing data and modeling and interpreting results. Lecocq et al. exemplify seismological progress through best practices in scientific research: public data, open-access software and hardware, global cooperation, and crowdsourcing of citizen-science projects. All of the data are publicly available through open-access data centers at the Incorporated Research Institutions for Seismology (IRIS), which hosts and redistributes real-time seismograms from most of the stations participating in the Federation of Digital Seismograph Networks archive. A large proportion of the data used in the Lecocq et al. study was measured on seismic instruments that are powered on open-source Raspberry Pi computers hosted by citizen scientists. The Raspberry Shake network counts more than 3500 stations globally, all installed in homes, schools, and research institutions at 2 to 7% of the cost of conventional research or industrial sensors. The authors performed data analyses with open-source Python software Obspy, demonstrating the prevalence and usefulness of open-source community codes in modern science. Like the pandemic, the seismological community also is shaking up norms. One important example is the reorganization of research activities. Although physical borders are closed, Lecocq et al. demonstrate that, much like the global medical research on SARS-CoV-2, seismological research is and ought to be without borders. The new study represents scientists from 25 countries on five continents, and the authors shared the manuscript on public editing platforms (Google Docs, Slack) that allowed for all members of the community to contribute. Indeed, social seismology, which directly relates human activities and seismic waves, has sparked enthusiasm in the scientific community for urban seismology. The fall meeting of the American Geophysical Union (December 2020) will highlight the imminent wave of SARS-CoV-2–related seismological science in a special session called “Social Seismology.” ![Figure][16] Humans and nature excite seismic waves Seismometers record vibrations from everything, not only earthquakes. Shown are sources that induce seismic waves of different vibration modes (harmonic, diffuse, transient), detectable over large distances. GRAPHIC: N. DESAI/ SCIENCE 1. [↵][17]1. M. A. Denolle, 2. E. M. Dunham, 3. G. A. Prieto, 4. G. C. Beroza , Science 343, 399 (2014). [OpenUrl][18][Abstract/FREE Full Text][19] 2. [↵][20]1. K. Hosseini et al ., Geophys. J. Int. 220, 96 (2020). [OpenUrl][21] 3. [↵][22]1. T. Lecocq et al ., Science 369, 1338 (2020). [OpenUrl][23][CrossRef][24][PubMed][25] 4. [↵][26]1. B. Mortimer, 2. W. L. Rees, 3. P. Koelemeijer, 4. T. Nissen-Meyer , Curr. Biol. 28, R547 (2018). [OpenUrl][27][CrossRef][28] 5. [↵][29]1. E. A. Podolskiy, 2. F. Walter , Rev. Geophys. 54, 708 (2016). [OpenUrl][30] 6. [↵][31]1. G. Ekström, 2. C. P. Stark , Science 339, 1416 (2013). [OpenUrl][32][Abstract/FREE Full Text][33] 7. [↵][34]1. G. Olivier, 2. F. Brenguier, 3. R. Carey, 4. P. Okubo, 5. C. Donaldson , Geophys. Res. Lett. 46, 3734 (2019). [OpenUrl][35] 8. [↵][36]1. T. Clements, 2. M. A. Denolle , Geophys. Res. Lett. 45, 6459 (2018). [OpenUrl][37] 9. [↵][38]1. L. Gualtieri, 2. S. J. Camargo, 3. S. Pascale, 4. F. M. E. Pons, 5. G. Ekström , Earth Planet. Sci. Lett. 484, 287 (2018). [OpenUrl][39] 10. [↵][40]1. A. Mordret, 2. T. D. Mikesell, 3. C. Harig, 4. B. P. Lipovsky, 5. G. A. Prieto , Sci. Adv. 2, e1501538 (2016). [OpenUrl][41][FREE Full Text][42] 11. [↵][43]1. J. Díaz, 2. M. Ruiz, 3. P. S. Sánchez-Pastor, 4. P. Romero , Sci. Rep. 7, 15296 (2017). [OpenUrl][44][CrossRef][45][PubMed][46] 12. [↵][47]1. S. Schippkus, 2. M. Garden, 3. G. Bokelmann , Seismol. Res. Lett. 91, 2803 (2020). [OpenUrl][48] 13. [↵][49]1. T.-K. Hong, 2. R. Martin-Short, 3. R. M. Allen , Seismol. Res. Lett. 91, 2343 (2020). [OpenUrl][50] 14. [↵][51]1. C. Le Quéré et al ., Nat. Clim. Chang. 10, 647 (2020). [OpenUrl][52] 15. [↵][53]1. P. Poli, 2. J. Boaga, 3. I. Molinari, 4. V. Cascone, 5. L. Boschi , Sci. Rep. 10, 9404 (2020). [OpenUrl][54][CrossRef][55][PubMed][56] Acknowledgments: We thank L. Ermert and B. Liposky for their comments. [1]: #ref-1 [2]: #ref-2 [3]: #ref-3 [4]: #ref-4 [5]: #ref-5 [6]: #ref-6 [7]: #ref-7 [8]: #ref-8 [9]: #ref-9 [10]: #ref-10 [11]: #ref-11 [12]: #ref-12 [13]: #ref-13 [14]: #ref-14 [15]: #ref-15 [16]: pending:yes [17]: #xref-ref-1-1 "View reference 1 in text" [18]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.aulast%253DDenolle%26rft.auinit1%253DM.%2BA.%26rft.volume%253D343%26rft.issue%253D6169%26rft.spage%253D399%26rft.epage%253D403%26rft.atitle%253DStrong%2BGround%2BMotion%2BPrediction%2BUsing%2BVirtual%2BEarthquakes%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fscience.1245678%26rft_id%253Dinfo%253Apmid%252F24458636%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [19]: /lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEyOiIzNDMvNjE2OS8zOTkiO3M6NDoiYXRvbSI7czoyMzoiL3NjaS8zNjkvNjUwOS8xMjk5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ== [20]: #xref-ref-2-1 "View reference 2 in text" [21]: {openurl}?query=rft.jtitle%253DGeophys.%2BJ.%2BInt.%26rft.volume%253D220%26rft.spage%253D96%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [22]: #xref-ref-3-1 "View reference 3 in text" [23]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.aulast%253DLecocq%26rft.auinit1%253DT.%26rft.volume%253D369%26rft.issue%253D6509%26rft.spage%253D1338%26rft.epage%253D1343%26rft.atitle%253DGlobal%2Bquieting%2Bof%2Bhigh-frequency%2Bseismic%2Bnoise%2Bdue%2Bto%2BCOVID-19%2Bpandemic%2Blockdown%2Bmeasures%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fscience.abd2438%26rft_id%253Dinfo%253Apmid%252F32703907%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [24]: /lookup/external-ref?access_num=10.1126/science.abd2438&link_type=DOI [25]: /lookup/external-ref?access_num=32703907&link_type=MED&atom=%2Fsci%2F369%2F6509%2F1299.atom [26]: #xref-ref-4-1 "View reference 4 in text" [27]: {openurl}?query=rft.jtitle%253DCurr.%2BBiol.%26rft.volume%253D28%26rft.spage%253DR547%26rft_id%253Dinfo%253Adoi%252F10.1016%252Fj.cub.2018.03.062%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [28]: /lookup/external-ref?access_num=10.1016/j.cub.2018.03.062&link_type=DOI [29]: #xref-ref-5-1 "View reference 5 in text" [30]: {openurl}?query=rft.jtitle%253DRev.%2BGeophys.%26rft.volume%253D54%26rft.spage%253D708%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [31]: #xref-ref-6-1 "View reference 6 in text" [32]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.aulast%253DEkstrom%26rft.auinit1%253DG.%26rft.volume%253D339%26rft.issue%253D6126%26rft.spage%253D1416%26rft.epage%253D1419%26rft.atitle%253DSimple%2BScaling%2Bof%2BCatastrophic%2BLandslide%2BDynamics%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fscience.1232887%26rft_id%253Dinfo%253Apmid%252F23520108%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [33]: /lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEzOiIzMzkvNjEyNi8xNDE2IjtzOjQ6ImF0b20iO3M6MjM6Ii9zY2kvMzY5LzY1MDkvMTI5OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30= [34]: #xref-ref-7-1 "View reference 7 in text" [35]: {openurl}?query=rft.jtitle%253DGeophys.%2BRes.%2BLett.%26rft.volume%253D46%26rft.spage%253D3734%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [36]: #xref-ref-8-1 "View reference 8 in text" [37]: {openurl}?query=rft.jtitle%253DGeophys.%2BRes.%2BLett.%26rft.volume%253D45%26rft.spage%253D6459%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [38]: #xref-ref-9-1 "View reference 9 in text" [39]: {openurl}?query=rft.jtitle%253DEarth%2BPlanet.%2BSci.%2BLett.%26rft.volume%253D484%26rft.spage%253D287%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [40]: #xref-ref-10-1 "View reference 10 in text" [41]: {openurl}?query=rft.jtitle%253DScience%2BAdvances%26rft.stitle%253DSci%2BAdv%26rft.aulast%253DMordret%26rft.auinit1%253DA.%26rft.volume%253D2%26rft.issue%253D5%26rft.spage%253De1501538%26rft.epage%253De1501538%26rft.atitle%253DMonitoring%2Bsouthwest%2BGreenlands%2Bice%2Bsheet%2Bmelt%2Bwith%2Bambient%2Bseismic%2Bnoise%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fsciadv.1501538%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [42]: /lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo4OiJhZHZhbmNlcyI7czo1OiJyZXNpZCI7czoxMjoiMi81L2UxNTAxNTM4IjtzOjQ6ImF0b20iO3M6MjM6Ii9zY2kvMzY5LzY1MDkvMTI5OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30= [43]: #xref-ref-11-1 "View reference 11 in text" [44]: {openurl}?query=rft.jtitle%253DSci.%2BRep.%26rft.volume%253D7%26rft.spage%253D15296%26rft_id%253Dinfo%253Adoi%252F10.1038%252Fs41598-016-0028-x%26rft_id%253Dinfo%253Apmid%252F28127051%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [45]: /lookup/external-ref?access_num=10.1038/s41598-016-0028-x&link_type=DOI [46]: /lookup/external-ref?access_num=28127051&link_type=MED&atom=%2Fsci%2F369%2F6509%2F1299.atom [47]: #xref-ref-12-1 "View reference 12 in text" [48]: {openurl}?query=rft.jtitle%253DSeismol.%2BRes.%2BLett.%26rft.volume%253D91%26rft.spage%253D2803%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [49]: #xref-ref-13-1 "View reference 13 in text" [50]: {openurl}?query=rft.jtitle%253DSeismol.%2BRes.%2BLett.%26rft.volume%253D91%26rft.spage%253D2343%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [51]: #xref-ref-14-1 "View reference 14 in text" [52]: {openurl}?query=rft.jtitle%253DNat.%2BClim.%2BChang.%26rft.volume%253D10%26rft.spage%253D647%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [53]: #xref-ref-15-1 "View reference 15 in text" [54]: {openurl}?query=rft.jtitle%253DSci.%2BRep.%26rft.volume%253D10%26rft.spage%253D9404%26rft_id%253Dinfo%253Adoi%252F10.1038%252Fs41598-020-66368-0%26rft_id%253Dinfo%253Apmid%252F32523080%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [55]: /lookup/external-ref?access_num=10.1038/s41598-020-66368-0&link_type=DOI [56]: /lookup/external-ref?access_num=32523080&link_type=MED&atom=%2Fsci%2F369%2F6509%2F1299.atom

Democratizing AI - new paper


Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data.

Democratizing AI: non-expert design of prediction tasks


Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data. We show that individuals without ML experience can collectively construct useful datasets and that predictive models can be learned on these datasets, but challenges remain. The prediction tasks designed by non-experts covered a broad range of domains, from politics and current events to health behavior, demographics, and more. Proper instructions are crucial for non-experts, so we also conducted a randomized trial to understand how different instructions may influence the types of prediction tasks being proposed. In general, understanding better how non-experts can contribute to ML can further leverage advances in Automatic machine learning and has important implications as ML continues to drive workplace automation.

Annotator Rationales for Labeling Tasks in Crowdsourcing

Journal of Artificial Intelligence Research

When collecting item ratings from human judges, it can be difficult to measure and enforce data quality due to task subjectivity and lack of transparency into how judges make each rating decision. To address this, we investigate asking judges to provide a specific form of rationale supporting each rating decision. We evaluate this approach on an information retrieval task in which human judges rate the relevance of Web pages for different search topics. Cost-benefit analysis over 10,000 judgments collected on Amazon's Mechanical Turk suggests a win-win. Firstly, rationales yield a multitude of benefits: more reliable judgments, greater transparency for evaluating both human raters and their judgments, reduced need for expert gold, the opportunity for dual-supervision from ratings and rationales, and added value from the rationales themselves. Secondly, once experienced in the task, crowd workers provide rationales with almost no increase in task completion time. Consequently, we can realize the above benefits with minimal additional cost.

Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy Artificial Intelligence

Mixed-initiative systems allow users to interactively provide feedback to potentially improve system performance. Human feedback can correct model errors and update model parameters to dynamically adapt to changing data. Additionally, many users desire the ability to have a greater level of control and fix perceived flaws in systems they rely on. However, how the ability to provide feedback to autonomous systems influences user trust is a largely unexplored area of research. Our research investigates how the act of providing feedback can affect user understanding of an intelligent system and its accuracy. We present a controlled experiment using a simulated object detection system with image data to study the effects of interactive feedback collection on user impressions. The results show that providing human-in-the-loop feedback lowered both participants' trust in the system and their perception of system accuracy, regardless of whether the system accuracy improved in response to their feedback. These results highlight the importance of considering the effects of allowing end-user feedback on user trust when designing intelligent systems.

AI-Labeling Crowdsourcing Platforms


Artificial intelligence (AI) is widely used in today's business such as for data analytics, natural language processing, or process automation. The emergence of artificial intelligence is based on decades of research for solving difficult computer science tasks and is now rapidly transforming business model innovation. Companies that are not considering artificial intelligence will be vulnerable to those companies that are equipped with artificial intelligence technology. While companies like Google, Amazon, and Tesla have already innovated their business models with artificial intelligence, medium and small caps have limited budgets for putting much effort into setting up such capabilities. One high-effort task in creating artificial intelligence services is the pre-processing of data and the training of machine learning models.