However, when the size of labeled data is very small (say a few labeled samples per class), SSL performs poorly and unstably, possibly due to the low qualityoflearnedpseudolabels.Inthispaper,weproposeanewSSLmethodcalled DP-SSL that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data. Different from existing DP methods that rely on human experts to provide initial labeling functions (LFs), we develop a multiple-choice learning (MCL) based approach to automatically generate LFs fromscratchinSSLstyle. Withthenoisylabelsproduced bytheLFs,wedesign a label model to resolve the conflict and overlap among the noisy labels, and finally infer probabilistic labels for unlabeled samples.

To create a large amount of training labels for machine learning models effectively and efficiently, researchers have turned to Weak Supervision (WS), which uses programmatic labeling sources rather than manual annotation. Existing works of WS for binary classification typically assume the presence of labeling sources that are able to assign both positive and negative labels to data in roughly balanced proportions. However, for many tasks of interest where there is a minority positive class, negative examples could be too diverse for developers to generate indicative labeling sources. Thus, in this work, we study the application of WS on binary classification tasks with positive labeling sources only. We propose WEAPO, a simple yet competitive WS method for producing training labels without negative labeling sources. On 10 benchmark datasets, we show WEAPO achieves the highest averaged performance in terms of both the quality of synthesized labels and the performance of the final classifier supervised with these labels. We incorporated the implementation of \method into WRENCH, an existing benchmarking platform.

For all the advances in the development of artificial intelligence algorithms and models, the majority of potential applications never make it to production because of the time and expense of labeling data to train the model. That's a problem Snorkel.ai has set out to automate. "The not-so-hidden secret about AI today is that despite all the technological and tooling enhancements, but 80 to 90%, of the cost, for many use cases, just goes into manually collecting and labeling and curating this data, this training data that the model learns from," said company co-founder and CEO Alex Ratner. Ratner concedes that this is not the first field or even the first decade in which appropriately labeled data has been considered paramount. In a contributed post to TNS last year, Vikram Bahl outlined the challenges of preparing data for machine learning and AI.

We are witnessing a data labeling market explosion: labeling platforms have hit prime time. S&P Global released an October 11 report entitled *Avoiding Garbage in Machine Learning* in which it termed unlabeled data "garbage data" to highlight the importance of labeling in AI. The Economist recently noted that while spending on AI is growing from $38bn this year to $98bn in 2023, only 1 in 5 companies interested in AI has deployed machine learning models because of a shortage of labeled data. This is why "the market for data-labeling services may triple to $5bn by 2023." It is difficult not to notice the abundance of labeling startups being funded of late that are chasing after this market.

Subscribe to the O'Reilly Data Show Podcast to explore the opportunities and techniques driving big data, data science, and AI. Find us on Stitcher, TuneIn, iTunes, SoundCloud, RSS. In this episode of the Data Show, I speak with Alex Ratner, project lead for Stanford's Snorkel open source project; Ratner also recently garnered a faculty position at the University of Washington and is currently working on a company supporting and extending the Snorkel project. Snorkel is a framework for building and managing training data. Based on our survey from earlier this year, labeled data remains a key bottleneck for organizations building machine learning applications and services.

America has a proud tradition of garage inventors, but this one needs to spend more time in the garage.

America has a proud tradition of garage inventors, but this one needs to spend more time in the garage.

Dr. Ratner has written a unique book that distinguishes between statistical and machine-learning data mining. The book includes 14 statistical data mining and 17 machine-learning data mining techniques. All techniques are quite practical, making this volume a handbook for every statistician, data miner, and machine-learner. Let me describe a few chapters that present approaches and techniques that I really favored. Chapter 3 introduces a new data mining method: a smoother scatterplot based on CHAID.