Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end model. With its increasing popularity, it is critical to have some tool for users to understand the influence of each component (e.g., the source vote or training data) in the pipeline and interpret the end model behavior. To achieve this, we build on Influence Function (IF) and propose source-aware IF2, which leverages the generation process of the probabilistic labels to decompose the end model's training objective and then calculate the influence associated with each (data, source, class) tuple. These primitive influence score can then be used to estimate the influence of individual component of PWS, such as source vote, supervision source, and training data. On datasets of diverse domains, we demonstrate multiple use cases: (1) interpreting incorrect predictions from multiple angles that reveals insights for debugging the PWS pipeline, (2) identifying mislabeling of sources with a gain of 9%-37% over baselines, and (3) improving the end model's generalization performance by removing harmful components in the training objective (13%-24% better than ordinary IF).

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

One of the biggest machine learning trends you can expect to see in 2021 and beyond is the broader adoption of a machine learning segment called deep learning and a method it employs called weak supervision. This method is faster, more streamlined, and has many benefits. Moving forward, companies will assess how they can leverage this in their business to execute on desired automation tasks and learn and predict the best ways to accomplish task completion– all without the heavy burden of human intervention. Weak supervision is bringing us ever closer to software that can think and act on its own, and here we will break down the pitfalls of the past, the methods of the future, and what it means for your business. Further solidifying weak supervision as an entirely accepted practice in our immediate future is the added element of Explainable AI (XAI) that serves to future-proof this new approach to ramping up an AI system.