Wang, Zhou
Learning in Log-Domain: Subthreshold Analog AI Accelerator Based on Stochastic Gradient Descent
Tageldeen, Momen K, Belgaid, Yacine, Mohan, Vivek, Wang, Zhou, Drakakis, Emmanuel M
In recent years, artificial intelligence (AI) has become an integral part of daily life, serving as a transformative tool across various professional domains [1] and driving personal applications through advancements in transformer models that power large language models (LLMs) [2]. However, both training and inference of AI models demand substantial computational and energy resources, which are becoming increasingly challenging to access [3, 4]. While server-class GPUs are effective for training, their energy inefficiency [5] and high costs present significant barriers [6]. Additionally, the environmental impact of energy-intensive AI systems has raised critical concerns about their role in exacerbating climate change [4]. Amdahl's law predicts that performance and efficiency gains are best achieved through innovative application-specific accelerator architectures rather than scaling up multi-core general-purpose processors [7]. Consequently, applicationspecific integrated circuits (ASICs), both digital and analog, have emerged as critical solutions for enabling highefficiency training and inference of artificial neural networks [7, 8, 9]. Digital accelerators are widely adopted for training workloads. Notable examples include the Brainwave Neural Processing Unit (NPU) [10], Google's Tensor Processing Unit (TPU) [11], and low-precision inference accelerators such as YodaNN [5], the Unified Neural Processing Unit (UNPU) [12], and BRein Memory [13].
Generated Contents Enrichment
Naseri, Mahdi, Qiu, Jiayan, Wang, Zhou
In this paper, we investigate a novel artificial intelligence generation task, termed as generated contents enrichment (GCE). Different from conventional artificial intelligence contents generation task that enriches the given textual description implicitly with limited semantics for generating visually real content, our proposed GCE strives to perform content enrichment explicitly on both the visual and textual domain, from which the enriched contents are visually real, structurally reasonable, and semantically abundant. Towards to solve GCE, we propose a deep end-to-end method that explicitly explores the semantics and inter-semantic relationships during the enrichment. Specifically, we first model the input description as a semantic graph, wherein each node represents an object and each edge corresponds to the inter-object relationship. We then adopt Graph Convolutional Networks on top of the input scene description to predict the enriching objects and their relationships with the input objects. Finally, the enriched description is fed into an image synthesis model to carry out the visual contents generation. Our experiments conducted on the Visual Genome dataset exhibit promising and visually plausible results.
Efficient Online Set-valued Classification with Bandit Feedback
Wang, Zhou, Qiao, Xingye
Conformal prediction is a distribution-free method that wraps a given machine learning model and returns a set of plausible labels that contain the true label with a prescribed coverage rate. In practice, the empirical coverage achieved highly relies on fully observed label information from data both in the training phase for model fitting and the calibration phase for quantile estimation. This dependency poses a challenge in the context of online learning with bandit feedback, where a learner only has access to the correctness of actions (i.e., pulled an arm) but not the full information of the true label. In particular, when the pulled arm is incorrect, the learner only knows that the pulled one is not the true class label, but does not know which label is true. Additionally, bandit feedback further results in a smaller labeled dataset for calibration, limited to instances with correct actions, thereby affecting the accuracy of quantile estimation. To address these limitations, we propose Bandit Class-specific Conformal Prediction (BCCP), offering coverage guarantees on a class-specific granularity. Using an unbiased estimation of an estimand involving the true label, BCCP trains the model and makes set-valued inferences through stochastic gradient descent. Our approach overcomes the challenges of sparsely labeled data in each iteration and generalizes the reliability and applicability of conformal prediction to online decision-making environments.
AutoML for Large Capacity Modeling of Meta's Ranking Systems
Yin, Hang, Liu, Kuang-Hung, Sun, Mengying, Chen, Yuxin, Zhang, Buyun, Liu, Jiang, Sehgal, Vivek, Panchal, Rudresh Rajnikant, Hotaj, Eugen, Liu, Xi, Guo, Daifeng, Zhang, Jamey, Wang, Zhou, Jiang, Shali, Li, Huayu, Chen, Zhengxing, Chen, Wen-Yen, Yang, Jiyan, Wen, Wei
Web-scale ranking systems at Meta serving billions of users is complex. Improving ranking models is essential but engineering heavy. Automated Machine Learning (AutoML) can release engineers from labor intensive work of tuning ranking models; however, it is unknown if AutoML is efficient enough to meet tight production timeline in real-world and, at the same time, bring additional improvements to the strong baselines. Moreover, to achieve higher ranking performance, there is an ever-increasing demand to scale up ranking models to even larger capacity, which imposes more challenges on the efficiency. The large scale of models and tight production schedule requires AutoML to outperform human baselines by only using a small number of model evaluation trials (around 100). We presents a sampling-based AutoML method, focusing on neural architecture search and hyperparameter optimization, addressing these challenges in Meta-scale production when building large capacity models. Our approach efficiently handles large-scale data demands. It leverages a lightweight predictor-based searcher and reinforcement learning to explore vast search spaces, significantly reducing the number of model evaluations. Through experiments in large capacity modeling for CTR and CVR applications, we show that our method achieves outstanding Return on Investment (ROI) versus human tuned baselines, with up to 0.09% Normalized Entropy (NE) loss reduction or $25\%$ Query per Second (QPS) increase by only sampling one hundred models on average from a curated search space. The proposed AutoML method has already made real-world impact where a discovered Instagram CTR model with up to -0.36% NE gain (over existing production baseline) was selected for large-scale online A/B test and show statistically significant gain. These production results proved AutoML efficacy and accelerated its adoption in ranking systems at Meta.
A Simple CW-SSIM Kernel-based Nearest Neighbor Method for Handwritten Digit Classification
Wang, Jiheng, Fan, Guangzhe, Wang, Zhou
We propose a simple kernel based nearest neighbor approach for handwritten digit classification. The "distance" here is actually a kernel defining the similarity between two images. We carefully study the effects of different number of neighbors and weight schemes and report the results. With only a few nearest neighbors (or most similar images) to vote, the test set error rate on MNIST database could reach about 1.5%-2.0%, which is very close to many advanced models.
Local Phase Coherence and the Perception of Blur
Wang, Zhou, Simoncelli, Eero P.
Blur is one of the most common forms of image distortion. It can arise from a variety of sources, such as atmospheric scatter, lens defocus, optical aberrations of the lens, and spatial and temporal sensor integration. Human observers are bothered by blur, and our visual systems are quite good at reporting whether an image appears blurred (or sharpened) [1, 2]. However, the mechanism by which this is accomplished is not well understood. Clearly, detection of blur requires some model of what constitutes an unblurred image. In recent years, there has been a surge of interest in the modelling of natural images, both for purposes of improving the performance of image processing and computer vision systems, and also for furthering our understanding of biological visual systems.