Mathematically characterizing the implicit regularization induced by gradient-based optimization is a longstanding pursuit in the theory of deep learning. A widespread hope is that a characterization based on minimization of norms may apply, and a standard test-bed for studying this prospect is matrix factorization (matrix completion via linear neural networks). It is an open question whether norms can explain the implicit regularization in matrix factorization. The current paper resolves this open question in the negative, by proving that there exist natural matrix factorization problems on which the implicit regularization drives all norms (and quasi-norms) towards infinity. Our results suggest that, rather than perceiving the implicit regularization via norms, a potentially more useful interpretation is minimization of rank. We demonstrate empirically that this interpretation extends to a certain class of non-linear neural networks, and hypothesize that it may be key to explaining generalization in deep learning.

Diffusion models have achieved remarkable performance in generative modeling, yet their theoretical foundations are often intricate, and the gap between mathematical formulations in papers and practical open-source implementations can be difficult to bridge. Existing tutorials primarily focus on deriving equations, offering limited guidance on how diffusion models actually operate in code. To address this, we present a concise implementation of approximately 300 lines that explains diffusion models from a code-execution perspective. Our minimal example preserves the essential components -- including forward diffusion, reverse sampling, the noise-prediction network, and the training loop -- while removing unnecessary engineering details. This technical report aims to provide researchers with a clear, implementation-first understanding of how diffusion models work in practice and how code and theory correspond. Our code and pre-trained models are available at: https://github.com/disanda/GM/tree/main/DDPM-DDIM-ClassifierFree.

The distribution closeness testing (DCT) assesses whether the distance between a distribution pair is at least $ε$-far. Existing DCT methods mainly measure discrepancies between a distribution pair defined on discrete one-dimensional spaces (e.g., using total variation), which limits their applications to complex data (e.g., images). To extend DCT to more types of data, a natural idea is to introduce maximum mean discrepancy (MMD), a powerful measurement of the distributional discrepancy between two complex distributions, into DCT scenarios. However, we find that MMD's value can be the same for many pairs of distributions that have different norms in the same reproducing kernel Hilbert space (RKHS), making MMD less informative when assessing the closeness levels for multiple distribution pairs. To mitigate the issue, we design a new measurement of distributional discrepancy, norm-adaptive MMD (NAMMD), which scales MMD's value using the RKHS norms of distributions. Based on the asymptotic distribution of NAMMD, we finally propose the NAMMD-based DCT to assess the closeness levels of a distribution pair. Theoretically, we prove that NAMMD-based DCT has higher test power compared to MMD-based DCT, with bounded type-I error, which is also validated by extensive experiments on many types of data (e.g., synthetic noise, real images). Furthermore, we also apply the proposed NAMMD for addressing the two-sample testing problem and find NAMMD-based two-sample test has higher test power than the MMD-based two-sample test in both theory and experiments.

With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact of their norms to the performance of VLMs. This motivates us to pose an unexplored research question: "Do we need to normalize the soft prompts in VLMs?" To fill this research gap, we first uncover a phenomenon, called the Low-Norm Effect by performing extensive corruption experiments, suggesting that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To harness this effect, we propose a novel method named Normalizing the soft-prompt vectors of vision-language models (Nemesis) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work is the first to systematically investigate the role of norms of soft-prompt vector in VLMs, offering valuable insights for future research in soft-prompt tuning. The code is available at https://github.com/ShyFoo/Nemesis. In the age of large-scale pretrained vision-language models (VLMs), such as CLIP (Radford et al., 2021), Flamingo (Alayrac et al., 2022), and BLIP (Li et al., 2022), soft-prompt-based methods, also known as prompt-tuning, have emerged as a dominant approach for adapting these models to a wide range of downstream tasks. For instance, Zhou et al. (2022b) propose a Context Optimization (CoOp) method to learn soft prompts in a continuous space of CLIP for image classification tasks. Additionally, Rao et al. (2022) and Du et al. (2022) also employ prompt-tuning to address dense prediction and open-vocabulary object detection tasks, respectively. Recent research in the field of VLMs has been primarily focused on enhancing model performance through the alignment of visual and textual features. For instance, in (Lu et al., 2022), the weight distribution of output embeddings is estimated, while Zang et al. (2022) propose a joint optimization approach for prompts across multiple modalities.

Speaker identity plays a significant role in human communication and is being increasingly used in societal applications, many through advances in machine learning. Speaker identity perception is an essential cognitive phenomenon that can be broadly reduced to two main tasks: recognizing a voice or discriminating between voices. Several studies have attempted to identify acoustic correlates of identity perception to pinpoint salient parameters for such a task. Unlike other communicative social signals, most efforts have yielded inefficacious conclusions. Furthermore, current neurocognitive models of voice identity processing consider the bases of perception as acoustic dimensions such as fundamental frequency, harmonics-to-noise ratio, and formant dispersion. However, these findings do not account for naturalistic speech and within-speaker variability. Representational spaces of current self-supervised models have shown significant performance in various speech-related tasks. In this work, we demonstrate that self-supervised representations from different families (e.g., generative, contrastive, and predictive models) are significantly better for speaker identification over acoustic representations. We also show that such a speaker identification task can be used to better understand the nature of acoustic information representation in different layers of these powerful networks. By evaluating speaker identification accuracy across acoustic, phonemic, prosodic, and linguistic variants, we report similarity between model performance and human identity perception. We further examine these similarities by juxtaposing the encoding spaces of models and humans and challenging the use of distance metrics as a proxy for speaker proximity. Lastly, we show that some models can predict brain responses in Auditory and Language regions during naturalistic stimuli.

In this article, we will go through the concept of outliers in statistics and its application in the field of Machine Learning. Starting from scratch, we will build up to identifying outliers and…

Step by step instructions to train Yolo-v5 & do Inference(from ultralytics) to count the blood cells and localize them. I vividly remember that I tried to do an object detection model to count the RBC, WBC, and platelets on microscopic blood-smeared images using Yolo v3-v4, but I couldn't get as much as accuracy I wanted and the model never made it to production. Now recently I came across the release of the Yolo-v5 model from Ultralytics, which is built using PyTorch. I was a bit skeptical to start, owing to my previous failures, but after reading the manual in their Github repo, I was very confident this time and I wanted to give it a shot. And it worked like a charm, Yolo-v5 is easy to train and easy to do inference.

Step by step instructions to train Yolo-v5 & do Inference(from ultralytics) to count the blood cells and localize them. I vividly remember that I tried to do an object detection model to count the RBC, WBC, and platelets on microscopic blood-smeared images using Yolo v3-v4, but I couldn't get as much as accuracy I wanted and the model never made it to the production. Now recently I came across the release of the Yolo-v5 model from Ultralytics, which is built using PyTorch. I was a bit skeptical to start, owing to my previous failures, but after reading the manual in their Github repo, I was very confident this time and I wanted to give it a shot. And it worked like a charm, Yolo-v5 is easy to train and easy to do inference.

As we continue to navigate our way through unknown territory, with lockdown in full swing around the world, the Coronavirus pandemic is changing online behaviours significantly and possibly permanently both for consumers and businesses. People have been forced to access information in new ways, interact and purchase new and different products and services online, and the longer this continues, the more likely these are to become habits. So I thought I would take this opportunity to explain what it might mean for businesses and recruitment in a short to longer timeframe as we progress through lockdown. My fellow co-founder at Leadoo Marketing Technologies, Mikael de Costa is author of The Startup Warrior and a multiple entrepreneur in our native Finland including founding Jobilla which was a recruitment digital marketing venture. Launching Leadoo MT, which concentrates on website conversion by engaging in chat bot conversations with customers, was born from Mikael's recruitment experience as he discovered that talent marketing and employer branding efforts were going to waste, as potential applicants were not converting.

Other difficult tasks, more generally, are how to obtain a robust performance in teamworks (Cohen and Levesque 1990); how to prevent agents from dropping their commitments; or better, how to regulate agents dropping their commitments to a joint action to not disrupt the common activity and preclude the common goal being achieved (Jennings 1995; Singh 1995; Kinny and Georgeff 1991). These tasks have now entered the MAS field's common knowledge. Other problems are perhaps less obvious. The Second International Conference on Multiagent Systems (ICMAS '96) Workshop on Norms, Obligations, and Conventions was held in Kyoto, Japan, from 10 to 13 December 1996. Participants included scientists from deontic logic, database framework, decision theory, agent architecture, cognitive modeling, and legal expert systems.