Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the class token [CLS] and the tokens. We propose in this paper a new yet simple normalization method that separately normalizes embedding vectors respectively corresponding to normal tokens and the [CLS]token, in order to better capture their distinct characteristics and enhance downstream task performance. Our empirical study shows that the [CLS]embeddings learned with our separate normalization layer better encode the global contextual information and are distributed more uniformly in its anisotropic space. When the conventional normalization layer is replaced with a separate normalization layer, we observe an average 2.7% performance improvement in learning tasks from the image, natural language, and graph domains.

We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a poisoned dataset. We demonstrate the efficacy of our attack when unlearning is performed via retraining from scratch, the idealized setting of machine unlearning which other efficient methods attempt to emulate, as well as against the approximate unlearning approach of Graves et al. [2021].

Musk testifies at OpenAI trial it's not OK to'loot a charity' Elon Musk has taken the stand at a high-stakes trial over the future of OpenAI, casting his lawsuit against the ChatGPT maker as a defence of charitable giving. The world's richest person is suing OpenAI, its cofounder and chief executive officer, Sam Altman, and its president, Greg Brockman, and said on the stand on Tuesday that they betrayed him and the public by abandoning OpenAI's mission to be a benevolent steward of AI for humanity and transforming the nonprofit into a profit-seeking juggernaut. Musk, who founded carmaker Tesla and rocket company SpaceX, also said he is committed to serving the public by working 80-to 100-hour weeks and generally not taking vacations. "I like working and solving problems that make people's lives better," he said. Before Musk began testifying, Bill Savitt, a lawyer for OpenAI and Altman, told jurors during his opening statement it was Musk who saw dollar signs as he helped finance OpenAI's early growth and pushed it to become a for-profit business, one he might eventually lead as CEO.

Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution (i.e. DNA "characters") where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity.

In this supplementary material, we provide the proofs of convergence analysis in Section 1, 1-vs-1 transformation employed in the classification and semantic segmentation tasks in Section 2, the coordinate-wise and the preprocessing method of the LSTM teacher in Section 3, the loss functions of YOLO-v3 in Section 4, more experiments of image classification in Section 5, and the inferences of semantic segmentation in Section 6. A differentiable function e()is L-smooth with gradient Lipschitz constant C (uniformly Lipschitz continuous), if e(x) e(y) C x y, x,y. The function is called block-wise smooth with gradient Lipschitz Ci, if i e(x i,xi) ie(x i,x i) Ci xi x i, x,x (1) or with gradient Lipschitz constants { Ci}, if i e(x i,xi) ie(x i,xi) Ci x i x i, x,x (2) Further, Let Gmax max{Ci, Ci, k} C. Definition 2. For a differentiable function e(), if e(x) = 0, then x is a first-order stationary solution (SS1). For a differentiable function e(), if x is a SS1, and there exists ϵ > 0 so that for any y in the ϵ-neighborhood of x, we have e(x) e(y), then xis a local minimum. A saddle point xis an SS1 that is not a local minimum. If λmin( 2e(x)) < 0, x is a strict (non-degenerate) saddle point.

With the concept of teaching being introduced to the machine learning community, a teacher model start using dynamic loss functions to teach the training of a student model. The dynamic intends to set adaptive loss functions to different phases of student model learning. In existing works, the teacher model 1) merely determines the loss function based on the present states of the student model, i.e., disregards the experience of the teacher; 2) only utilizes the states of the student model, e.g., training iteration number and loss/accuracy from training/validation sets, while ignoring the states of the loss function. In this paper, we first formulate the loss adjustment as a temporal task by designing a teacher model with memory units, and, therefore, enables the student learning to be guided by the experience of the teacher model. Then, with a dynamic loss network, we can additionally use the states of the loss to assist the teacher learning in enhancing the interactions between the teacher and the student model. Extensive experiments demonstrate our approach can enhance student learning and improve the performance of various deep models on real-world tasks, including classification, objective detection, and semantic segmentation scenarios.

Elon Musk Testifies That He Started OpenAI to Prevent a'Terminator Outcome' The judge also warned Musk and Sam Altman to curb their "propensity to use social media to make things worse outside the courtroom" after both sides traded attacks online. Elon Musk and Sam Altman appeared in a federal courtroom together for the first time on Tuesday as they fight over OpenAI's decade-long evolution and what it means for the company's future. The trial in Musk's lawsuit against Altman could result in financial damages and, more significantly, governance changes at OpenAI that may complicate its plans for an initial public offering as soon as this year. As the first witness on the stand, Musk immediately sought to frame his case as more than just about OpenAI. Siding with Altman "will give license to looting every charity in America" and shake the "entire foundation of charitable giving," Musk told a panel of nine jurors advising US District Judge Yvonne Gonzalez Rogers on how to rule.

A trial pitting two founders of OpenAI - Sam Altman and Elon Musk - against each other has opened in California, with the sides presenting duelling narratives about the company's history and obligations to consumers. Musk, wearing a dark suit and tie, was asked by one of his lawyers what the lawsuit was about when he took the stand. It's actually very simple, he said. It's not okay to steal a charity... If it's okay to loot a charity, the entire foundation of charitable giving will be destroyed.

You can currently use the Photos app across Apple's operating systems to adjust things like saturation and contrast, apply filters, crop photos or use AI to remove objects with the Clean Up tool . Clean Up will apparently be one of several Apple Intelligence Tools after these new updates roll out, writes. Along with Clean Up, users will be able to use Extend to expand the background of the photo with generative AI, Enhance to automatically improve things like lighting and image quality and Reframe to shift the perspective of a photo after it's taken, primarily for Apple's spatial photos. The new features, if released, will bring Apple's photo-editing tools more in line with competitors like Google and Samsung, though both companies still lap Apple in their willingness to create entirely generated images. Google's Magic Editor feature, which debuted in 2023, still takes the cake in terms of giving users leeway to radically add to and change their photos.