We present the Mixture-of-Tunable-Experts (MoTE), a method that extends the Mixture-of-Experts architecture of Large Language Models (LLMs). Without additional training, MoTE enables meaningful and focused behavior changes in LLMs on-the-fly during inference time. By analyzing the digital LLM brain of DeepSeek-R1 using a technique we dub 'functional Token Resonance Imaging' (fTRI) - inspired by fMRI and using prompts designed to elicit specific behavior (e.g., 'What happened {time}{place}?') - we empirically identify distinctive experts associated with behaviors like refusal responses. Using MoTE we are able to intervene and control such specific behavior. We switched off the top 10 most refusal-relevant experts (0.07% of R1's 14,848 routed experts), achieving a 52% refusal reduction on sensitive reference prompts without performance degradation on MT-Bench. Random expert deactivation resulted in smaller behavioral shifts with increased noise, whereas forced expert activation led to significantly higher refusal rates. Our approach shares similarities with sparse autoencoders (SAEs) in terms of explainability and steerability. Unlike SAEs, MoTE does not require large training efforts, as within MoEs with a vast number of experts, specialization already emerged naturally during pretraining. Our findings suggest that significant functional mechanisms in Mixture-of-Experts architectures can at least partially be localized in a small number of specific experts, rather than being distributed throughout the model's weights. Expert subgroups can be tuned to trigger significant behavior variations, providing insights into the inner workings of LLMs.

We report the effects of replacing the scaled dot-product (within softmax) attention with the negative-log of Euclidean distance. This form of attention simplifies to inverse distance weighting interpolation. Used in simple one hidden layer networks and trained with vanilla cross-entropy loss on classification problems, it tends to produce a key matrix containing prototypes and a value matrix with corresponding logits. We also show that the resulting interpretable networks can be augmented with manually-constructed prototypes to perform low-impact handling of special cases.

TikTok may get Banned in 2023-24 due to A.I. Ethics and privacy rules (Dec, 2022)

This is an Op-ed about the future of the internet and, while speculative, it's an example and an attempt to demonstrate how Artificial Intelligence at scale in a human would or could have disastrous impacts without AI regulation and AI ethics to protect us. GPT-3 stands for Generative Pre-trained Transformer. As you likely already know GPT-3 is an autoregressive language model that uses deep learning to produce human-like text. It is the third-generation language prediction model in the GPT-n series (and the successor to GPT-2) created by Microsoft-funded OpenAI (that was supposed to be a not for profit firm). In 2021 we've had a NLP-explosion year in terms of Artificial Intelligence activity.

Abstract: The fields of machine learning and statistics have invested great efforts into designing algorithms, models, and approaches that better predict future observations. Larger and richer data have also been shown to improve predictive power. This is especially true in the world of human behavioral big data, as is evident from recent advances in behavioral prediction technology. Large internet platforms that collect behavioral big data predict user behavior for their internal commercial purposes as well as for third parties, such as advertisers, insurers, security forces, and political consulting firms, who utilize the predictions for user-level personalization, targeting, and other decision-making. While machine learning algorithmic and data efforts are directed at improving predicted values, the internet platforms can minimize prediction error by «pushing» users' actions towards their predicted values using behavior modification techniques.

The fields of statistics and machine learning design algorithms, models, and approaches to improve prediction. Larger and richer behavioral data increase predictive power, as evident from recent advances in behavioral prediction technology. Large internet platforms that collect behavioral big data predict user behavior for internal purposes and for third parties (advertisers, insurers, security forces, political consulting firms) who utilize the predictions for personalization, targeting and other decision-making. While standard data collection and modeling efforts are directed at improving predicted values, internet platforms can minimize prediction error by "pushing" users' actions towards their predicted values using behavior modification techniques. The better the platform can make users conform to their predicted outcomes, the more it can boast its predictive accuracy and ability to induce behavior change. Hence, platforms are strongly incentivized to "make predictions true". This strategy is absent from the ML and statistics literature. Investigating its properties requires incorporating causal notation into the correlation-based predictive environment---an integration currently missing. To tackle this void, we integrate Pearl's causal do(.) operator into the predictive framework. We then decompose the expected prediction error given behavior modification, and identify the components impacting predictive power. Our derivation elucidates the implications of such behavior modification to data scientists, platforms, their clients, and the humans whose behavior is manipulated. Behavior modification can make users' behavior more predictable and even more homogeneous; yet this apparent predictability might not generalize when clients use predictions in practice. Outcomes pushed towards their predictions can be at odds with clients' intentions, and harmful to manipulated users.

Banks love to brag about how many data scientists they're hiring and their shiny machine-learning "centers of excellence." In the 2018 JP Morgan Chase annual report, CEO Jamie Dimon said the company had gone "all in" on artificial intelligence, adding that artificial intelligence and machine learning were "being deployed across virtually everything we do." Not to be outdone, HSBC has opened multiple "data and innovation labs" around the world, in order to build artificial intelligence tools that can take in the bank's more than 10 petabytes of data. Citigroup, Bank of America, and Capital One also boast about their artificial intelligence capabilities, particularly to their would-be investors. Of course, some of this is hype: Banks believe they can get a certain brand patina from looking and acting like tech companies.

In the future according to robotics researchers, robots will likely fight our wars, care for our elderly, babysit our children, and serve and entertain us in a wide variety of situations. But as robotic development continues to grow, one subfield of robotics research is lagging behind other areas: roboethics, or ensuring that robot behavior adheres to certain moral standards. In a new paper that provides a broad overview of ethical behavior in robots, researchers emphasize the importance of being proactive rather than reactive in this area. The authors, Ronald Craig Arkin, Regents' Professor and Director of the Mobile Robot Laboratory at the Georgia Institute of Technology in Atlanta, Georgia, along with researchers Patrick Ulam and Alan R. Wagner, have published their overview of moral decision making in autonomous systems in a recent issue of the Proceedings of the IEEE. "Probably at the highest level, the most important message is that people need to start to think and talk about these issues, and some are more pressing than others," Arkin told PhysOrg.com.