Goto

Collaborating Authors

 brain activity


Brain-Informed Fine-Tuning for Improved Multilingual Understanding in Language Models

Neural Information Processing Systems

Recent studies have demonstrated that fine-tuning language models with brain data can improve their semantic understanding, although these findings have so far been limited to English. Interestingly, similar to the shared multilingual embedding space of pretrained multilingual language models, human studies provide strong evidence for a shared semantic system in bilingual individuals. Here, we investigate whether fine-tuning language models with bilingual brain data changes model representations in a way that improves them across multiple languages. To test this, we fine-tune monolingual and multilingual language models using brain activity recorded while bilingual participants read stories in English and Chinese. We then evaluate how well these representations generalize to the bilingual participants' first language, their second language, and several other languages that the participants are not fluent in. We assess the fine-tuned language models on brain encoding performance and downstream NLP tasks. Our results show that bilingual brain-informed fine-tuned language models outperform their vanilla (pretrained) counterparts in both brain encoding performance and most downstream NLP tasks across multiple languages. These findings suggest that brain-informed fine-tuning improves multilingual understanding in language models, offering a bridge between cognitive neuroscience and NLP research. We make our code publicly available.


iMIND: Insightful Multi-subject Invariant Neural Decoding

Neural Information Processing Systems

Decoding visual signals holds an appealing potential to unravel the complexities of cognition and perception. While recent reconstruction tasks leverage powerful generative models to produce high-fidelity images from neural recordings, they often pay limited attention to the underlying neural representations and rely heavily on pretrained priors. As a result, they provide little insight into how individual voxels encode and differentiate semantic content or how these representations vary across subjects. To mitigate this gap, we present an insightful Multi-subject Invariant Neural Decoding (iMIND) model, which employs a novel dual-decoding framework-both biometric and semantic decoding-to offer neural interpretability in a data-driven manner and deepen our understanding of brain-based visual functionalities. Our iMIND model operates through three core steps: establishing a shared neural representation space across subjects using a ViT-based masked autoencoder, disentangling neural features into complementary subject-specific and object-specific components, and performing dual decoding to support both biometric and semantic classification tasks. Experimental results demonstrate that iMIND achieves state-of-the-art decoding performance with minimal scalability limitations. Furthermore, iMIND empirically generates voxel-object activation fingerprints that reveal object-specific neural patterns and enable investigation of subject-specific variations in attention to identical stimuli. These findings provide a foundation for more interpretable and generalizable subject-invariant neural decoding, advancing our understanding of the voxel semantic selectivity as well as the neural vision processing dynamics.


Embracing Trustworthy Brain Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies

Neural Information Processing Systems

However, their widespread adoption is hindered by critical limitations, such as low information transfer rates and extensive user-specific calibration. To overcome these challenges, recent research has explored the integration of Large Language Models (LLMs), extending the focus from simple command decoding to understanding complex cognitive states. Despite these advancements, deploying agentic AI faces technical hurdles and ethical concerns. Due to the lack of comprehensive discussion on this emerging direction, this position paper argues that the field is poised for a paradigm extension from BCI to Brain-Agent Collaboration (BAC). We emphasize reframing agents as active and collaborative partners for intelligent assistance rather than passive brain signal data processors, demanding a focus on ethical data handling, model reliability, and a robust human-agent collaboration framework to ensure these systems are safe, trustworthy, and effective.


Bridging Brains and Concepts: Interpretable Visual Decoding from fMRI with Semantic Bottlenecks

Neural Information Processing Systems

Decoding of visual stimuli from noninvasive neuroimaging techniques such as functional magnetic resonance (fMRI) has advanced rapidly in the last years; yet, most high-performing brain decoding models rely on complicated, non-interpretable latent spaces. In this study we present an interpretable brain decoding framework that inserts a semantic bottleneck into BrainDiffuser, a well established, simple and linear decoding pipeline. We firstly produce a 214 dimensional binary interpretable space L for images, in which each dimension answers to a specific question about the image (e.g., "Is there a person?",


This man with ALS is "the first power user" of a brain implant that lets him speak

MIT Technology Review

Casey Harrell has had a set of electrodes embedded in his brain for almost three years. Harrell, who has amyotrophic lateral sclerosis (ALS) and is paralyzed, first used his brain-computer interface (BCI) to "speak" sentences with the help of a research team in 2023. Since then, Harrell has clocked thousands of hours of use. He can use the device largely independently, once he's been "plugged in" with the help of a carer. His team has added new features to it, and Harrell also uses it to surf the web and perform his job.


REFED: A Subject Real-time Dynamic Labeled EEG-fNIRS Synchronized Recorded Emotion Dataset

Neural Information Processing Systems

Affective brain-computer interfaces (aBCIs) play a crucial role in personalized human-computer interaction and neurofeedback modulation. To develop practical and effective aBCI paradigms and to investigate the spatial-temporal dynamics of brain activity under emotional inducement, portable electroencephalography (EEG) signals have been widely adopted. To further enhance spatial-temporal perception, functional near-infrared spectroscopy (fNIRS) has attracted increasing interest in the aBCI field and has been explored in combination with EEG. However, existing datasets typically provide only static fixation labels, overlooking the dynamic changes in subjects' emotions. Notably, some studies have attempted to collect continuously annotated emotional data, but they have recorded only peripheral physiological signals without directly observing brain activity, limiting insight into underlying neural states under different emotions.


Three near-death experiences that convinced doctors the soul may exist

Daily Mail - Science & tech

SNL season finale cold open sees ghost of Jeffrey Epstein played by Will Ferrell'haunt' Trump as dark jokes leave viewers shocked Jordon Hudson blasts double standards over Mike Vrabel and Dianna Russini'affair' scandal: 'What is going on?' No one wants to hang out with her': Why Meghan and Harry have been ditched by A-list friends as insiders reveal Oprah's merciless snub, why the Clooneys now want nothing to do with them - and how SHE'S the problem Truth about Kate Middleton's past before Prince William... we Americans see this for what it is: KENNEDY Kim Kardashian roasted over'ridiculous' outfit at Gucci show as she sits front row with Anna Wintour and Mariah Carey I was on track to make $1 million... then I quit my job and moved into an off-grid tiny home with no running water or electricity Professional tasters decide best and worst fast food cheeseburger - do you agree? Hamptons cancer cluster: Rates are spiking in summer enclave of New York's wealthy elite... and doctors think they know the tragic reason why Disturbing trove of images woke Los Angeles mayor Karen Bass doesn't want you to see: Filthy truth is so much worse than people think... Taylor Swift dazzles in glittering gown as she and Travis Kelce steal the spotlight at friend's wedding in NYC Golf star becomes instant fan favorite after stopping to smoke a cigarette with crowd in the middle of the PGA Championship: 'Man of the people' New kind of penis enlargement surgery will add inches, claims the doctor set to offer it... but there is a gruesome detail that may make some think twice She was every bit the adoring mother... then a leaked video exposed a'sadistic' secret even cops said'will bring tears to your eyes' I saw a 40-year-old middle-class mom in a psychiatric ward after a single hit of this drug. Her symptoms were terrifying but it's so common now... here's what you must know: DR MAX PEMBERTON Expert reveals the best way to cut the bread - and why you should never leave a'hinge' 'I saw things I can never unsee': Man who snuck into Air India crash morgue reveals what he saw... why it could blow apart the pilot suicide theory... and what happened when we visited the lone survivor Many people have reported near-death experiences, but in some cases, survivors appeared to bring back something far more unsettling than memories. Some survivors claimed they saw and heard things that should have been impossible while they were clinically dead, including conversations in operating rooms and objects located far outside their hospital beds. Several of the most famous cases involved patients whose brains allegedly showed little or no measurable activity at the time of their experiences.


Inverting Foundation Models of Brain Function with Simulation-Based Inference

arXiv.org Machine Learning

Foundation models of brain activity promise a new frontier for in silico neuroscience by emulating neural responses to complex stimuli across tasks and modalities. A natural next step is to ask whether these models can also be used in reverse. Can we recover a stimulus or its properties from synthetic brain activity? We study this question in a proof-of-concept setting using TRIBEv2. We pair the brain emulator with large language models (LLMs) that generate news headlines from linguistic parameters such as valence, arousal, and dominance. We then use simulation-based inference to learn a probabilistic mapping from brain maps to latent stimulus parameters. Our results show that these parameters can be recovered from predicted brain maps, validating the quality of neural encodings. They also show that LLMs can serve as controllable stimulus generators for simulated experiments. Together, these findings provide a step toward decoding and inverse design with foundation brain models.



Exploring the trade-off between deep-learning and explainable models for brain-machine interfaces

Neural Information Processing Systems

People with brain or spinal cord-related paralysis often need to rely on others for basic tasks, limiting their independence. A potential solution is brain-machine interfaces (BMIs), which could allow them to voluntarily control external devices (e.g., robotic arm) by decoding brain activity to movement commands. In the past decade, deep-learning decoders have achieved state-of-the-art results in most BMI applications, ranging from speech production to finger control. However, the'black-box' nature of deep-learning decoders could lead to unexpected behaviors, resulting in major safety concerns in real-world physical control scenarios. In these applications, explainable but lower-performing decoders, such as the Kalman filter (KF), remain the norm. In this study, we designed a BMI decoder based on KalmanNet, an extension of the KF that augments its operation with recurrent neural networks to compute the Kalman gain.