Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning architecture trained to decode sentences from either electro- (EEG) or magneto-encephalography (MEG), while participants typed briefly memorized sentences on a QWERTY keyboard. With MEG, Brain2Qwerty reaches, on average, a character-error-rate (CER) of 32% and substantially outperforms EEG (CER: 67%). For the best participants, the model achieves a CER of 19%, and can perfectly decode a variety of sentences outside of the training set. While error analyses suggest that decoding depends on motor processes, the analysis of typographical errors suggests that it also involves higher-level cognitive factors. Overall, these results narrow the gap between invasive and non-invasive methods and thus open the path for developing safe brain-computer interfaces for non-communicating patients.

In this paper, we introduce a groundbreaking end-to-end (E2E) framework for decoding invasive brain signals, marking a significant advancement in the field of speech neuroprosthesis. Our methodology leverages the comprehensive reasoning abilities of large language models (LLMs) to facilitate direct decoding. By fully integrating LLMs, we achieve results comparable to the state-of-the-art cascade models. Our findings underscore the immense potential of E2E frameworks in speech neuroprosthesis, particularly as the technology behind brain-computer interfaces (BCIs) and the availability of relevant datasets continue to evolve. This work not only showcases the efficacy of combining LLMs with E2E decoding for enhancing speech neuroprosthesis but also sets a new direction for future research in BCI applications, underscoring the impact of LLMs in decoding complex neural signals for communication restoration. Code will be made available at https://github.com/FsFrancis15/BrainLLM.

Scientists affiliated with the University of Melbourne and Synchron, Inc. published earlier this week in the Journal of NeuroInterventional Surgery the first-in-human study of Stentrode, a wireless neuroprosthesis that uses machine learning and a stent. What makes the Stentrode technology unique is that it is a stent that records brain activity inside a blood vessel in the brain. It is implanted through the jugular vein so there is no need for open brain surgery. The technology platform originated from the University of Melbourne, in a collaborative effort with the Royal Melbourne Hospital, the Florey Institute of Neuroscience and Mental Health, Monash University, and Synchron, Inc.. A brain-computer interface (BCI) enables two-way communications between the biological brain and a machine.

Extracting stimulus features from neuronal ensembles is of great interest to the development of neuroprosthetics that project sensory information directly to the brain via electrical stimulation. Machine learning strategies that optimize stimulation parameters as the subject learns to interpret the artificial input could improve device efficacy, increase prosthetic performance, ensure stability of evoked sensations, and improve power consumption by eliminating extraneous input. Recent advances extending deep learning techniques to non-Euclidean graph data provide a novel approach to interpreting neuronal spiking activity. For this study, we apply graph convolutional networks (GCNs) to infer the underlying functional relationship between neurons that are involved in the processing of artificial sensory information. Data was collected from a freely behaving rat using a four infrared (IR) sensor, ICMS-based neuroprosthesis to localize IR light sources. We use GCNs to predict the stimulation frequency across four stimulating channels in the prosthesis, which encode relative distance and directional information to an IR-emitting reward port. Our GCN model is able to achieve a peak performance of 73.5% on a modified ordinal regression performance metric in a multiclass classification problem consisting of 7 classes, where chance is 14.3%. Additionally, the inferred adjacency matrix provides a adequate representation of the underlying neural circuitry encoding the artificial sensation.

The patient pauses, recognizes the shape, then points to it with her finger. What she's doing is remarkable, not for what she remembers, but for how well she remembers. On average, she and seven other test subjects perform 37 percent better at the memory game with the brain pulses than they do without--making them the first humans on Earth to experience the memory-boosting benefits of a tailored neural prosthesis. If you want to get technical, the brain-booster in question is a "closed-loop hippocampal neural prosthesis." Closed loop because the signals passing between each patient's brain and the computer to which it's attached are zipping back and forth in near-real-time.

In an ordinary hospital room in Los Angeles, a young woman named Lauren Dickerson waits for her chance to make history. She's 25 years old, a teacher's assistant in a middle school, with warm eyes and computer cables emerging like futuristic dreadlocks from the bandages wrapped around her head. Three days earlier, a neurosurgeon drilled 11 holes through her skull, slid 11 wires the size of spaghetti into her brain, and connected the wires to a bank of computers. Now she's caged in by bed rails, with plastic tubes snaking up her arm and medical monitors tracking her vital signs. She tries not to move.

The age of the machine is well underway, and there is a very good chance that humanity will be left behind. Artificial intelligence is beating us at poker. It is beating us at Go. It is saving lives by identifying diseases when human doctors fail. It is running our grocery stores.

Neuroprosthesis aims to supersede a damaged or degenerated brain caused by accidents or aging by an artificial brain that simulates and thereby restores the impaired brain functions. To replace the real brain, the artificial brain has to simulate the same functions of the real brain, and the simulation has to be conducted in realtime. We have built a large-scale spiking network model of the cerebellum that is composed of more than 100,000 neuron units and acts as a versatile supervised learning machine for spatiotemporal information. We implement it on a graphics processing unit (GPU) to conduct the numerical simulation in realtime owing to the parallel computing capability of GPUs. We propose to use the present model towards neuroprosthesis.