Established in 2017, Facebook Reality Labs’ (FRL) Brain-Computer Interface (BCI) project began with an ambitious long-term goal: to develop a silent, non-invasive speech interface that would let people type just by imagining the words they want to say.
The team has made great progress on this mission over the course of four years, investing deeply in the exploration of head-mounted optical BCI as a potential input method for the next computing platform — in other words, a way to communicate in AR/VR with the speed of voice and the discreetness of typing. In addition to our internal efforts, we’ve supported a team of researchers at University of California San Francisco (UCSF) who are developing an implantable communications prosthesis for people who have lost the ability to speak. Facebook’s goal in funding this research has been to determine whether a silent interface capable of typing 100 words per minute is possible, and if so, what neural signals are required — a goal that is well aligned with UCSF’s work.
UCSF published the first results two years ago in Nature Communications, showing for the first time that a small set of spoken words and phrases can be decoded from brain activity in real time. Since then, UCSF also demonstrated full-sentence decoding from brain to text using machine learning.
Today we’re excited to celebrate the next chapter of this work and a new milestone that the UCSF team has achieved and published in The New England Journal of Medicine: the first time someone with severe speech loss has been able to type out what they wanted to say almost instantly, simply by attempting speech. In other words, UCSF has restored a person’s ability to communicate by decoding brain signals sent from the motor cortex to the muscles that control the vocal tract — a milestone in neuroscience. These results mark the culmination of a decade of Dr. Edward Chang’s research at UCSF.
“My research team at UCSF has been working on this [speech neuroprosthesis] goal for over a decade. We’ve learned so much about how speech is processed in the brain during this time, but it’s only in the last five years that advances in machine learning have allowed us to get to this key milestone,” says Edward Chang, Chair of Neurosurgery, UCSF. “That combined with Facebook’s machine learning advice and funding really accelerated our progress.”
A new milestone for the field of BCI
This final phase of the project, which we call Project Steno, kicked off in 2019 in the Chang Lab at UCSF and involved a research participant who had lost the ability to speak normally following a series of strokes. The participant underwent elective surgery to place electrodes on the surface of his brain. During the course of the study, the participant worked directly with the UCSF team to collect dozens of hours of attempted speech with a BCI. This data was in turn used by UCSF to create machine learning models for speech detection and word classification. Through this study, the participant was able to truly communicate in real time despite the strokes that paralyzed him over 16 years ago.
Previous studies from UCSF, such as the Nature Communications study, have successfully decoded a small set of full, spoken words and phrases from brain activity in real time, and other Chang Lab studies showed that their system is able to recognize a significantly larger vocabulary with extremely low word error rates. However, these results were all achieved while people were actually speaking out loud, and we didn’t yet know whether it would be possible to decode words in real time when people simply intended to speak. The results from the study published today bring this all together, demonstrating the successful decoding of attempted conversational speech in real time. We’ve learned a lot from Project Steno, particularly as it applies to how algorithms can use language models to improve accuracy for brain-to-text communication.
“Project Steno is the first demonstration of attempted speech combined with language models to drive a BCI,” notes FRL Neural Engineering Research Manager Emily Mugler. “The result is a beautiful example of how we can leverage the statistical properties inherent in language — how one word can lead to another in the construction of a sentence — to dramatically increase the accuracy of a BCI.”
Just like your phone can use auto-correct and auto-complete to improve the accuracy of what you type in a text, we can use the same techniques with a BCI to improve the accuracy of the algorithm’s prediction of what someone wants to communicate.
Facebook’s contribution to Project Steno
Facebook provided high level feedback, machine learning advice, and funding throughout Project Steno, but UCSF designed and oversaw the study, and worked directly with the participant. Facebook was not involved in data collection with the research participant in any way; all the data remains onsite at UCSF and under the control of UCSF at all times. To be clear, Facebook has no interest in developing products that require implanted electrodes. Facebook’s funding enabled UCSF to dramatically increase their server capacity, allowing them to test more models simultaneously and achieve more accurate results.
Finally, Mugler led the FRL Research BCI team’s technical feedback, advising on the methods used to help the participant learn how to use the BCI. How do you train someone to speak with only their brain? This is a non-trivial feat given nothing like it has ever been done before. Mugler, who joined Facebook in the early days of our BCI program in 2017, has spent much of her career focused on restorative communication BCIs for patients who have lost the ability to speak due to conditions such as ALS.
“To see this work come to fruition has been a dream for the field for so long, and for me personally,” says Mugler. “As a BCI scientist, a core pursuit for me throughout my career has been to demonstrate that the neural signals that drive speech articulation can be decoded for a more efficient BCI that can be used for communication. These results have unlocked a lot of possibilities for assistive technologies that could significantly improve quality of life for those with speech impairment.”
The results published today by UCSF carry crucial implications for the future of assistive technology, as this has the potential to help unlock conversational communication for patients with similar injuries. We’re excited to see Project Steno’s impact manifest across the field of neuroscience long into the future.
Exploring high-bandwidth interaction for AR/VR