Peng-Jen Chen is well aware of how language barriers can affect people’s ability to communicate.

Chen grew up in Taiwan speaking Mandarin Chinese, but his father, Sheng-Jiang Chen, a 70-year-old retired factory lead technician, hails from Southern Taiwan, where Taiwanese Hokkien is widely spoken. Though the two languages are related, they’re different enough that Chen’s father sometimes finds it tricky to conduct complex conversations in Mandarin. “I have always wished my father could communicate with everyone in Taiwanese Hokkien, which is the language he’s most comfortable speaking,” said Chen, a Meta AI researcher. “He understands Mandarin well but speaks more slowly when communicating about complex topics.”

But rather than simply worrying, Chen is doing something about the problem — he’s leading the development of new technology to translate between Hokkien and English.

This is a daunting task, because while languages like Mandarin, English, and Spanish are both written and spoken, Hokkien — which is widely spoken within the Chinese diaspora — is primarily oral. In fact, Chen and his team of researchers are among the first to use artificial intelligence (AI) to construct a translation system for languages like Hokkien that lack a formal or widely known writing system. While the initial stage of the project translates between English and Hokkien, researchers plan to allow the translation of more unwritten languages. It’s part of Meta’s ongoing effort to develop a Universal Speech Translator that will allow the translation of many languages in real time and could eventually help millions of people around the world like Chen’s father become more effective communicators.

“The ability to communicate with anyone in any language — that’s a superpower people have dreamed of forever, and AI is going to deliver that within our lifetimes,” said Meta Founder and CEO Mark Zuckerberg in an online presentation earlier this year.

Using computers to translate languages isn’t a new concept, but previous efforts have focused on written languages. Yet of the 7,000-plus living languages, over 40 percent are primarily oral and do not have a standard or widely known writing system like Hokkien.

AI translation

Building an AI speech translation system for Hokkien was no easy task. These tools are usually trained on large quantities of text. But for Hokkien, there is no widely known standard writing system. Furthermore, Hokkien is what’s known as an underresourced language, which means there isn’t much paired speech data available in comparison with, say, Spanish or English. Also, with few human English-to-Hokkien translators, it was difficult to collect and annotate data to train the model. 

To get around these problems, Meta researchers used text written in Mandarin, which is similar to Hokkien. The team also worked closely with Hokkien speakers to ensure that the translations were correct. “Our team first translated English or Hokkien speech to Mandarin text, and then translated it to Hokkien or English — both with human annotators and automatically,” said Meta researcher Juan Pino. “They then added the paired sentences to the data used to train the AI model.”

The researchers will make their model, code, and benchmark data freely available to allow others to build on their work. While the model is still a work in progress and can currently translate only one full sentence at a time, it’s a step toward a future where simultaneous translation between many languages is possible. 

Challenges of communication

Speakers of unwritten languages often face hurdles when trying to participate in online communities, said Laura Brown, a Meta researcher and linguistic anthropologist. Many of these speakers are not able to easily communicate in the digital realm because they are not used to writing in their language. 

“It can be a barrier to confidence, fluency, and authenticity,” Brown said. “We know at Meta that there are tons of people all over the world who have their interface set to English, who use English on our platforms — even though they are much more confident in other languages and writing systems. As soon as we give them the ability to do audio in their own language, their comfort and confidence in the digital space shoot way up.”

Communicating with speakers of a different language can be challenging for speakers of unwritten languages. It can be hard to recognize the units of sound in an unwritten language when it’s transcribed in a way meant to be understood as it’s heard. This complication often makes it harder to teach unwritten languages and can result in younger generations losing the ability to communicate in the language of their parents. 

Some languages without a standardized written form are at risk of dying out. Linguists are trying to preserve languages with a dwindling number of speakers by writing the languages down, but that can be challenging when they don’t have a conventional written form. Mexico’s National Institute of Indigenous Languages is one institution that is working to preserve the unwritten languages of Indigenous peoples by recording the vocabulary. 

The many possibilities of AI translation

Meta researchers believe AI could help solve many communication challenges for speakers of unwritten languages. Pino said that the new translation system could eventually make it easier to navigate the internet and communicate in different languages, whether virtually or in real life. 

For Chen, though, the goal of the new Hokkien translation system is more personal. “I just want my father to be able to speak to whomever he wants,” he said.