AI like the way you move: How Facebook researchers built an inspirational dancing machine

Since the system’s only computational constraint is ensuring that its movements synchronize with the music, it is able to generate novel dance routines, which human judges have evaluated as highly creative in comparison to other generated dances. 

“The AI generates new sequences of movements that might not have come naturally to people,” says Parikh. “It adds a layer of creativity, because you can visualize a dance in a different form.” This is an important milestone in a research field that holds huge potential in the near future: creative AI. Researchers have made significant progress in teaching machines perception, reasoning, and language in recent years, but teaching AI to be creative and to generate aesthetic attributes is a different open challenge because of how subjective these areas are. We all have different tastes, feelings, and opinions.

The lead researcher behind the system, Purva Tendulkar, who works at Georgia Tech, is keen to emphasize that dancing AI is about assisting the creative process, not taking control of it. “Our dancing agents are meant to augment, not replace, human creations by combining the best of what people and machines each excel at.”

It’s only a research project for now, but Parikh and Tendulkar say the system could someday provide inspiration and creative insights for dancers and choreographers, whether they are amateurs busting some moves in front of the bathroom mirror, or dance industry professionals looking for a new take on a classical ballet production. In the longer run, dancing AI might have potential in video games or fitness apps where people imitate the movements of an avatar.

The system could spark plenty of joy as well as inspiration. Parikh, who has a background in generative art (the process of algorithmically generating new ideas, shapes, and patterns), says that experimenting with dancing AI can be an enjoyable and therapeutic activity, like doodling or painting. “Simply put, this can be fun for people.”

While the concept of dancing AI isn’t new, past research projects have been mainly imitators rather than creators. In recent years, a model was trained by a Swedish team to extract high-level features from a professional dancer’s movements to generate new sequences for solo dancers. A year later, British choreographer Wayne McGregor teamed up with Google engineers to train an algorithm on thousands of hours of video of his work stretching back over 25 years. They built a model that could generate new choreography that was faithful to McGregor’s particular style.

Parikh explains that her system is different from traditional dancing AI because it doesn’t learn from humans and then mimic them. “It generates completely original moves to any music,” she says. The Facebook researchers use a range of dancing agents, including pulsating discs, deforming geometric shapes, and humanoid stick figures. As Tendulkar explains: “The same dance visualized with different patterns or other agents affects the way people perceive the movement and the creative value they draw out of it.” For example, someone might find that a stick figure doesn’t have the right feel for their relaxed mood music but that a pulsating mandala is strangely hypnotic and soothing. 

The way the system picks up on a groove is, computationally speaking, simple but effective. The model receives a snippet of music as its input: This could be anything from the recurring eight-beat phrases of Latin salsa, the highly complex rhythms of Carnatic music from southern India, or even an a capella track with no instrumentation. Whatever the input, the system’s algorithm represents the track’s sound waves in a matrix that plots time against frequency, similar to a spectrogram. When rendered visually, these matrices look a bit like heat maps, with ascending and descending clusters of audio frequencies.   

A search procedure algorithm then identifies digital doppelgangers in the sound wave, analyzing where there are similar patterns of audio features: It could be a crescendo where all the instruments are playing simultaneously, a distinctive drum fill leading from the bridge to the chorus, or even total silence. “It is capturing how similar the music is at two different points in time,” explains Parikh.

Once the algorithm has identified how similar two clusters are, it looks for their double in a dance sequence matrix, where each block represents a dancing agent’s single, discrete state or action within a larger sequence. If you want to visualize the dance sequence matrix, imagine transposing the images from a giant flipbook or zoetrope strip onto a huge one-dimensional grid and then zooming out to reveal the patterns. 

This alignment of digital footprints means that the system can now generate a sequence of movements that is synchronized with the music. This is important, because although people have different tastes in dance, we generally share a couple of principles about the art form. One of those is that good, creative dances stay in time. (We also judge good singers to be pitch perfect, which is why the record industry has made extensive use of audio processing technology such as Auto-Tune in recent years). As Parikh puts it, her system has been primed to express something qualitative — a creative dance — quantitatively.

We also deem dances to be creative if they are unpredictable. The system manages to throw some surprising shapes, because it is arbitrary by design. Once it has moved beyond the sole constraint on its movements — synchronization — then the shackles are off. The alignment of the matrices is, in artistic terms, completely accidental. This means the system is just as likely to generate a stick figure dancing agent that stands stock-still during a recurring crescendo as it is to generate one that throws its arms up in the air in the orthodox way.

Arbitrariness is also baked into the system in other ways. That’s because the system does not cotton to a sequence of movements and follow them in a single direction. “It can choose one of three options at every stage,” explains Parikh. “Go forwards or go backwards to adjacent states, or just stay in place.” Users can also vary the number of steps they want in a dance (25, 50, 100, and so on) for endless permutations. The combination of identifying patterns from a large number of possibilities and electing to stick or twist in an arbitrary fashion keeps the system moving in mysterious ways.

That is not just the subjective view of the researchers on Parikh’s team, which includes Abhishek Das at Facebook AI and Aniruddha Kembhavi at the Allen Institute for AI. The team showed a series of dances to 210 people on Amazon Mechanical Turk, a crowdsourcing website, and asked them to judge the dances on four criteria: Which one went best with the music; Which one was most unpredictable; Which one was most creative; and Which one was most inspiring. The participants used four baseline dances, which varied in terms of how synchronized and sequential they were. The participants judged the AI-generated dances to be overall more creative and inspiring than the baselines.

Parikh’s system is still in the early days of development, but dancing AI that can invent real-time moves in a Zumba class may not be too far off. She hopes to train a neural network to generate dances directly based on the input music without having to perform the search procedure, which currently can take a couple of minutes to analyze, identify, sequence, and sync the whole dance to a music piece that is a few minutes long. “Using a neural network would probably be much faster because it wouldn’t explicitly search through the large number of possible sequences and instead would directly generate a good sequence based on patterns it has seen during training,” she explains. Parikh is also pondering the ability of the system to work back to front, generating music that matches human movements. “That could be a great system that children would enjoy,” she says.

But don’t expect dancing AI systems to replace human choreographers. In fact, it is more likely to just bring out the best in them. Parikh explains that humans often use creative AI to enhance their existing artistic projects, whether it’s dancing, sketches, or painting. She expects both amateur and professional dancers to use her system for flashes of inspiration and insight rather than for mapping out entire routines.  

Parikh is working on more than dancing AI. She has built creative AI systems that generate visual artifacts or motifs for people who are keeping journals or diaries, which allows them to record their feelings in abstract as well as concrete ways. Parikh has also developed an unsupervised approach to creating typography that can doodle themes and words, as well as a different system for neuro-symbolic generative art.  

Facebook researchers have been building creative AI for years, from techniques that enhanced an award-winning virtual reality film to a system that helps people style an outfit. In all these examples, AI does more than just carry out transactional requests: It understands creative and subjective concepts, allowing it to complement and enhance human creativity rather than the reverse. Parikh circles back to the writer’s block analogy. “The ways humans and machines approach problems are so different — but together that diversity can help to unblock,” she says. “That’s more engaging and more satisfying for people.”